gstreamer dual rendering from webcams

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

gstreamer dual rendering from webcams

1,253 Views
jeromebolduc
Contributor III

Hello,

We are having issues with gstreamer using dual USB webcam when we render them through HDMI.

We are using the Variscite VAR-SOM-MX6 Quad. We have the same problem using the NXP SabreSD DualLite processor.

When we render 2x Logitech C930 webcams at 720p30 MJPEG onto screen, the video stutters (skipping/repeat) frames when using the following pipelines.

Here is our variables for the pipelines :

DEVICE="/dev/video1"
VELEM="v4l2src device=$DEVICE do-timestamp=false"
VCAPS="image/jpeg, width=1280, height=720, framerate=30/1"
VSOURCE="$VELEM ! $VCAPS"

DEVICE2="/dev/video2"
VELEM2="v4l2src device=$DEVICE2"
VCAPS2="image/jpeg, width=1280, height=720, framerate=30/1"
VSOURCE2="$VELEM2 ! $VCAPS2"

VIDEO_DECODE="vpudec"

HDMI_DISPLAY="imxv4l2sink device=/dev/video18 sync=false"
LCD_DISPLAY="fpsdisplaysink video-sink=overlaysink text-overlay=false sync=false"
VIDEO_CONVERT="imxvideoconvert_g2d"


gst-launch-1.0 -v $VSOURCE ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $HDMI_DISPLAY & $VSOURCE2 ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $LCD_DISPLAY

Same problem when we execute the two pipelines in two separate processes.

gst-launch-1.0 -v $VSOURCE ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $HDMI_DISPLAY \

gst-launch-1.0 -v $VSOURCE2 ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $LCD_DISPLAY

Same problem when we use the g2d compositor and render them to the hdmi interface.

GST_DEBUG=0 gst-launch-1.0 -v imxcompositor_g2d name=mix \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720 \
sink_1::xpos=1280 sink_1::ypos=180 sink_1::width=640 sink_1::height=360 \
! imxv4l2sink device=/dev/video18 sync=false\
$VSOURCE ! $VIDEO_DECODE ! mix.sink_0 \
$VSOURCE2 ! $VIDEO_DECODE ! mix.sink_1

The streams seem to render fluid some time and then it starts to choke for unknown reasons. 

Do you have any ideas why it is happening?

This is not a Variscite issue since I can duplicate the problem on the NXP DualLite eval board.

I'm using kernel 4.1.15 with gstreamer 1.8.1.

Tell me if you need any more info.

Regards,

Jérôme

0 Kudos
3 Replies

757 Views
jeromebolduc
Contributor III

Another point to add. 

Do you think I will get better display performance using Wayland instead of X11?

Also, the stuttering problem seem to occur more often when there are a lot of CPU operations like SSH transfers.

If the system if idle, the CPU usage is about 10% while grabbing the 2 sources.

0 Kudos

757 Views
igorpadykov
NXP Employee
NXP Employee

Hi jerome

one can try pipelines described in sect.7.3.9 Web camera attached Linux Guide

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

0 Kudos

757 Views
jeromebolduc
Contributor III

Hi Igor,

Our setup is more complex than that. We need to do compositing with the two webcam streams and show them on HDMI and LVDS at the same time. We would like to have PiP layouts. The following pipeline seem to do the trick [showing 720p over a 1080p picture] but for some reasons, the video stutters randomly in both streams, not necessarily at the same time. Do you know why? Sometimes the video is really smooth and then, the video starts to stutter and then it become smooth again.

DEVICE="/dev/video1"
VELEM="v4l2src device=$DEVICE"
VCAPS="image/jpeg, width=1920, height=1080, framerate=30/1"
VSOURCE="$VELEM ! $VCAPS"

DEVICE2="/dev/video0"
VELEM2="v4l2src device=$DEVICE2"
VCAPS2="image/jpeg, width=1280, height=720, framerate=30/1"
VSOURCE2="$VELEM2 ! $VCAPS2"

VIDEO_DECODE="jpegparse ! vpudec name=vpudec1 "
VIDEO_DECODE2="jpegparse ! vpudec name=vpudec2 "
VIDEO_SOFT_DECODE="jpegparse ! jpegdec"

GST_DEBUG=*vpudec*:0 gst-launch-1.0 -v imxcompositor_g2d name=mix \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 \
! fpsdisplaysink video-sink="overlaysink display-slave=false" text-overlay=false \
$VSOURCE ! $VIDEO_DECODE ! mix.sink_0 \
$VSOURCE2 ! $VIDEO_DECODE2 ! mix.sink_1 

We are using vpudec and decode the stream from MJPEG to RAW. 

We do not have any problem when decoding only 1 stream from any Logitech camera. Do you think we might have an issue with vpudec when using two streams? We do not have access to vpudec settings through gstreamer. Is there a way to play with the vpudec pool?

Do you know if that could be a graphical issue? Is Wayland be more efficient than X11?

Regards,

Jérôme

0 Kudos