gstreamer dual rendering from webcams

[jeromebolduc]

Hello,

We are having issues with gstreamer using dual USB webcam when we render them through HDMI.

We are using the Variscite VAR-SOM-MX6 Quad. We have the same problem using the NXP SabreSD DualLite processor.

When we render 2x Logitech C930 webcams at 720p30 MJPEG onto screen, the video stutters (skipping/repeat) frames when using the following pipelines.

Here is our variables for the pipelines :

DEVICE="/dev/video1"
VELEM="v4l2src device=$DEVICE do-timestamp=false"
VCAPS="image/jpeg, width=1280, height=720, framerate=30/1"
VSOURCE="$VELEM ! $VCAPS"

DEVICE2="/dev/video2"
VELEM2="v4l2src device=$DEVICE2"
VCAPS2="image/jpeg, width=1280, height=720, framerate=30/1"
VSOURCE2="$VELEM2 ! $VCAPS2"

VIDEO_DECODE="vpudec"

HDMI_DISPLAY="imxv4l2sink device=/dev/video18 sync=false"
LCD_DISPLAY="fpsdisplaysink video-sink=overlaysink text-overlay=false sync=false"
VIDEO_CONVERT="imxvideoconvert_g2d"

gst-launch-1.0 -v $VSOURCE ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $HDMI_DISPLAY & $VSOURCE2 ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $LCD_DISPLAY

Same problem when we execute the two pipelines in two separate processes.

gst-launch-1.0 -v $VSOURCE ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $HDMI_DISPLAY \

gst-launch-1.0 -v $VSOURCE2 ! $VIDEO_DECODE ! $VIDEO_CONVERT ! $LCD_DISPLAY

Same problem when we use the g2d compositor and render them to the hdmi interface.

GST_DEBUG=0 gst-launch-1.0 -v imxcompositor_g2d name=mix \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720 \
sink_1::xpos=1280 sink_1::ypos=180 sink_1::width=640 sink_1::height=360 \
! imxv4l2sink device=/dev/video18 sync=false\
$VSOURCE ! $VIDEO_DECODE ! mix.sink_0 \
$VSOURCE2 ! $VIDEO_DECODE ! mix.sink_1

The streams seem to render fluid some time and then it starts to choke for unknown reasons. 

Do you have any ideas why it is happening?

This is not a Variscite issue since I can duplicate the problem on the NXP DualLite eval board.

I'm using kernel 4.1.15 with gstreamer 1.8.1.

Tell me if you need any more info.

Regards,

Jérôme

[jeromebolduc]

Another point to add. 

Do you think I will get better display performance using Wayland instead of X11?

Also, the stuttering problem seem to occur more often when there are a lot of CPU operations like SSH transfers.

If the system if idle, the CPU usage is about 10% while grabbing the 2 sources.

[igorpadykov]

Hi jerome

one can try pipelines described in sect.7.3.9 Web camera attached Linux Guide

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[jeromebolduc]

Hi Igor,

Our setup is more complex than that. We need to do compositing with the two webcam streams and show them on HDMI and LVDS at the same time. We would like to have PiP layouts. The following pipeline seem to do the trick [showing 720p over a 1080p picture] but for some reasons, the video stutters randomly in both streams, not necessarily at the same time. Do you know why? Sometimes the video is really smooth and then, the video starts to stutter and then it become smooth again.

DEVICE="/dev/video1"
VELEM="v4l2src device=$DEVICE"
VCAPS="image/jpeg, width=1920, height=1080, framerate=30/1"
VSOURCE="$VELEM ! $VCAPS"

DEVICE2="/dev/video0"
VELEM2="v4l2src device=$DEVICE2"
VCAPS2="image/jpeg, width=1280, height=720, framerate=30/1"
VSOURCE2="$VELEM2 ! $VCAPS2"

VIDEO_DECODE="jpegparse ! vpudec name=vpudec1 "
VIDEO_DECODE2="jpegparse ! vpudec name=vpudec2 "
VIDEO_SOFT_DECODE="jpegparse ! jpegdec"

GST_DEBUG=*vpudec*:0 gst-launch-1.0 -v imxcompositor_g2d name=mix \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 \
! fpsdisplaysink video-sink="overlaysink display-slave=false" text-overlay=false \
$VSOURCE ! $VIDEO_DECODE ! mix.sink_0 \
$VSOURCE2 ! $VIDEO_DECODE2 ! mix.sink_1 

We are using vpudec and decode the stream from MJPEG to RAW. 

We do not have any problem when decoding only 1 stream from any Logitech camera. Do you think we might have an issue with vpudec when using two streams? We do not have access to vpudec settings through gstreamer. Is there a way to play with the vpudec pool?

Do you know if that could be a graphical issue? Is Wayland be more efficient than X11?

Regards,

Jérôme