iMX8 Plus "lost frames detected" with 2 image sensors and imxcompositor_g2d

[imsOPX]

Hi,

I'm working on a project involving 2 image sensors connected to iMX8 Plus as follows: 

Scenario 1: Sensor 1 (YUY2) to ISI 0 port and Sensor 2 (YUY2) to ISI 1 port [both ov5640 with the original driver]

Scenario 2: Sensor 1 (RAW Bayer) to ISP 0 port and Sensor 2 (YUY2) to ISI 1 port

The sensors are with resolution 1024x768, frame rate up to 60fps, data format is YUY2 for ISI and RAW Bayer for ISP. I have written the appropriate driver to IMX-ISP for the case with RAW Bayer data sensor.

So in any case I have /dev/video0 and /dev/video1 devices shown up and I'm able to capture the data from every one of the sensors in all of the scenarios using gstreamer. The video is smooth and running without interruptions for hours. The sink is either fbdevsink (disabled weston) or waylandsink (enabled weston).

I'm also able to play the video from 2 sensor simultaneously over weston, so I'm sure that the pipelines are ok and it seems the DMA channels have enough bandwidth to pass the two streams. There is only one warning in the beginning of the stream, which is "lost frames detected: count=1". I guess it is normal.

Anyway, I'm also trying to compose an overlay between the data streams from 2 sensors using imxcompositor_g2d. I have tried both Scenario 1 and 2, and the results are the same - there are lost frames. The pipeline I'm using is as follows.

Pipeline 1:

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1024 sink_0::height=768 sink_0::zorder=55 sink_0::fill_color=0xff00ff00 sink_0::alpha=0.5 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1024 sink_1::height=768 sink_1::fill_color=0x44441133 ! \
 queue ! imxvideoconvert_g2d ! waylandsink \
v4l2src device=/dev/video1 ! "video/x-raw,format=YUY2,width=1024,height=768,framerate=30/1" ! c.sink_0 \
v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1024,height=768,framerate=30/1" ! c.sink_1 

Every time I'm getting A LOT of warnings like this (count may vary, I have seen up to 14 frames lost):

v4l2src gstv4l2src.c:1123:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:30.309684875 

The video is choppy, so there is obviously lost of frames.

So I've tried another approach.
Pipeline 2:

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1024 sink_0::height=768 sink_0::zorder=55 sink_0::fill_color=0xff00ff00 sink_0::alpha=0.5 \
sink_1::xpos=0 sink_1::ypos=20 sink_1::width=1024 sink_1::height=768 sink_1::fill_color=0x44441133 ! \
 queue ! imxvideoconvert_g2d ! waylandsink \
v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1024,height=768,framerate=30/1" ! tee name=t \
t. ! c.sink_0 \
t. ! c.sink_1

As you can see the second stream is 20 pixels shifted so I'm able to see both streams produced by the tee element. There is no problems, no lost frames, the video is smooth.

The CPU usage is no more than 10%, so I don't believe the reason is a lack of CPU time.

So do you know what can be the reason for Pipeline 2 to work without losing frames and Pipeline 1 to fail?

Is there some problem with imxcompositor_g2d when this component is fed by 2 separate video sources (ISI), something like DMA bottleneck or other related to G2D?

[imsOPX · Accepted solution]

And an update:

The problem was in the pipe - the imxvideoconvert_g2d is before imxcompositor_g2d as in the examples "IMX8 Gstreamer User Guide". But anyway, this is the correct pipe:

GST_DEBUG=3 gst-launch-1.0 imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1024 sink_0::height=768 sink_0::zorder=55 sink_0::fill_color=0xff00ff00 sink_0::alpha=0.5 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1024 sink_1::height=768 sink_1::fill_color=0x44441133 ! waylandsink sync=false \
v4l2src device=/dev/video1 ! "video/x-raw,format=YUY2,width=1024,height=768,framerate=30/1" ! imxvideoconvert_g2d ! c.sink_0 \
v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1024,height=768,framerate=30/1" ! imxvideoconvert_g2d ! c.sink_1