Imxcompositor Gstreamer Lagging Problem on IMX8MPlus

[dwightk]

I have three pipelines (as below). Sink pipeline takes videotestsrc and /dev/video3 input and then uses imxcompositor_g2d to convert them into a single stacked image which is then outputted to the appsink element named sourcesink

The sourcesink frames are then passed into the appsrc of the other two pipelines where they are then cropped and turned into .mp4 videos.

Eg. src Pipeline 1 and 2 respectively take the same image, split it into two and then create a video from the top and bottom of it. Pipeline 1 creates an image from /dev/video3 and Pipeline 2 from videotestsrc.

The behaviour I observed is that the videotestsrc is creating videos fine. However, the /dev/video3 after about 30 minutes starts producing frozen videos, where its the same image in every frame. Also the frame is from 3-4 minutes ago.

I noticed on the forums others are using imxv4l2src whereas we are using only v4l2src. What is the difference between the two ?

Please see attached video for further clarity. 157533 is the video at the start after about 10 minutes of running which shows the frames are scrolling normally coming out of /dev/video3. 

161325 is the same pipeline after running for about 1 hr. It is frozen and lagging. However, notice that the corresponding stereo image that comes off the videotestsrc is showing normal streaming.

Any ideas of what might be the problem ?

Sink Pipeline:

imxcompositor_g2d name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=0 sink_1::ypos=1080 sink_1::width=1920 sink_1::height=1080 ! imxvideoconvert_g2d ! video/x-raw,format=BGRx,width=1920,height=2160 !appsink name=sourcesink emit-signals=True videotestsrc ! video/x-raw, width=1920, height=1080, framerate=15/1 ! queue ! comp.sink_1 v4l2src device=/dev/video3 ! video/x-raw, width=1920, height=1080, framerate=15/1 ! queue ! comp.sink_0

Src Pipeline 1:

appsrc name=mpsrc ! videoparse format=8 width=1920 height=2160 framerate=15/1 ! videocrop top=0 bottom=1080 right=0 left=0 ! imxvideoconvert_g2d ! queue ! vpuenc_h264 ! queue ! h264parse ! splitmuxsink name=mpsink location=./cropped_mkv/output%02d.mkv

Src Pipeline 2:

appsrc name=mpsrc2 ! videoparse format=8 width=1920 height=2160 framerate=15/1 ! videocrop top=1080 bottom=0 right=0 left=0 ! imxvideoconvert_g2d ! queue ! vpuenc_h264 ! queue ! h264parse ! splitmuxsink name=mpsink2 location=./cropped_mkv/output%02d.mkv

[Bio_TICFSL]

Hello,

Just for top of my head, since you have working with three videos and the pipeline is correct you might have trouble with DDR. or the connection where is video3 is lost.

the imxv4l2src is exclusively for imx devices where is optimized and tested.

regards

[dwightk]

Hi everyone,

The pipeline is ok syntactically and the camera is streaming normally for the first 20 minutes. Then it starts to develop a slight lag. The lag then appears to accelerate until the pipeline freezes.

For example, a new video is created ever 1 second:

157533 the test starts. The video is 2 seconds behind, which is expected.

158000. The video is 2 seconds behind

159000. The video is 3 seconds behind

160000. The video is 7 seconds behind

160500. The video is 104 second behind. 

What is strange about this is that the videotestsrc is ok, so that means the imx_compositor isn't the problem, otherwise the videotestsrc frames should also be freezing up since both frames are composited into a single image. The /dev/video3 is getting backed up for some reason.

I am testing on the IMX8MPLUS EVK with the NXP MINISASTOCSI  camera module. The frame rate of 15 fps should be manageable. But it doesn't appear to be the case.