We are having a problem with a gstreamer pipeline locking up or actually going into a severe stuttering state (frames updating every 10 - 20s), occasionally when paused/unpaused which looks to be something todo with the imxcompositor_g2d gstreamer module.
The video pipeline will run fine for many hours normally, but if you pause it for a few minutes and repeatedly unpause/pause it will sometimes go into this state. If you leave the pipeline running, it will eventually (minutes) sort itself out.The imxcompositor_g2d sources are a CSI2 camera via v4l2src and an appsrc for an overlay. Both input streams are 720x576@25. On the viewed output video it is only the camera source that has this issue, the overlay is seen updating fine.
There is one of our own gstreamer modules, beamdeinterlace, in the path but this pretty simple (attached). Debug messages in beamdeinterlace show that it is processing video frames at 25 fps when the problem happens fine. Adding probes to the imxcompositor_g2d sinks show bufferers passing through on both sinks fine during the issue.
Our simplified pipeline is:
imxcompositor_g2d name=c sink_1::alpha=1.0 ! queue ! waylandsink name="videoSink" v4l2src device=/dev/video3 ! beamdeinterlace ! video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc name="appsrc" ! videoconvert ! c.sink_1
I have even tried tee'ing the output of the beamdeinterlace into a video file and that is fine. So all points to the imxcompositor_g2d dropping the camera video stream frames somehow. There are no error/warning messages printed.
Are there any known issues with imxcompositor_g2d that could cause this sort of behaviour ?
We are still having this issue.
I have created a simple test example that will run on an imx8mp-evk board with the NXP MINISASTOCSI OV5640 camera that shows this issue with the latest NXP Yocto kirkstone LF5.15.52_2.1.0 BSP. It also includes a patch to fix a minor bug in gstimxcompositor.c.
I enclose the test code and associate information. From the Readme.txt:
This code shows a bug in the NXP Yocto kirkstone LF5.15.52_2.1.0 gstreamer stack with an imx8mp-evk board.
This directory contains a simple test example built to run on an NXP imx8mp-evk board to show the problem
we are having on our custom board.
imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 ! queue ! waylandsink sync=false v4l2src device=/dev/video3 ! queue ! beamdeinterlace ! video/x-raw,width=720,height=576,framerate=30/1 ! c.sink_0 appsrc name=appsrc ! c.sink_1
A camera feeds the imxcompositor_g2d sink0 with a 720x576@30 video stream via the simple beamdeinterlacer plugin and an appsrc feeds
the imxcompositor_g2d sink1. The output of the imxcompositor_g2d is passed to a wayland sink.
The program "test1" performs this. It has a single argument "0, 1 or 2) which defines which test scenario to run.
"test 0" is the normal test with queue before beamdeinterlace.
"test 1" is the test without queue before beamdeinterlace.
"test 2" is the test using the software composer.
The beamdeinterlacer plugin is a simple software deinterlacer setup for this camera to just copy buffers through it.
The appsrc generates a 720x576@30 video stream with a flashing area once per second.
When the imxcompositor_g2d is in use and the pipeline paused/unpaused the system will occassionaly (1 in 10 ?) times get
into a state where the cameras video stream is more or less stalled or has heavy stuttering whilst the overlap appsrc stream
1. Pause the gstreamer pipeline for about 1 minute (Type <Enter> key to pause/unpause).
2. Then pause and unpause the pipleline a few times. Eventually (about 1 in 10 times) after the unpause the video stream will be
seen to be stalled with frame updates occassionaly whislt the appsrc generated overlay continues to run.
Eventually the camera stream will come up to full speed.
When the problem occurs:
1. There are no obvious gstreamer warnings with GST_DEBUG=2
2. Pad probes on the two imxcompositor_g2d sinks show the input frame buffer times incrementing as expected.
3. Debug code inside imxcompositor_g2d shows the camara frame being sent to the HW is stationary.
There is a minor bug in the imxcompositor_g2d where it emits the warning with GST_DEBUG=2
"imxcompositor gstimxcompositor.c:1540:gst_imxcompositor_aggregate_frames:<c:sink_1> set rotate failed 0xffff5c01ca20"
This is a small typo in the imxcompositor_g2d where the message is printed before the actual "if" test.
A patch imx-gst1.0-plugin-beampatch1.patch and an rpm with the fix is included.
If the "queue" is not used before the beamdeinterlacer plugin then the system drops lots of video frames (about 1 in every 3).
1. imx8mp-evk board.
2. MINISASTOCSI OV5640 CSI2 camera plugged into CSI1 MIPI port
3. HDMI montor plugged in
4. NXP BSP Yocto kirkstone LF5.15.52_2.1.0 build and install on Microsdcard
5. USB debug, USB power and Ethernet plugged in.
Any ideas on what is happening and/or where to look in the gstreamer/NXP library code to fix this ?