i.MX8M Mini: imxcompositor_g2d to vpuenc_h264 drops frames

[vincentz63]

I am trying to use imxcompositor_g2d  to downscale video on the i.MX8M Mini before encoding it with vpuenc_h264 and streaming over RTP.

What is the max throughput of imxcompositor_g2d? It seems to be just about to keep-up at 1280x720p60 with some frames dropped, but drops most frames at 1920x1080p30.

Below my pipeline for 1280x720:

gst-launch-1.0 imxcompositor_g2d sink_0::keep-ratio=true sink_0::width=1280 name=c ! video/x-raw,width=1280,height=720,format=BGRx ! queue ! vpuenc_h264 gop-size=60 ! rtph264
pay pt=96 ! udpsink host=192.168.1.214 port=5000 v4l2src ! video/x-raw,width=1280,height=720,format=YUY2 ! queue ! c.sink_0

1280x720p60 + debug output disabled (GST_DEBUG=1): the stream looks stable with seldom dropped frames (could be due to network).

1280x720p60 + debug output enabled (GST_DEBUG=3): the plenty of dropped frames.

0:00:07.756956875 3780 0x5c17400 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:07.664126000
0:00:07.807397125 3780 0x5c17400 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:07.714580000
0:00:07.857458125 3780 0x5c17400 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:07.764623875
0:00:07.908124875 3780 0x5c17400 WARN v4l2src gstv4l2src.c:980:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:07.815306000

1920x1080p30 + debug output disabled (GST_DEBUG=1): most frames are dropped

[Dhruvit]

Hi @vincentz63,

I hope you are doing well.

Video composition:-
Note: the imxcompositor_g2d plugin caused an exception when not adding output caps. Also, internal errors are thrown when any of the images are out of the screen.

Please use this below command and share the observation whether it works for you or not.

Using OV5640 camera at 1080p 30 fps and videotestsrc:
gst-launch-1.0 imxcompositor_g2d background=0x000000FF name=comp sink_0::width=960 sink_0::height=540 sink_0::rotate=4 sink_1::width=960 sink_1::height=540 sink_1::xpos=960 sink_1::ypos=540 sink_1::rotate=1 ! video/x-raw,format=RGBA,width=1920,height=1080 ! perf print-cpu-load=true ! queue ! fakesink v4l2src device=/dev/video1 ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! comp.sink_0 videotestsrc ! video/x-raw,format=I420,width=1920,height=1080 ! queue ! comp.sink_1

Using OV5640 camera at 1080p 30 fps and 1080p-hevc-30fps.mkv file:
gst-launch-1.0 imxcompositor_g2d background=0x000000FF name=comp sink_0::width=960 sink_0::height=540 sink_0::rotate=4 sink_1::width=960 sink_1::height=540 sink_1::xpos=960 sink_1::ypos=540 sink_1::rotate=1 ! video/x-raw,format=RGBA,width=1920,height=1080 ! vpuenc_hevc ! perf print-cpu-load=true ! queue ! fakesink \
\
v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=1920,height=1080 ! queue ! comp.sink_0 \
\
filesrc location="1080p-hevc-30fps.mkv" ! matroskademux ! queue ! vpudec ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! comp.sink_1

I hope this information helps!

Thanks & Regards,
Dhruvit Vasavada