Gstreamer dropping frames at source when streaming the data from camera

[lazy12316]

Hi all,

We have imx8m plus eval board and a Tamron Camera (with optical zoom capability). We have connected the camera to eval board via USB-C Port. We are planning to stream data over RTSP from the camera at 1080p@30fps using gstreamer framework.

We are able to stream the data at required resolution i.e 1080p using gstreamer commands. When we enable GStreamer debug and check the feed, we observe a lot of frames being dropped in the debug messages. 

My pipeline is as follows;

 GST_DEBUG=2 gst-launch-1.0 v4l2src device=/dev/video3 ! videoconvert ! video/x-raw,width=1920,height=1080,framerate=30/1 ! videoconvert ! tee name=t ! queue ! vpuenc_h264 bitrate=10000 ! h264parse ! splitmuxsink location=video10_%02d.mp4 max-size-time=10000000000 t. ! queue ! vpuenc_h264 bitrate=6000 ! rtph264pay name=pay0 pt=96 ! udpsink port=5000

The debug info is shown below;

Setting pipeline to PAUSED ...
0:00:00.417745927 689 0xaaaaf8c078c0 WARN v4l2 v4l2_calls.c:547:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video3'.
0:00:00.418518678 689 0xaaaaf8ea6460 WARN v4l2src gstv4l2src.c:694:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.418598704 689 0xaaaaf8ea6460 WARN aggregator gstaggregator.c:1990:gst_aggregator_query_latency_unlocked:<muxer> Latency query failed
====== VPUENC: 4.6.1 build on May 11 2021 03:19:55. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Jun 3 2021 04:20:32)
vpulib: 1.1.1
firmware: 1.1.1.43690
====== VPUENC: 4.6.1 build on May 11 2021 03:19:55. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Jun 3 2021 04:20:32)
vpulib: 1.1.1
firmware: 1.1.1.43690
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.645896594 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:00.182123950
0:00:01.046239018 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:00.582521950
0:00:01.345941780 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:00.882591200
0:00:01.712827214 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.249077950
0:00:01.900602723 689 0xaaaaf8ea6460 WARN qtmux gstqtmux.c:3076:gst_qt_mux_start_file:<muxer> Robust muxing requires reserved-moov-update-period to be set
0:00:02.946463302 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.483026951
0:00:03.312895458 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.849774325
0:00:03.746807334 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.283214075
0:00:04.947101525 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:04.483445826
0:00:05.213629234 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:09.550845076
0:00:10.481054522 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:10.017426575
0:00:10.681378141 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:10.217449576
0:00:10.743540104 689 0xaaaaf8ea6460 WARN qtmux gstqtmux.c:3076:gst_qt_mux_start_file:<muxer> Robust muxing requires reserved-moov-update-period to be set
0:00:10.781366797 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:10.317465451
0:00:10.947472088 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:10.484462326
0:00:11.081193614 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:10.617506200
0:00:11.281255814 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:10.817539201
0:00:11.747986657 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:11.284571450
0:00:12.214678434 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:11.751290201
0:00:12.348024478 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:11.884612450
0:00:12.514728899 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:12.051311575
0:00:12.914359609 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:12.451053325
0:00:13.214386155 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:12.751083451
0:00:13.581511764 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:13.117779576
0:00:13.714538938 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:13.251438451
0:00:14.314866462 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:13.851494450
0:00:14.447858318 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:13.984532701
0:00:15.048279720 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:14.584593576
0:00:15.481659588 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:15.017946576
0:00:15.747958201 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:15.284635450
0:00:16.015052037 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:15.551667076
0:00:16.181642095 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:15.718330576
0:00:16.281641186 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:15.818324326
0:00:16.415048922 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:15.951362451
0:00:16.581769087 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:16.118035325
0:00:16.714796854 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:16.251382576
0:00:17.048138674 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:16.584738326
0:00:17.581450975 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:17.118391325
0:00:17.781753956 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:17.318412451
0:00:17.915160054 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:17.451462325
0:00:18.115186129 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:17.651800576
0:00:19.381968875 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:18.918215950
0:00:19.581983952 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:19.118235701
0:00:19.681709088 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:19.218529451
0:00:20.015264651 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:19.551584450
0:00:20.612442411 689 0xaaaaf8ea6460 WARN qtmux gstqtmux.c:3076:gst_qt_mux_start_file:<muxer> Robust muxing requires reserved-moov-update-period to be set
0:00:20.881612482 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:20.418282451
0:00:21.315360646 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:20.851643451
0:00:23.948488516 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:23.485091701
0:00:24.115455200 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:23.651744451
0:00:24.215349939 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:23.751746576
0:00:24.415092079 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:23.951759576
0:00:24.582108249 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:24.118423700
0:00:24.948738593 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:24.485429701
0:00:25.182149352 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:24.718441201
0:00:25.315457443 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:24.851774076
0:00:25.415389777 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:24.951774451
0:00:25.848846873 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:25.385125450
0:00:26.182104099 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:25.718761201
0:00:29.448488834 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:28.985177576
0:00:29.781811289 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:29.318497451
0:00:29.982139204 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:29.518820576
0:00:30.248482052 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:29.785505325
0:00:30.348515940 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:29.885511076
0:00:30.748885208 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:30.285173325
0:00:30.788016672 689 0xaaaaf8ea6460 WARN qtmux gstqtmux.c:3076:gst_qt_mux_start_file:<muxer> Robust muxing requires reserved-moov-update-period to be set
0:00:30.982138045 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:30.518522576
0:00:31.248595692 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:30.785182576
0:00:31.681811910 689 0xaaaaf8ea6520 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:31.218482577
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:31.400461867
Setting pipeline to NULL ...
0:00:31.833183399 689 0xaaaaf8ea6460 WARN qtmux gstqtmux.c:1930:gst_qt_mux_send_buffer:<muxer> Failed to send buffer (0xffff8c077a20) size 11239
0:00:31.833524366 689 0xaaaaf8ea6460 WARN qtmux gstqtmux.c:5024:gst_qt_mux_add_buffer:<muxer> error: Failed to push sample.
ERROR: from element /GstPipeline:pipeline0/GstSplitMuxSink:splitmuxsink0/GstMP4Mux:muxer: Could not multiplex stream.
Additional debug info:
../git/gst/isomp4/gstqtmux.c(5024): gst_qt_mux_add_buffer (): /GstPipeline:pipeline0/GstSplitMuxSink:splitmuxsink0/GstMP4Mux:muxer:
Failed to push sample.
An error happened while waiting for EOS
Freeing pipeline ...

Since in the above pipeline, we are streaming data over udp (for testing purpose) and recording as well, we planned to use simple capture pipeline to verify if the frameloss is valid for just capture as well. 

The simple capture pipeline as follows:

GST_DEBUG=2 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! videoconvert ! fakesink

The debug info is below;

GST_DEBUG=2 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! videoconvert ! fakesink
Setting pipeline to PAUSED ...
0:00:00.292327509 7077 0xaaab0141f890 WARN v4l2 v4l2_calls.c:547:gst_v4l2_subscribe_event:<v4l2src0> Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video3'.
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.302473733 7077 0xaaab01475360 WARN v4l2bufferpool gstv4l2bufferpool.c:820:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> Uncertain or not enough buffers, enabling copy threshold
0:00:02.007465253 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.681790654
0:00:02.840816451 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.515141779
0:00:03.174052943 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.848472654
0:00:03.340999896 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:03.015155529
0:00:05.040648902 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:04.715166029
0:00:05.574015421 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:05.248505904
0:00:07.007349147 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:06.681867029
0:00:07.407437421 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 3 - ts: 0:00:07.081865904
0:00:08.574338860 7077 0xaaab01475360 WARN v4l2src gstv4l2src.c:1001:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:08.248545779
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.615954037
Setting pipeline to NULL ...
Freeing pipeline ...

We could see the frame drops for simple capture pipeline as well. We tried the solution available in the post . Increased the V4l2 MIN_BUFFER to 7 but facing the same issue.

Could anyone please help in fixing this issue.

[joanxie]

what bsp version do you test? did you test the latest bsp?