Latency issue while streaming a video over RTSP on IMX 8M Plus

[vinay_hasyagar]

Hi

We are trying to achieve live streaming over RTSP with hardware encoder and decoder. We did few tests and it turned out that the latency is coming as more than 1 sec. Below is the gstreamer pipeline that we have used

Server

test-launch -p 1111 "v4l2src device=/dev/video3 ! videorate ! video/x-raw,format=RGB16,framerate=60/1 ! vpuenc_h264 bitrate=4000 ! rtph264pay name=pay0"

Client

GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element+pipeline)" GST_DEBUG_FILE=test.log gst-launch-1.0 rtspsrc latency=0 buffer-mode=0 location=rtsp://127.0.0.1:1111/test ! rtpjitterbuffer latency=0 ! rtph264depay !  queue ! h264parse ! queue ! vpudec ! queue ! imxvideoconvert_g2d ! video/x-raw,format=RGBx ! queue ! kmssink driver-name="imx-drm" can-scale=false && gst-stats-1.0 test.log

This is the result that we obtained Latency Statistics:

0xffff8c01a200.udpsrc0.src|0xaaaaead651e0.waylandsink0.sink: mean=0:00:01.150504971 min=0:00:00.061445875 max=0:00:01.275384875 Element Latency Statistics:

0xffff8c02a160.rtpsession0.recv_rtp_src: mean=0:00:00.000175178 min=0:00:00.000105750 max=0:00:00.001147500

0xaaaaead9c120.rtpstorage0.src: mean=0:00:00.000154823 min=0:00:00.000076750 max=0:00:00.001986000

0xffff7c01fdf0.rtpjitterbuffer1.src: mean=0:00:00.072478069 min=0:00:00.000081250 max=0:00:00.663777250

0xffff8c0320c0.rtpssrcdemux0.src_4125171769: mean=0:00:00.000132382 min=0:00:00.000080000 max=0:00:00.002362125

0xaaaaead301b0.queue0.src: mean=0:00:00.000408832 min=0:00:00.000093000 max=0:00:00.010825250

0xffff8c032700.rtpptdemux0.src_96: mean=0:00:00.000152245 min=0:00:00.000078875 max=0:00:00.003532750

0xaaaaeaba8a10.rtpjitterbuffer0.src: mean=0:00:00.000349613 min=0:00:00.000070625 max=0:00:00.004207125

0xaaaaead304b0.queue1.src: mean=0:00:00.000463768 min=0:00:00.000069875 max=0:00:00.007037875

0xaaaaead540a0.rtph264depay0.src: mean=0:00:00.000261981 min=0:00:00.000094625 max=0:00:00.006137750

0xaaaaead307b0.queue2.src: mean=0:00:00.000431892 min=0:00:00.000082375 max=0:00:00.006226250

0xaaaaead66090.h264parse0.src: mean=0:00:00.000330666 min=0:00:00.000112500 max=0:00:00.006174625

0xaaaaead30ab0.queue3.src: mean=0:00:00.968540419 min=0:00:00.000081875 max=0:00:01.161314875

0xaaaaead78da0.vpudec0.src: mean=0:00:00.015938137 min=0:00:00.001889875 max=0:00:00.053517500

0xaaaaead30db0.queue4.src: mean=0:00:00.093285762 min=0:00:00.000148500 max=0:00:00.098662375

0xffff8c02a160.rtpsession0.sync_src: mean=0:00:00.000372062 min=0:00:00.000252125 max=0:00:00.000492000

0xffff8c0320c0.rtpssrcdemux0.rtcp_src_4125171769: mean=0:00:00.000259562 min=0:00:00.000180000 max=0:00:00.000339125

We tried with different sinks like autovideosink and waylandsink. But results are similar. We want to achieve latency less than 100ms.

Please tell us where we are making it wrong. Its an urgent.

Thank you

[vinay_hasyagar]

Update

Latency is now reduced with the below pipeline.

Server

test-launch -p 1111 "v4l2src device=/dev/video3 ! videorate ! video/x-raw,format=RGB16,framerate=60/1 ! queue ! vpuenc_h264 bitrate=4000 gop-size=0 ! rtph264pay aggregate-mode=1 name=pay0"

Client

GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=test.log gst-launch-1.0 rtspsrc latency=0 buffer-mode=1 drop-on-latency=true location=rtsp://127.0.0.1:1111/test ! rtph264depay ! h264parse ! vpudec disable-reorder=true ! imxvideoconvert_g2d ! video/x-raw,format=RGBx ! kmssink driver-name="imx-drm" can-scale=false processing-deadline=0 && gst-stats-1.0 test.log

Debug Results

Latency Statistics:
0xffff8801ad10.udpsrc0.src|0xaaaad6b37f30.kmssink0.sink: mean=0:00:00.022436174 min=0:00:00.010349750 max=0:00:00.358622500

Element Latency Statistics:
0xffff8802e0a0.rtpsession0.recv_rtp_src: mean=0:00:00.000177477 min=0:00:00.000105375 max=0:00:00.001256375
0xaaaad6b16b40.rtpstorage0.src: mean=0:00:00.000142858 min=0:00:00.000052125 max=0:00:00.001547500
0xffff70021700.rtpjitterbuffer0.src: mean=0:00:00.071563056 min=0:00:00.000082000 max=0:00:00.458135125
0xffff88037000.rtpssrcdemux0.src_3584889917: mean=0:00:00.000126677 min=0:00:00.000079375 max=0:00:00.000657125
0xffff88037640.rtpptdemux0.src_96: mean=0:00:00.000131004 min=0:00:00.000077625 max=0:00:00.001811000
0xaaaad6b12130.rtph264depay0.src: mean=0:00:00.000237500 min=0:00:00.000141875 max=0:00:00.002613625
0xaaaad6b1ae20.h264parse0.src: mean=0:00:00.000274038 min=0:00:00.000154250 max=0:00:00.003266625
0xaaaad6b29910.vpudec0.src: mean=0:00:00.004713775 min=0:00:00.003972750 max=0:00:00.065610375
0xaaaad6b33160.imxvideoconvert_g2d0.src: mean=0:00:00.005498437 min=0:00:00.004700500 max=0:00:00.008582125
0xaaaad6b48350.capsfilter0.src: mean=0:00:00.000198678 min=0:00:00.000113750 max=0:00:00.000558000
0xffff8802e0a0.rtpsession0.sync_src: mean=0:00:00.000343875 min=0:00:00.000278250 max=0:00:00.000405625
0xffff88037000.rtpssrcdemux0.rtcp_src_3584889917: mean=0:00:00.000273500 min=0:00:00.000193250 max=0:00:00.000379000

Latency is keep on varying, maximum pipeline latency is coming as 358ms which is not acceptable. We need latency less than 80ms. The elements which are having more latency are rtpjitterbuffer and vpudec. 

Is anybody know how to still reduce the latency of these two elements?

Thanks