How to achieve lowest latency while decoding RTSP h264 stream using imxvpudec_h264 on imx8mm?

[VincasD]

Our software on Linux Desktop uses this GSTreamer v1.22.3 to achieve VA-API-based hardware-accelerated IP camera video decoding with just about 220ms delay using this pipeline:

rtspsrc location=rtsp://... protocols=tcp latency=100 buffer-mode=slave ! queue max-size-buffers=0 ! rtph264depay ! h264parse ! vah264dec compliance=3 ! glupload ! glcolorconvert ! qmlglsink

Meanwhile,  to show RTSP stream on imx8mm this pipeline is used:

rtspsrc location=rtsp://... protocols=udp latency=200 buffer-mode=slave ! queue max-size-buffers=0 ! rtph264depay ! queue max-size-buffers=0 ! h264parse ! queue max-size-buffers=0 ! imxvpudec_h264 ! queue max-size-buffers=0 ! glupload ! qmlglsink sync=1

It seems that I need to use latency=200 in order to keep stable 25fps. If I reduce rtspsrc latency to same 100 as on Desktop, I get "wobbling", unstable and low framerate, some times dropping to 11fps for example. And if I keep 200ms latency, final real delay (measured by screencaping running timer, etc) is near 400ms, which is rather high for our use case.

I have same FPS issues even if I use "raw" glimagesink, launched via gst-launch-1.0 via terminal.

Maybe there are some "tricks" what I don't know that would help to display RTSP streames with lower latency, but with still stable 25fps video in i.MX8M Mini machines?

Thanks!

[VincasD]

It seems there's sort of regression in GStreamer itself since 1.22.2 - I actually no longer get ~220ms on PC either: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2702

I do get good performance in older 1.22.1 on PC, but I can't build it for NXP because of other bug (seems fixed in 1.22.3):

../src/ext/qt/gstqtglutility.cc:48:10: fatal error: gst/gl/viv-fb/gstgldisplay_viv_fb.h: No such file or directory                                                                                                                                                                        
  48 | #include <gst/gl/viv-fb/gstgldisplay_viv_fb.h>                                                                                                                                                                             

[VincasD]

I've fixed build issues, I believe GStreamer regression is only about vah264dec. I can reach low latency with 1.22.1 on PC, but NXP still produces high latency (~400ms) video .

[Bio_TICFSL]

Hello,

NXP don't have any latency measurement of this, because is relative you are measurement with the keyword = 200, but this is not real, it based on your connection and some fails that  can be present in the line, I use tune=zerolatency and it does the trick for me (200ms latency).

Regards

[VincasD]

> tune=zerolatency
You are probably talking about ENcoding, but I am talking about DEcoding.

I understand that latency=200 is not real latency, is more like a buffer size of rtspsrc. I measure real latency by filming timer on different screen, and taking photo of both timer and result on NXP screen.

On PC with Intel hw decoding I can achieve ~220ms real physical latencu, lag from reality. With NXP so far I can get only about 400.