MX6Q VPU gstreamer-imx and intra refresh

[pawelcern]

Hi,

I have to set up low latency H.264 video stream through low throughput data link. Video comes from PAL camera. Data link throughput is around 5Mbps. In classical approach, I-frames are problematic. They are pretty bandwidth consuming and transition from one end to other takes unacceptable period of time. I read about "intra refresh" technology, i. e. redistributing I-frame information across multiple P-frames. And started testing this approach on i.MX6Q with analog camera connected through ADV7180. Encoder gstreamer pipeline looks as follows:

gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxvpuenc_h264 bitrate=4000 gop-size=0 intra-refresh=90 ! rtph264pay pt=96 ! udpsink host=<....> port=6000

As you can see, I-frame generation is disabled (only one is produced at the beginning), as gop-size is set to 0. Intra-refresh is set to 90. All the screen in PAL resolution consists of 1620 macroblocks, so intra refresh period should take at most 18 frames.

When there is no data loss on the link, the picture looks fine. But when frame drop occurs, there are artifacts on the screen, going slowly right/down. The distortion is visible ~ 10 - 20 seconds. It seems intra refresh just doesn't work in this scenario.

For a comparison, I established similar pipeline on PC, with software x264 encoder plugin:

gst-launch-1.0 v4l2src device=/dev/video0 ! x264enc intra-refresh=true bitrate=1000 ! rtph264pay pt=96 ! udpsink host=<....> port=6000

And it looks better. Intra refresh process is clearly visible - vertical stripes of the image are processed step by step and finally the picture is OK.

I am pretty sure I did something wrong with encoder parameters. Could you please provide working example, which does H.264 encoding with intra-refresh and without I-frames? It may be without gstreamer plugin, may be just with imxvpuapi.

Kind regards

Pawel

[bobsmith1914]

Is this work available on a github repo?

[pawelcern]

OK, I solved the problem. I modified gstreamer imx plugin.

To make it working as I expected I had to:

- alter "constrained intra prediction" parameter. It is supported by imxvpuapi, but gstreamer plugin keeps this untouched. Turning on this option makes things better. So I exposed this parameter to user.

- periodically add headers. Original gstreamer imx plugin appends headers to I frames. I added possibility (new parameter available to user) to periodically add headers to P frames. Without headers, receiver has no idea concerning video parameters, so even can't show the window.

[joanxie]

the difference between two commands is the first one useing vpu encoder, the second one using software plugin,

if you want to use vpu, pls try to refer to the enclosed document, you can focus on the skippicture option, also you can refer to the encoder option for your case.