topic Keyframes missing in gstreamer vpuenc h264 stream? in i.MX Processors

Keyframes missing in gstreamer vpuenc h264 stream?

patrickrogers — Tue, 24 Dec 2013 01:40:22 GMT

I'm experiencing an issue where keyframes do not appear (or are not marked correctly) in a gstreamer h264 stream encoded using vpuenc.

There are two reasons for this claim:

After capturing the h264 to disk using filesink, I dump the frame data using ffprobe. If the h264 stream is recorded directly to disk, key_frame=1 appears for the very first frame but then never again (all other frames marked key_frame=0). When captured from the network via UDP, there is never a key_frame=1, most likely because the initial keyframe was missed.
On the consumer side, I inspect incoming gstreamer video buffers, looking for keyframes using the condition GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DELTA_UNIT) == FALSE. This condition is never satisfied.

GStreamer producer pipeline:

videotestsrc is-live=1 horizontal-speed=1 ! video/x-raw-yuv, width=640, height=480, framerate=30/1, pixel-aspect-ratio=1/1 ! vpuenc codec=6 force-framerate=true seqheader-method=3 bitrate=524288 ! video/x-h264, format=byte-stream ! rtph264pay pt=96 ! udpsink host=192.168.3.120 port=5010

GStreamer consumer pipeline:

udpsrc port=5010 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 ! rtph264depay ! h264parse ! filesink location=vpuenc-sample.h264

GStreamer details:

gst-inspect-0.10 version 0.10.35
GStreamer 0.10.35

vpuenc details:

vpuenc versions
plugin: 3.0.7
wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Dec 20 2013 15:09:05)
vpulib: 5.4.12
firmware: 2.1.9.36350

I do not have this issue when encoding with x264enc instead of vpuenc. Is it possible to get vpuenc to correctly generate/mark periodic keyframes?

Attached are ffprobe frame dumps for recorded streams encoded with vpuenc and x264enc. Notice the key_frame=1 appearing throughout the x264enc dump.

Original Attachment has been moved to: x264enc.h264.frames.txt.zip

Original Attachment has been moved to: vpuenc.h264.frames.txt.zip

Re: Keyframes missing in gstreamer vpuenc h264 stream?

LeonardoSandova — Tue, 24 Dec 2013 03:27:41 GMT

BTW, did you get this error when using x264enc?

https://community.freescale.com/message/362478#362478

Re: Keyframes missing in gstreamer vpuenc h264 stream?

patrickrogers — Sat, 28 Dec 2013 07:20:34 GMT

No I did not. I am not experiencing an issue with x264enc. My issue is related to vpuenc.

Re: Keyframes missing in gstreamer vpuenc h264 stream?

martinmaranski — Fri, 07 Mar 2014 16:20:06 GMT

Has anyone found a solution to this yet? I've run into the same issue using vpuenc 3.0.8 on the i.MX6 SABRE SDB.

Re: Keyframes missing in gstreamer vpuenc h264 stream?

LeonardoSandova — Thu, 27 Mar 2014 21:42:36 GMT

Martin,

Can you try encoding using unit_tests/mxc_vpu_test.out app and examine the resulting file? This will help us to identify if the problem is at gst level. For example:

./mxc_vpu_test.out -E "-w 640 -h 480 -f 2 -o enc.h264"

Re: Keyframes missing in gstreamer vpuenc h264 stream?

martinmaranski — Fri, 28 Mar 2014 16:49:35 GMT

For direct comparison, I did this:

gst-launch -v videotestsrc num-buffers=500 ! filesink location=in.yuv

./mxc_vpu_test.out -E "-i in.yuv -w 320 -h 240 -o testout3.h264 -f 2"

Then I used ffprobe -show_frames to look for the key_frame=1 flag. This output also only shows the first frame as a keyframe, so the same problem occurs using those options with mxc_vpu_test.out.

I've written some test code that sets the VPU's "nForceIPicture" flag to 1 every few frames and analyzed that output using the same method; that output does have the key_frame=1 periodically, so the VPU can generate I frames on demand. I just can't find a way in gstreamer to make that happen periodically.

Re: Re: Keyframes missing in gstreamer vpuenc h264 stream?

KevinSong — Tue, 15 Apr 2014 05:48:34 GMT

vpuenc only output I frame based on GOP setting. Default GOP is 15. seems customer need IDR frame based on GOP. Please try attached patch.

BTW, why customer need IDR frame output?

Re: Keyframes missing in gstreamer vpuenc h264 stream?

martinmaranski — Tue, 15 Apr 2014 20:24:52 GMT

We're using gstreamer for network streaming where the player may pick up the stream already in progress and miss the first frame.

What version of the plugin is this patch for? I can't build either 3.0.7 or 3.0.8 with it, the error is that there is no gop_frm_cnt member of GstVpuEnc.

Re: Re: Keyframes missing in gstreamer vpuenc h264 stream?

KevinSong — Wed, 16 Apr 2014 02:55:59 GMT

I used our latest code. As we fixed many bug, can you try our latest code? Attached latest VPU encoder plugin code. Attached code include previous patch.

Re: Keyframes missing in gstreamer vpuenc h264 stream?

martinmaranski — Wed, 16 Apr 2014 15:57:53 GMT

I'm sorry, I can't get these files to build either - I get this error:

vpuenc.c:538:30: error: 'VpuEncOpenParamSimp' has no member named 'nIsAvcc'

I'm using the source I downloaded from i.MX 6 Series Software and Development Tool Resources as my baseline, that's "L3.0.35_4.1.0_130816_source.gz" with "IMX_MMCODEC_3.0.35_4.1.0_BUNDLE_CODA.gz" installed. Is there something newer I should be using?

Re: Re: Keyframes missing in gstreamer vpuenc h264 stream?

KevinSong — Thu, 17 Apr 2014 06:30:16 GMT

Please also update vpu_wrapper to latest code as VPU encoder plugin depend on VPU wrapper. I prefer update to our latest code as we fixed many bug compare with 3.0.7 release. Attached VPU wrapper code.

Re: Keyframes missing in gstreamer vpuenc h264 stream?

martinmaranski — Fri, 18 Apr 2014 14:31:18 GMT

Okay, with the patch applied to the latest code I'm seeing the IDR frames and the client is able to pick up the stream. Thank you for your help.