Keyframes missing in gstreamer vpuenc h264 stream?

[patrickrogers]

I'm experiencing an issue where keyframes do not appear (or are not marked correctly) in a gstreamer h264 stream encoded using vpuenc. 

There are two reasons for this claim:

1. After capturing the h264 to disk using filesink, I dump the frame data using ffprobe.  If the h264 stream is recorded directly to disk, key_frame=1 appears for the very first frame but then never again (all other frames marked key_frame=0).  When captured from the network via UDP, there is never a key_frame=1, most likely because the initial keyframe was missed.
2. On the consumer side, I inspect incoming gstreamer video buffers, looking for keyframes using the condition GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DELTA_UNIT) == FALSE.  This condition is never satisfied.

GStreamer producer pipeline:

videotestsrc is-live=1 horizontal-speed=1 ! video/x-raw-yuv, width=640, height=480, framerate=30/1, pixel-aspect-ratio=1/1 ! vpuenc codec=6 force-framerate=true seqheader-method=3 bitrate=524288 ! video/x-h264, format=byte-stream ! rtph264pay pt=96 ! udpsink host=192.168.3.120 port=5010

GStreamer consumer pipeline:

udpsrc port=5010 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 ! rtph264depay ! h264parse ! filesink location=vpuenc-sample.h264

GStreamer details:

gst-inspect-0.10 version 0.10.35

GStreamer 0.10.35

vpuenc details:

vpuenc versions

  plugin: 3.0.7

  wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Dec 20 2013 15:09:05)

  vpulib: 5.4.12

  firmware: 2.1.9.36350

I do not have this issue when encoding with x264enc instead of vpuenc.  Is it possible to get vpuenc to correctly generate/mark periodic keyframes?

Attached are ffprobe frame dumps for recorded streams encoded with vpuenc and x264enc.  Notice the key_frame=1 appearing throughout the x264enc dump.

Original Attachment has been moved to: x264enc.h264.frames.txt.zip

Original Attachment has been moved to: vpuenc.h264.frames.txt.zip

[martinmaranski]

Has anyone found a solution to this yet?  I've run into the same issue using vpuenc 3.0.8 on the i.MX6 SABRE SDB.

[LeonardoSandova]

Martin,

Can you try encoding using unit_tests/mxc_vpu_test.out app and examine the resulting file? This will help us to identify if the problem is at gst level. For example:

./mxc_vpu_test.out -E "-w 640 -h 480 -f 2 -o enc.h264"

[martinmaranski]

For direct comparison, I did this:

gst-launch -v videotestsrc num-buffers=500 ! filesink location=in.yuv

./mxc_vpu_test.out -E "-i in.yuv -w 320 -h 240 -o testout3.h264 -f 2"

Then I used ffprobe -show_frames to look for the key_frame=1 flag.  This output also only shows the first frame as a keyframe, so the same problem occurs using those options with mxc_vpu_test.out.

I've written some test code that sets the VPU's "nForceIPicture" flag to 1 every few frames and analyzed that output using the same method; that output does have the key_frame=1 periodically, so the VPU can generate I frames on demand.  I just can't find a way in gstreamer to make that happen periodically.

[KevinSong · Accepted solution]

vpuenc only output I frame based on GOP setting. Default GOP is 15. seems customer need IDR frame based on GOP. Please try attached patch.

BTW, why customer need IDR frame output?

[martinmaranski]

We're using gstreamer for network streaming where the player may pick up the stream already in progress and miss the first frame.

What version of the plugin is this patch for?  I can't build either 3.0.7 or 3.0.8 with it, the error is that there is no gop_frm_cnt member of GstVpuEnc.

[KevinSong]

I used our latest code. As we fixed many bug, can you try our latest code? Attached latest VPU encoder plugin code. Attached code include previous patch.

[martinmaranski]

I'm sorry, I can't get these files to build either - I get this error:

vpuenc.c:538:30: error: 'VpuEncOpenParamSimp' has no member named 'nIsAvcc'

I'm using the source I downloaded from i.MX 6 Series Software and Development Tool Resources as my baseline, that's "L3.0.35_4.1.0_130816_source.gz" with "IMX_MMCODEC_3.0.35_4.1.0_BUNDLE_CODA.gz" installed.  Is there something newer I should be using?

[KevinSong]

Please also update vpu_wrapper to latest code as VPU encoder plugin depend on VPU wrapper. I prefer update to our latest code as we fixed many bug compare with 3.0.7 release. Attached VPU wrapper code.

[martinmaranski]

Okay, with the patch applied to the latest code I'm seeing the IDR frames and the client is able to pick up the stream.  Thank you for your help.

[LeonardoSandova]

BTW, did you get this error when using x264enc?

https://community.freescale.com/message/362478#362478

[patrickrogers]

No I did not.  I am not experiencing an issue with x264enc.  My issue is related to vpuenc.