Keyframes missing in gstreamer vpuenc h264 stream?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Keyframes missing in gstreamer vpuenc h264 stream?

Jump to solution
8,189 Views
patrickrogers
Contributor I

I'm experiencing an issue where keyframes do not appear (or are not marked correctly) in a gstreamer h264 stream encoded using vpuenc. 

 

There are two reasons for this claim:

  1. After capturing the h264 to disk using filesink, I dump the frame data using ffprobe.  If the h264 stream is recorded directly to disk, key_frame=1 appears for the very first frame but then never again (all other frames marked key_frame=0).  When captured from the network via UDP, there is never a key_frame=1, most likely because the initial keyframe was missed.
  2. On the consumer side, I inspect incoming gstreamer video buffers, looking for keyframes using the condition GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DELTA_UNIT) == FALSE.  This condition is never satisfied.

 

GStreamer producer pipeline:

videotestsrc is-live=1 horizontal-speed=1 ! video/x-raw-yuv, width=640, height=480, framerate=30/1, pixel-aspect-ratio=1/1 ! vpuenc codec=6 force-framerate=true seqheader-method=3 bitrate=524288 ! video/x-h264, format=byte-stream ! rtph264pay pt=96 ! udpsink host=192.168.3.120 port=5010

 

GStreamer consumer pipeline:

udpsrc port=5010 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 ! rtph264depay ! h264parse ! filesink location=vpuenc-sample.h264

 

GStreamer details:

gst-inspect-0.10 version 0.10.35

GStreamer 0.10.35

 

vpuenc details:

vpuenc versions

  plugin: 3.0.7

  wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Dec 20 2013 15:09:05)

  vpulib: 5.4.12

  firmware: 2.1.9.36350

 

I do not have this issue when encoding with x264enc instead of vpuenc.  Is it possible to get vpuenc to correctly generate/mark periodic keyframes?

 

Attached are ffprobe frame dumps for recorded streams encoded with vpuenc and x264enc.  Notice the key_frame=1 appearing throughout the x264enc dump.

Original Attachment has been moved to: x264enc.h264.frames.txt.zip

Original Attachment has been moved to: vpuenc.h264.frames.txt.zip

1 Solution
3,838 Views
KevinSong
NXP Employee
NXP Employee

vpuenc only output I frame based on GOP setting. Default GOP is 15. seems customer need IDR frame based on GOP. Please try attached patch.

BTW, why customer need IDR frame output?

View solution in original post

11 Replies
3,838 Views
martinmaranski
Contributor I

Has anyone found a solution to this yet?  I've run into the same issue using vpuenc 3.0.8 on the i.MX6 SABRE SDB.

0 Kudos
3,838 Views
LeonardoSandova
Specialist I

Martin,

Can you try encoding using unit_tests/mxc_vpu_test.out app and examine the resulting file? This will help us to identify if the problem is at gst level. For example:

./mxc_vpu_test.out -E "-w 640 -h 480 -f 2 -o enc.h264"

0 Kudos
3,838 Views
martinmaranski
Contributor I

For direct comparison, I did this:

gst-launch -v videotestsrc num-buffers=500 ! filesink location=in.yuv

./mxc_vpu_test.out -E "-i in.yuv -w 320 -h 240 -o testout3.h264 -f 2"

Then I used ffprobe -show_frames to look for the key_frame=1 flag.  This output also only shows the first frame as a keyframe, so the same problem occurs using those options with mxc_vpu_test.out.

I've written some test code that sets the VPU's "nForceIPicture" flag to 1 every few frames and analyzed that output using the same method; that output does have the key_frame=1 periodically, so the VPU can generate I frames on demand.  I just can't find a way in gstreamer to make that happen periodically.

0 Kudos
3,839 Views
KevinSong
NXP Employee
NXP Employee

vpuenc only output I frame based on GOP setting. Default GOP is 15. seems customer need IDR frame based on GOP. Please try attached patch.

BTW, why customer need IDR frame output?

3,838 Views
martinmaranski
Contributor I

We're using gstreamer for network streaming where the player may pick up the stream already in progress and miss the first frame.

What version of the plugin is this patch for?  I can't build either 3.0.7 or 3.0.8 with it, the error is that there is no gop_frm_cnt member of GstVpuEnc.

0 Kudos
3,838 Views
KevinSong
NXP Employee
NXP Employee

I used our latest code. As we fixed many bug, can you try our latest code? Attached latest VPU encoder plugin code. Attached code include previous patch.

3,838 Views
martinmaranski
Contributor I

I'm sorry, I can't get these files to build either - I get this error:

vpuenc.c:538:30: error: 'VpuEncOpenParamSimp' has no member named 'nIsAvcc'

I'm using the source I downloaded from i.MX 6 Series Software and Development Tool Resources as my baseline, that's "L3.0.35_4.1.0_130816_source.gz" with "IMX_MMCODEC_3.0.35_4.1.0_BUNDLE_CODA.gz" installed.  Is there something newer I should be using?

0 Kudos
3,838 Views
KevinSong
NXP Employee
NXP Employee

Please also update vpu_wrapper to latest code as VPU encoder plugin depend on VPU wrapper. I prefer update to our latest code as we fixed many bug compare with 3.0.7 release. Attached VPU wrapper code.

3,838 Views
martinmaranski
Contributor I

Okay, with the patch applied to the latest code I'm seeing the IDR frames and the client is able to pick up the stream.  Thank you for your help.

0 Kudos
3,838 Views
LeonardoSandova
Specialist I

BTW, did you get this error when using x264enc?

https://community.freescale.com/message/362478#362478

0 Kudos
3,838 Views
patrickrogers
Contributor I

No I did not.  I am not experiencing an issue with x264enc.  My issue is related to vpuenc.

0 Kudos