i.MX ProcessorsのトピックRe: H.264 decoding using ffmpeg + using GPU for display acceleration

H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Thu, 13 Jun 2013 23:25:55 GMT

Hello,

I have a situation where I need to decode low-latency H.264 streams (also known as intra-refresh streams). I have confirmed that the iMX6 cannot decode such streams in hardware because the hardware decoder requires a full I frame which these streams do not contain. So what I would like to do is to use ffmpeg to decode the stream in software and then push the decoded output to the GPU for display.

Has anyone done this or something similar? What would be a recommended architecture for such a sequence? I have tried using MPlayer + SDL + directFB but the video is still choppy. I am open to using gstreamer if at all possible (perhaps use ffmpeg plugin?). I am just trying to brainstorm the best solution for this.

Any input appreciated.

PS: I am using the SabreSD development board with L3.0.35_4.0.0_130424_source.tar.gz to build my rootfs, bootloader, and kernel.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

LeonardoSandova — Tue, 18 Jun 2013 16:18:25 GMT

GPU is just for graphics processing, so what you mean is IPU, right? Have you execute a pipeline using ffmpeg elements? Is the performance good?

Leo

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

ChucoChe — Fri, 21 Jun 2013 22:39:52 GMT

If you mean using a video as a texture there is an example in the GPUSDK under Demos/GLES2.0/simple_gpu_player.

You can download the GPUSDK from freescale's webpage.

Michel

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Tue, 25 Jun 2013 15:37:50 GMT

Sorry I was away for a bit.

So I am trying two approaches: mplayer (which does decode intra-refresh videos) and gstreamer with ffmpeg plugins (not sure if this does intra-refresh videos but I just wanna get it working and maybe will be able to tweak ffmpeg to decode these videos later).

For mplayer:

I did try mplayer (latest) but the video was very choppy. I don't know if it is the decoding of the H264 algorithm that is taking time or if its the colorspace conversion that is slowing things down. So I recompiled mplayer with directfb (old version included in LTIB) but this pretty much hangs the system sometimes or just craps out without hanging the system and also not showing any video, choppy or otherwise. I will post the details of these directfb errors later.

For gstreamer ffmpeg plugin:

I just enabled FFMPEG plugins for gstreamer in LTIB. The ffmpeg plugins did not run because a couple APIs have been removed so when I use gst-inspect, it came up with unresolved symbol error on these plugins. I will probably update these ffmpeg plugins to the latest version but decided to just fix the minor issues in the source code for now. So now I do have the ffmpeg plugins and I will try to build a pipeline to decode H264 Transport Stream files in software.

If anyone knows the pipeline for ffmpeg using gst-launch, please advise.

I will update with any issues I encounter.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

LeonardoSandova — Tue, 25 Jun 2013 16:30:24 GMT

Try this patch https://community.freescale.com/docs/DOC-93450

Leo

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Tue, 23 Jul 2013 16:10:16 GMT

It doesn't look like the i.MX6Q has enough juice to decode full frame rate 1080p streams in software only.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

ChucoChe — Fri, 26 Jul 2013 20:18:53 GMT

What do you mean by full frame rate and software only?

From the reference manual on the VPU section:

68.4 Functional Description

VPU is a high-performance multi-standard video processing block that supports up to full

HD 1920 x 1088 at 30 fps plus D1 at 30 fps decoding and 1920 x 1088 at 30 fps

encoding.

Michel

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Fri, 26 Jul 2013 20:56:49 GMT

By software only I mean through ffmpeg or any software only decoder (no VPU assist). The reason why I was forced to go this route is because the gstreamer software provided cannot decode low-latency streams. Please see this discussion:

https://community.freescale.com/thread/306365

It looks like gstreamer is unable to work with low latency TS files. If I extract the raw video h.264 data from a low latency stream TS using mplayer and pass the file to mxc_vpu_test.out then the VPU is able to decode the file just fine:

mplayer -dumpvideo -dumpfile myclip.264 input.ts

mxc_vpu_test.out -D "-i myclip.264 -f 2 -w 1920 -h 1080 -s 1 -t 1

So it seems to me that the hardware decoder is ok with low latency streams but gstreamer plugin cannot handle it. That is too bad because i.MX6 satisfies all our requirements except this one and now we are forced to look for other chips that can do this (OMAP4/OMAP5 is able to).

There are example low latency stream files that I have uploaded in the message referenced above.

Is there anyway freescale can implement decoding of intra-refresh/low latency H.264 streams by modifying gstreamer? This will make this chip fully able to handle broadcast quality stream.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

ChucoChe — Fri, 26 Jul 2013 21:20:29 GMT

DaianeAngolini, Is it posible the customer codes their own decoder that takes no iframes?

karinavalencia Could we consider a feature request for the customer?

Michel

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

karina_valencia — Mon, 29 Jul 2013 15:19:34 GMT

ChucoChe I answered you by email.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Mon, 29 Jul 2013 16:03:38 GMT

Please let me know the outcome.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

LeonardoSandova — Mon, 29 Jul 2013 17:31:24 GMT

It would be interesting to investigate either if there is an gstreamer element which does the parsing (what you are doing manually with mplayer) or implement one. Just an idea.

Leo

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Mon, 29 Jul 2013 19:13:30 GMT

Can the freescale experts at least provide hints on which gstreamer plugin to look at? Is this an issue with the vpudec plugin? Would I need to change the vpudec plugin code (gst-fsl-plugins/gst-fsl-plugins-3.0.7/src/video/vpu_dec.full/src/mfw_gst_vpu_decoder.c) to prevent it from waiting for an i-frame?

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

LeonardoSandova — Mon, 29 Jul 2013 19:49:24 GMT

I am not an expert. jackmao any idea?

Leo

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

jack_mao — Tue, 30 Jul 2013 01:39:22 GMT

all fsl plugin is based on fsl VPU, codec and parser, to this case, vpu doesn't support the low-latency H.264 decoder, so the related plugins can't fix this issue also, have to use open source plugin to decode this kind of stream.

Jack

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

Tarek — Tue, 30 Jul 2013 07:49:44 GMT

Hi Jack,

I just want to understand this further.

If mxc_vpu_test app can decode the stream then VPU must be capable of decoding it, right? The only difference between a gstreamer pipeline and mxc_vpu_test is the demultiplexer. If Freescale demux doesn't sport this type of streams then it's easy enough to replace it with opensource library and still use the hardware acceleration for the decoding and display!

please advise.

Thanks

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Tue, 30 Jul 2013 17:13:51 GMT

Jack,

Let me reiterate: if I use 'open source' plugins including ffmpeg for gstreamer or mplayer for full decoding then the performance is horrible. So that right there is a no-go.

So that leaves us with using gstreamer or the mxc_vpu_test.out as the only options. As stated above, I can take a low-latency TS, demux it using mplayer, and pass it to mxc_vpu_test.out and it decodes fine. So we know the hardware can support low latency H.264.

Here are my questions regarding gstreamer:

1) In the gst-fsl-plugins-3.0.7 package, what is the difference between vpu and vpu_dec.full directories? Pardon my ignorance, but is vpu_dec.full even being compiled?

2) In vpu_dec.full/src/mfw_gst_vpu_decoder.c, I see references in the code for skipping i-frames. Is this where the decoder is waiting for the i-frame that it never gets from the demuxer?

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

rebelalliance — Tue, 30 Jul 2013 17:22:42 GMT

Tarek,

I am not sure if it's just the demuxer. If you look at vpu_dec.full/src/mfw_gst_vpu_decoder.c, it looks as if the decoder is actively waiting for iframes. I am not an expert on that code but that's what it looks to me and I know that that piece of code is not the demuxer.

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

LeonardoSandova — Tue, 30 Jul 2013 17:33:01 GMT

Hi,

you should post a question to the gstreamer-devel list, asking if it exists this particular demuxer.

Leo

Re: H.264 decoding using ffmpeg + using GPU for display acceleration

Tarek — Tue, 30 Jul 2013 19:10:24 GMT

Hi Rebelalliance

You should look at vpudec.c this is what's being compiled. I think mfw_vpu_.... is the old imx5 plugin