Using FFMPEG on top of lib-imx

Tarek · ‎09-28-2012

Hello,

I'm porting an application that's using libavformat and libavcodec to the i.MX6 platform. The application must use libavformat for playing video because it's a modified version which supports proprietary formatting but the codecs are standard.

Would it be possible to hook libav to lib-imx directly in order to get the hardware accelerated video or does it have to be through Gstreamer?

If Gstreamer is mandatory, Can I set-up this configuration? "App-Video-Player* ------> Gstreamer -----> libavformat* -----> lib-imx -----> VPU"

Is there any examples?

Thanks

小辉尤 · ‎10-18-2013

The following is my test results：

（1）gst-launch

gst-launch playbin2 uri=file:///yujian.avi, the video played good.

freescale IMX535：

#root@freescale /$ gst-launch playbin2 uri=file:///yujian.avi

Setting pipeline to PAUSED ...

Pipeline is PREROLLING ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstAudioSinkClock

test: mp4--OK ts--ok wmv--ok flv--ok wma--ok avi--ok mkv--ok //rmvb--no ,swf--no

rmvb failed:

root@freescale /$ gst-launch playbin2 uri=file:///mingyanghuagu.rmvb

Setting pipeline to PAUSED ...

Pipeline is PREROLLING ...

[INFO] Product Info: i.MX53

VPU Version: firmware 1.4.41; libvpu: 5.3.2

MFW_GST_VPU_DECODER_PLUGIN 2.0.3 build on Sep 18 2013 01:42:31.

MFW_GST_V4LSINK_PLUGIN 2.0.3 build on Sep 18 2013 01:43:10.

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

Got EOS from element "playbin20".

Execution ended after 1247625 ns.

Setting pipeline to PAUSED ...

total time:0:00:00.004183250 ,Render fps:0

Setting pipeline to READY ...

Total rendered:0

>>VPU_DEC: State: Ready to Null

Setting pipeline to NULL ...

[--->FINALIZE v4l_sink

[--->FINALIZE vpu_dec

Freeing pipeline ...

root@freescale /$

(2) gplay //gplay and gst-launch has the same problem.(test: mp4--OK ts--ok wmv--ok flv--ok wma--ok avi--ok mkv--ok //rmvb--no ,swf--no)

root@freescale /video$ gplay hls.rmvb

playbin2 is employed!

fsl_player_init(): Successfully initialize!

fsl_player_set_media_location(): filename=hls.rmvb

[Stopped ][Vol=01][00:00:00/00:00:00][fps:0]Wait status change from 2 to 4

[Stopped ][Vol=01][00:00:00/00:00:00][fps:0][INFO] Product Info: i.MX53

VPU Version: firmware 1.4.41; libvpu: 5.3.2

MFW_GST_VPU_DECODER_PLUGIN 2.0.3 build on Oct 14 2013 17:26:37.

[Stopped ][Vol=01][00:00:00/00:00:00][fps:0]FOUND GST_MESSAGE_TAG!

audio codec: Real Audio G2 (Cook)

FOUND GST_MESSAGE_TAG!

video codec: Real Video 4.0

Wait status change from 2 to 4

MFW_GST_V4LSINK_PLUGIN 2.0.3 build on Oct 14 2013 17:26:59.

Wait status change from 2 to 4

[Stopped ][Vol=01][00:00:00/5124095:34:33][fps:0]Wait status change from 2 to 4

Debug: VPU Decoder Initialization failed

Error: fatal error

[Stopped ][Vol=01][00:00:00/5124095:34:33][fps:0]Wait status change from 2 to 4

[Stopped ][Vol=01][00:00:00/5124095:34:33][fps:0]^C Aborted by signal Interrupt...

[Stopped ][Vol=01][00:00:00/5124095:34:33][fps:0]x

[Stopped ][Vol=01][00:00:00/5124095:34:33][fps:0]^C Aborted by signal Interrupt...

root@freescale /video$ ls

(3) totem //why? how to use it?

root@freescale /video$ totem zhongguoweidao.mkv

Cannot open display:

Run 'totem --help' to see a full list of available command line options.

root@freescale /video$

(4)mplay

Mplay can play all video format well,but the voice is not normal!!

(test: mp4--OK ts--ok wmv--ok flv--ok wma--ok avi--ok mkv--ok rmvb--ok ,swf--no)

root@freescale ~$ mplayer -ao help

Available audio output drivers:

oss OSS/ioctl audio output

mpegpes DVB audio output

v4l2 V4L2 MPEG Audio Decoder output

null Null audio output

pcm RAW PCM/WAVE file writer audio output

**********************************

Thats all！！

The most important thing is that I want to play RMVB，but gplay 、gst-lanuch and totem can't play rmvb.I don't know how to do.Thank you!!

Tarek · ‎10-17-2012

Hi Wei,

I'm calling the VPU Wrapper API's from my application similar to the unit test example.

Now the decoder loop is consuming frames which sounds like the decoder is working.

My problem is with VPU_DecGetOutputFrame()

I'm calling this function after I receive VPU_DEC_OUTPUT_DIS which means the decoded frame is ready.

This function is returning out of range virtual address in pbufVirtY, pbufVirtCb and pbufVirtCr.

I need to copy from the correct Frame virtual address pointer to the display buffer but of course the memcpy is failing.

Can you please explain how to convert the returned address to a valid range?

Thanks,

Tarek

Tarek · ‎10-17-2012

Hi Wei,

Using gdb I found that IOGetVirtMem() is returning an address that is not accessible when allocating buffer size of 537601.

inside the function VPU_DecGetMem()

(gdb) print *buff.virt_uaddr

Cannot access memory at address 0x32d16000

Is there any explanation for that ?

Thanks,

Tarek

KanHU · ‎10-17-2012

1. vpu_wrap unit test should only accept raw video stream, that's means you need extract the video if you are using a container file like mp4.

2. the vpu only accept physical continuous memory with physical address. You need make sure pbufY, pbufCr, pbufCb are valid physical pointer when you register output frames by using VPU_DecRegisterFrameBuffer API.

3, If you need allocate physical continuous memory, you can call VPU_DecGetMem, it will return both physical address and virtual address you can access.

Tarek · ‎10-18-2012

1. I have copied the ProcessInitInfo() function from your example into my application. This function does what you described. Allocate continuous memory by calling VPU_DecGetMem() then register the frame buffer VPU_DecRegisterFrameBuffer().

2. The problem is VPU_DecGetMem() is giving back an out of range address although the function returns VPU_DEC_RET_SUCCESS . So I can't read or write to this pointer.

3. In u-boot I'm passing this command line argument fbmem=28M,10M,10M Do I need to increase these values?

4. Finally, can you please provide a raw video stream file to run the unit test example?

Thanks,

Tarek

fengwei · ‎10-18-2012

You can use "avconv" to covert your mp4 file to raw mpeg4 video format. avconf is a tool from ffmpeg.

Tarek · ‎10-18-2012

I found that the problem is not in the VPU Frame buffer allocation but the IPU.

So fb_render_drawYUVframe() is memcopying to YUV address that was not allocated.

Is this the correct way of using the IPU?

In my situation I'm using SDL - DirectFB for display, How can I ask SDL to use the IPU?

Thanks,

Tarek

fengwei · ‎10-22-2012

I don't know how directfb handle the buffer, but IPU has nothing to do with buffer allocation. If you want video to render into displaying buffer, it should be something mapped from /dev/fbN.

Tarek · ‎10-23-2012

OK, I have a decoded frame in YUV formate:

frameInfo.pDisplayFrameBuf->pbufVirtY;

frameInfo.pDisplayFrameBuf->pbufVirtCb;

frameInfo.pDisplayFrameBuf->pbufVirtCr;

What is the most efficient method to display this frame? "I'm using SDL for my program"

Thanks

fengwei · ‎10-07-2012

Would it be possible to hook libav to lib-imx directly in order to get the hardware accelerated video or does it have to be through Gstreamer?

Yes! "App-Video-Player* ------> libavcodec* -----> vpu_wrapper ------> lib-imx -----> VPU"

We didn't implement ffmpeg avcodec for VPU, so you can try it yourself.

I will give you an unit test for vpu_wrapper as example, which will be in next release. You could build it in yocto by reference to makefile.unit.

https://community.freescale.com/servlet/JiveServlet/downloadBody/93583-102-1-3111/vpu_wrapper_test.t...

Tarek · ‎10-17-2012

Hi Wei,

I've compiled the unit test "decoder part". I get the following error when I ran it:

root@imx6qsabrelite:~# ./test_dec_arm_elinux -i clipcanvas_14348_offline.mp4 -f 2 -interleave 1 -map 0 -tile2linear 0

input bitstream : clipcanvas_14348_offline.mp4

max frame_number : 2147483647, display: 0

[INFO] Product Info: i.MX6Q/D/S

vpu lib version : major.minor.rel=5.4.5

vpu fw version : major.minor.rel_rcode=2.1.2_r25776

vpu wrapper version : major.minor.rel=1.0.24: VPUWRAPPER_ARM_LINUX Build on Sep 19 2012 10:30:47

capability: file mode supported: 0

capability: tile format supported: 1

[INFO] bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0

capability: report frame size supported: 1

normal mode

set input type : normal(0)

DecodeLoop: vpu dec buf failure: ret=1

decode_stream: vpu reset: handle=0x1A180

[INFO] vpu_SWResetFrame Num: 0, [width x height] = [716367420 x 0], dec FPS: 0, total FPS: 0

Decode Failure

Is it expecting a certain type of files?

Using FFMPEG on top of lib-imx

Using FFMPEG on top of lib-imx

Graphics & Display

i.MX6_All

Linux