ffmpeg and vpu library

[henrymckeggie]

I'm trying to use ffmpeg with the vpu library.  I'm having trouble with the connection.  I'm using the mxc_vpu_test as an example code base, and trying to have ffmpeg read a video stream from a file and having it decoded by the vpu.  I'm very new to ffmpeg, and am having a difficult time finding examples and pointers to doing this.

[peterhale]

Hey guys.  I'm helping Henry in getting this VPU direct access to work.  Here's more background.  We have a custom C++ video player in Android to play videos with an adjustable frame rate, it currently uses FFMPEG to decode the frames, then uses libyuv to display them to the surface.  The requirement is to pipe the video to the 2nd monitor via HDMI.  We have a prototype working by opening up the port (/dev/graphics/fb1) directly and piping video there directly, but it's really high on CPU and gets even worse if we need to scale it. 

Along game this mxc_vpu_test, where it loaded a test video and played it..scaled and all with very low CPU.  Only trouble is, it only works on certain videos.  At the moment, when we load up one of our H264 videos, this mxc_vpu_test comes back with an error when calling vpu_DecGetInitialInfo.  I've looked in that method (going to get low level here), and it seems to be on line 2642 when calling VpuReadReg, and to get the error, it calls the same VpuReadReg with the flag of RET_DEC_SEQ_ERR_REASON with a value of 1.  So basically, it seems the driver/firmware is giving some error, but I have no idea what the "1" means.

I know you guys say use gstreamer, but we've been having issues getting that to work with the Freescale plug-ins in Android.

Does anyone know what that "1" means, when trying to get initial info about the video stream?

Thanks for any information.

[henrymckeggie]

Thanks again for your help.  I have compiled gstreamer and plugins in ltib.  I'm running the ltib rootfs on an mx53 board.  I'm having problems with it stopping and then it hangs playing nothing when restarted.  Requiring a reboot to play again.

Here's my line:  gst-launch filesrc location=video.mp4  typefind=true ! aiurdemux ! mfw_vpudecoder ! mfw_v4lsink

[LeonardoSandova]

How are you stopping the pipeline? ControlC? ControC should stop the pipeline correctly.

Leo

[henrymckeggie]

Yes.  I'm using Ctl-C to stop, sometime I can start it again, and sometimes I can't, and have to reboot.  Also it stops on it's own sometimes and then needs to be rebooted again.

Also I'm not able to use   fbdevsink as a sink.  I get.

WARNING: erroneous pipeline: could not link mfwgstvpu_dec0 to fbdevsink0

Then is exits.

that command line is:

gst-launch filesrc location=video.mp4  typefind=true ! aiurdemux ! mfw_vpudecoder ! fbdevsink

[LeonardoSandova]

If you do the playback with playbin2, do you get the same error? If yes, then we are in trouble :smileyhappy: Playbin automatically takes the relevant elements, so it would be interesting to see which extra elements is taking. For the fbdevsink, any reason for using it? when using it, mfw_ipucsc needs to be insterted before fbdevsink.

Leo

[henrymckeggie]

playbin2 seems to work the same.  I did just notice that it seems to stop sending signal to the display.  All of a sudden the penguin in the top left corner disappears, the display says it's lost it's signal, and I have to reboot to recover.

Also if I add:   mfw_ipucsc

gst-launch filesrc location=video.mp4  typefind=true ! aiurdemux ! mfw_vpudecoder ! mfw_ipucsc |  fbdevsink

I get: 

fbdevsink: command not found

[LeonardoSandova]

You mean that playbin2 has the same issue? Use ! instead of | when building pipelines :smileyhappy:

Leo

[henrymckeggie]

Hi,

Do you have anything further on this?  Also are there examples using the framebuffer?

[LeonardoSandova]

Any reason why you want to use fbdevsink instead of mfw_v4lsink?

Leo

[henrymckeggie]

Yes.  The end result is for an android device.  We need to play the video on the hdmi display, while the vga display is used for the Android app.  The framebuffer is the easiest way to do that.  We also need to be able to vary the frame rate, at a frame by frame basis.

[LeonardoSandova]

I see. We need some Android export on this thread. Unless you are planning to use gstreamer on Android (not sure if the porting has been successful) I can not help much :smileyhappy:

Leo

[henrymckeggie]

Yes.  it now looks like it's an issue with the video, in this case the vga just stopping as though the power was turned off. 

[henrymckeggie]

Thanks.  This is all very helpful.  I'm looking at the freescale plugin.  I've built it in ltib and now I'm wondering how much work it will be to link it with android libraries.  We are playing video on the hdmi output using the framebuffer with the vpu libraries and the mxc-test program.  This works very well except for the container problem.  We want to play h.264 video only.  No audio. and I've been having difficulty getting that from a mp4 file to the vpu library.

[Tarek]

Forget about mxc-test and just use gstreamer. There are some many examples that Leo wrote on this community forum on how to use gstreamer.

This should handle the container, decoding and HDMI display in a very straight forward way.

eg:

gst-launch filesrc location=file.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_v4lsink

[varsmolta]

If Henry want to link his application in an android environment, shouldn't he be using OpenMax, instead of gstreamer?

[Tarek]

Use ffmpeg to read blocks of data to a buffer then use appsrc gstreamer element to feed the buffers to gstreamer pipeline.

The pipeline will be something like:

appsrc ! vpudec ! mfw_isink

[varsmolta]

Tarek is right. I was able to use this link as a guide to get appsrc working in imx6: Ghosh blog: gstreamer appsrc in action

[Tarek]

Example:

[LeonardoSandova]

Hi Tarek! You rock! Thanks for sharing code.

Leo

[henrymckeggie]

Thanks.  Where do I find the freescale ffmpeg library?