IM6Q Saber SD - vpuenc h264 with mp4 container

[matthewreynolds]

I would like to get the imx6 vpu to encode h264 into a mp4 container. I have seen a few post with people also having this issue but no verified solution.  Has anyone been able to successfully get this to work on the imx6q platform?  FYI, am using the latest 3.0.7 fsl plugins and the oneiric file system.

The test pipeline I am using is the following.

gst-launch -v --gst-debug=2 videotestsrc ! capsfilter caps="video/x-raw-yuv, width=(int)640, height=(int)480, blocksize=460800" ! queue ! vpuenc codec=6 ! ffmux_mp4 ! filesink location=./output.mp4 sync=false

It appears to preroll and start encoding, however the video it generates can't be played back.  Totem reports a video stream error and does not @recognize any video streams in the container.  Other things I have tried include using mp4mux instead of ffmux_mp4 and using the h264parse.

Thank you again for any help,

Matthew

Here is the log from running the pipeline:

gst-launch -v --gst-debug=2 videotestsrc ! capsfilter caps="video/x-raw-yuv, width=(int)640, height=(int)480, blocksize=460800" ! queue ! vpuenc codec=6 ! ffmux_mp4 ! filesink location=./output.mp4 sync=false
Setting pipeline to PAUSED ...
[INFO]  Product Info: i.MX6Q/D/S
vpuenc versions :)
        plugin: 3.0.7
        wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Jul 30 2013 23:00:47)
        vpulib: 5.4.16
        firmware: 2.3.10.40778
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
[INFO]  chromaInterleave 1, mapType 0, linear2TiledEnable 0
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:src: caps = video/x-h264, width=(int)640, height=(int)480, framerate=(fraction)30/1, framed=(boolean)true
/GstPipeline:pipeline0/ffmux_mp4:ffmux_mp40.GstPad:video_0: caps = video/x-h264, width=(int)640, height=(int)480, framerate=(fraction)30/1, framed=(boolean)true
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = video/quicktime, variant=(string)iso
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
0:00:00.261813059  6832    0x18050 WARN                     bin gstbin.c:2380:gst_bin_do_latency_func:<pipeline0> did not really configure latency of 0:00:00.000000000
New clock: GstSystemClock
^CCaught interrupt -- handling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 3254794337 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/ffmux_mp4:ffmux_mp40.GstPad:video_0: caps = NULL
/GstPipeline:pipeline0/ffmux_mp4:ffmux_mp40.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = NULL
Setting pipeline to NULL ...
Freeing pipeline ...

[LeonardoSandova]

You may need to include the format on the capfilter element, something like this

... !  'video/x-raw-yuv, format=(fourcc)I420, width=(int)1280, height=(int)720, framerate=(fraction)30/1' ! ...

[matthewreynolds]

Thank you Leonardo, those elements should be included.  They did not help fix the encoding problem though.

[LeonardoSandova]

Hi Matthew, I believe the reason is on this post

https://community.freescale.com/message/344977#344977

In general, some metadata is not being provided by vpuenc to muxer so containers are not playable. I will query this issue internally.

Leo

[matthewreynolds]

Thank you Leo.  I agree that is most likely the root cause.  When I tested the mp4mux it printed errors about refusing the caps (see below).

Test pipeline:

gst-launch -v --gst-debug=2 mfw_v4lsrc device=/dev/video0 fps-n=30 ! queue ! vpuenc codec=6 ! mp4mux ! filesink location=./output.mp4 sync=false

Errors:

0:00:06.579955000  6696   0x137f80 WARN	qtmux gstqtmux.c:2994:gst_qt_mux_video_sink_set_caps:<mp4mux0> no stream-format field in h264 caps
0:00:06.580211667  6696   0x137f80 WARN	qtmux gstqtmux.c:3150:gst_qt_mux_video_sink_set_caps:<mp4mux0> pad video_00 refused caps video/x-h264, width=(int)640, height=(int)480, framerate=(fraction)30/1, framed=(boolean)true

I am hoping the this is the same issue as in the other post and I can apply the patch to fix the problem.

Sincerely,

Matthew

[LeonardoSandova]

Matt,

I did not realize but there is a patch from uwekaiser but he tested on a 53 board.

https://community.freescale.com/message/367704#367704

I will prepare the setup including it and build. I let you know the result.

Leo

[matthewreynolds]

I would greatly appreciate that Leo. If I had not already mentioned, I am using the prebuilt 3.0.7 plugins with the oneiric file system.

Regards,

Matthew

[LeonardoSandova]

Hi Matt. the patch mentioned before cannot be applied directly to 3.0.35_4.1.0 BSP. According to the MM team, this issue has been solved on 3.10.17 BUT has not been released yet. In the meantime, Can you use any other muxer besides the mp4 ones?

[matthewreynolds]

Leo, thank you for the quick response.  Unfortunately, using a mp4 container is a requiment for the project that I am working on and would not be easy to change.  In the meantime I have been using the mkv mux as a placeholder so I can test my application, but have been eager to get the mp4 working.  Is there a planned release date for the new BSP?  I have about two months left until my project deadline and would need them before that.  If the date is tbd, can I get a early copy of the vpuenc codec to try?

Sincerely,

Matthew

[LeonardoSandova]

I do not have the exact date but on January 2014 the 3.10.17-1.0.0 beta will be released. This should contain many fixes, including the one you are interested.

[LeonardoSandova]

Matt,

what is the status of your issue? Can you try building a Yocto image with latest code 3.10.17? Follow the instructions here

http://git.freescale.com/git/cgit.cgi/imx/meta-fsl-bsp-release.git/tree/imx/README?h=dora_3.10.17-1....

just make sure you are using the manifest under imx/doc and NOT the one on doc/doc directory.

[matthewreynolds]

Leo,

I am stuck at the "repo sync" step.  It is reporting "fatal: no <manifest> in /home/matthew/Desktop/ImxYocto/.repo/manifest.xml".  Do you know what this could be?  The <manifest> opening and closing tags are definitely in the xml file.   I used "fsl-yocto-release-manifest.xml" found in the root/imx/doc  (meta-fsl-bsp-release.git - Freescale i.MX Yocto BSP Release Layer) .

Also, it was not listed in the steps but, I downloaded the repo like this

$ curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo

Sincerely,

Matthew

[LeonardoSandova]

Matt,

make sure you have the file on ImxYocto/.repo/manifest.xml and run the repo sync under ImxYocto directory.

[tongchunyang]

Hi Leo

Now I am working on iMX6Q SabreAI(auto). And I got the same problem just like above.

Detail:

1, I download the file named "fsl-yocto-release-manifest.xml" at the url meta-fsl-bsp-release.git - Freescale i.MX Yocto BSP Release Layer .

2, I use the file download above to cover the manifest.xml at the dir "<my work dir>/.repo/manifest.xml" by command cp /<location>/fsl-yocto-release-manifest.xml  /media/work/FSL-IMX-YOCTO/fsl-community-bsp/.repo/manifest.xml.

3, Then I run "repo sync" on the dir "/media/work/FSL-IMX-YOCTO/fsl-community-bsp/"

the error is:"fatal: no <manifest> in /media/work/FSL-IMX-YOCTO/fsl-community-bsp/.repo/manifest.xml"

And then I checked the file like you said, I it has exited well.

Please help me.

What wrong with it? And how can I do?

[LeonardoSandova]

Tong, I just execute these steps and it worked

2022  mkdir fsl-community-bsp-master-next
2025  cd fsl-community-bsp-master-next/
2026  repo init -u https://github.com/Freescale/fsl-community-bsp-platform -b master-next
2031  rm .repo/manifest.xml
2033  curl http://git.freescale.com/git/cgit.cgi/imx/meta-fsl-bsp-release.git/plain/imx/doc/fsl-yocto-release-m... > .repo/manifest.xml
2034  repo sync

Let me know your output on these commands.

[tongchunyang]

Hi Leo

The problem of rpm-5.4.9 I have finished by using rpm-5.4.9-0-2012.src.rpm

But where is the "gpu-viv-bin-mx6q-3.10.17-1.0.0-beta-hfp.bin", when bitbaking the error message shown about this thing. And without this thing, building error and never going on...

"WARNING: Failed to fetch URL http://www.freescale.com/lgfiles/MNG/MAD/YOCTO//gpu-viv-bin-mx6q-3.10.17-1.0.0-beta-hfp.bin;fsl-eula..., attempting MIRRORS if available"

And the new thing named "imx-vpu-3.10.17-1.0.0_beta.bin", when bitbaking the error message shown about this thing.

"WARNING: Failed to fetch URL http://www.freescale.com/lgfiles/MNG/MAD/YOCTO//imx-vpu-3.10.17-1.0.0_beta.bin;fsl-eula=ture, attempting MIRRORS if available"

Could not be found files list:

1, gpu-viv-bin-mx6q-3.10.17-1.0.0-beta-hfp.bin

2, imx-vpu-3.10.17-1.0.0_beta.bin

3, imx-lib-3.10.17-1.0.0_beta.tar.gz

Maybe there should have more error informaiton about the compile dependeny about the files just like above...

Now I just confirmed the compile file and set the build image name with fsl-image-x11

The using bitbake at the build dir "bitbake fsl-image-x11".

---------------------------------------------------------------------------------------------------------<here>---------------------------------------------------------------------------------------------------------

Sorry, Leo. I have just solved it by directlly download the missing files from given URL.

So updated it.

---------------------------------------------------------------------------------------------------------<here>---------------------------------------------------------------------------------------------------------

Thanks a lot.

And best wishes.

[LeonardoSandova]

Glad you solved it. BTW, the rpm issue I solved by starting a new build on a clean build folder....

[tongchunyang]

Hi Leo

Please forgive me my stupid question.

As so far, I have build the image for sabre ai sucessfully with bitbake.

But when rerun bitbake, the OS show me the message just like this:

<myworkdir>/fsl-community-bsp/build/$ bitbake

The program ‘bitbake’ is currently not installed. You can install it by typing: sudo apt-get install bitbake.

Could you please tell me what is wrong with it?

Thanks a lot

And best wished

[LeonardoSandova]

Hi Tong, no problem, there are never stupid questions, good that you ask  it.

Every time you open a new terminal, you need to run the setup script. This scripts adds the correct path where bitbake is located.

$ $SHELL

$ echo $PATH
/opt/Adobe/Reader8/bin:/home/b42214/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

$ . setup-environment build
$ echo $PATH
/home/b42214/fsl/yocto/fsl-community-bsp-dora/sources/poky/scripts:/home/b42214/fsl/yocto/fsl-community-bsp-dora/sources/poky/bitbake/bin:/opt/Adobe/Reader8/bin:/home/b42214/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

[tongchunyang]

Hi Leo

That is greate!

Thanks for your helpful answer.

[matthewreynolds]

Leo,

I was also able to build the Yocto image and run it on my board.  I followed the instructions and choose to use a fsl-image-x11 configuration.   However, the h264 encoding is not working like before.   I think the vpu.enc and gstreamer libraries I got are out of date.  A gst-inspect of vpu.enc shows the codec version 3.0.10 and the gstreamer library is 0.10.  Should the vpu enc be 3.10.17 and is gstreamer 1.0 of 0.10 used?  Maybe I used an old recipe?

Please Advise,

Matthew

EDIT:
The serial debug info printed by the kernel during boot at least verifies the 3.10.17 kernel and Pokey 1.5 Yoco version were used.  I had saw a post saying gstreamer 1.0 libraries would be used in the Yocto 1.5 release.  Is that accurate?  Here is some of the  debug info that printed.

Booting Linux on physical CPU 0x0
Linux version 3.10.17-1.0.0_beta+gec1af9f (matthew@matthew-desktop) (gcc version 4.8.1 (GCC) ) #1 SMP Tue Feb 4 13:12:42 MST 2014
CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: Freescale i.MX6 Quad/DualLite (Device Tree), model: Freescale i.MX6 Quad SABRE Smart Device Board

...

Poky (Yocto Project Reference Distro) 1.5 imx6qsabresd /dev/ttymxc0