topic Re: IM6Q Saber SD - vpuenc h264 with mp4 container in i.MX Processors

IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Fri, 13 Dec 2013 19:28:12 GMT

I would like to get the imx6 vpu to encode h264 into a mp4 container. I have seen a few post with people also having this issue but no verified solution. Has anyone been able to successfully get this to work on the imx6q platform? FYI, am using the latest 3.0.7 fsl plugins and the oneiric file system.

The test pipeline I am using is the following.

It appears to preroll and start encoding, however the video it generates can't be played back. Totem reports a video stream error and does not @recognize any video streams in the container. Other things I have tried include using mp4mux instead of ffmux_mp4 and using the h264parse.

Thank you again for any help,

Matthew

Here is the log from running the pipeline:

gst-launch -v --gst-debug=2 videotestsrc ! capsfilter caps="video/x-raw-yuv, width=(int)640, height=(int)480, blocksize=460800" ! queue ! vpuenc codec=6 ! ffmux_mp4 ! filesink location=./output.mp4 sync=false
Setting pipeline to PAUSED ...
[INFO] Product Info: i.MX6Q/D/S
vpuenc versions :)
        plugin: 3.0.7
        wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Jul 30 2013 23:00:47)
        vpulib: 5.4.16
        firmware: 2.3.10.40778
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)480, blocksize=(int)460800, color-matrix=(string)sdtv, chroma-site=(string)mpeg2, framerate=(fraction)30/1
[INFO] chromaInterleave 1, mapType 0, linear2TiledEnable 0
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:src: caps = video/x-h264, width=(int)640, height=(int)480, framerate=(fraction)30/1, framed=(boolean)true
/GstPipeline:pipeline0/ffmux_mp4:ffmux_mp40.GstPad:video_0: caps = video/x-h264, width=(int)640, height=(int)480, framerate=(fraction)30/1, framed=(boolean)true
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = video/quicktime, variant=(string)iso
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
0:00:00.261813059 6832    0x18050 WARN                     bin gstbin.c:2380:gst_bin_do_latency_func:<pipeline0> did not really configure latency of 0:00:00.000000000
New clock: GstSystemClock
^CCaught interrupt -- handling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 3254794337 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/ffmux_mp4:ffmux_mp40.GstPad:video_0: caps = NULL
/GstPipeline:pipeline0/ffmux_mp4:ffmux_mp40.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstVpuEnc:vpuenc0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = NULL
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = NULL
Setting pipeline to NULL ...
Freeing pipeline ...

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Fri, 13 Dec 2013 19:55:30 GMT

You may need to include the format on the capfilter element, something like this

... ! 'video/x-raw-yuv, format=(fourcc)I420, width=(int)1280, height=(int)720, framerate=(fraction)30/1' ! ...

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Fri, 13 Dec 2013 20:58:21 GMT

Thank you Leonardo, those elements should be included. They did not help fix the encoding problem though.

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Sun, 15 Dec 2013 03:53:21 GMT

Hi Matthew, I believe the reason is on this post

https://community.freescale.com/message/344977#344977

In general, some metadata is not being provided by vpuenc to muxer so containers are not playable. I will query this issue internally.

Leo

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Mon, 16 Dec 2013 18:22:03 GMT

Thank you Leo. I agree that is most likely the root cause. When I tested the mp4mux it printed errors about refusing the caps (see below).

Test pipeline:

gst-launch -v --gst-debug=2 mfw_v4lsrc device=/dev/video0 fps-n=30 ! queue ! vpuenc codec=6 ! mp4mux ! filesink location=./output.mp4 sync=false

Errors:

0:00:06.579955000 6696 0x137f80 WARN	qtmux gstqtmux.c:2994:gst_qt_mux_video_sink_set_caps:<mp4mux0> no stream-format field in h264 caps
0:00:06.580211667 6696 0x137f80 WARN	qtmux gstqtmux.c:3150:gst_qt_mux_video_sink_set_caps:<mp4mux0> pad video_00 refused caps video/x-h264, width=(int)640, height=(int)480, framerate=(fraction)30/1, framed=(boolean)true

I am hoping the this is the same issue as in the other post and I can apply the patch to fix the problem.

Sincerely,

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Mon, 16 Dec 2013 22:12:25 GMT

Matt,

I did not realize but there is a patch from uwekaiser but he tested on a 53 board.

https://community.freescale.com/message/367704#367704

I will prepare the setup including it and build. I let you know the result.

Leo

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Mon, 16 Dec 2013 22:20:21 GMT

I would greatly appreciate that Leo. If I had not already mentioned, I am using the prebuilt 3.0.7 plugins with the oneiric file system.

Regards,

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Tue, 17 Dec 2013 16:55:19 GMT

Hi Matt. the patch mentioned before cannot be applied directly to 3.0.35_4.1.0 BSP. According to the MM team, this issue has been solved on 3.10.17 BUT has not been released yet. In the meantime, Can you use any other muxer besides the mp4 ones?

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Tue, 17 Dec 2013 17:33:33 GMT

Leo, thank you for the quick response. Unfortunately, using a mp4 container is a requiment for the project that I am working on and would not be easy to change. In the meantime I have been using the mkv mux as a placeholder so I can test my application, but have been eager to get the mp4 working. Is there a planned release date for the new BSP? I have about two months left until my project deadline and would need them before that. If the date is tbd, can I get a early copy of the vpuenc codec to try?

Sincerely,

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Tue, 17 Dec 2013 17:55:40 GMT

I do not have the exact date but on January 2014 the 3.10.17-1.0.0 beta will be released. This should contain many fixes, including the one you are interested.

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Tue, 17 Dec 2013 18:13:49 GMT

Thank you very much for investigating this issue for me Leo.

Regards,

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Thu, 23 Jan 2014 18:49:08 GMT

Matt,

what is the status of your issue? Can you try building a Yocto image with latest code 3.10.17? Follow the instructions here

http://git.freescale.com/git/cgit.cgi/imx/meta-fsl-bsp-release.git/tree/imx/README?h=dora_3.10.17-1.0.0_beta

just make sure you are using the manifest under imx/doc and NOT the one on doc/doc directory.

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Thu, 23 Jan 2014 21:28:17 GMT

Leo,

The FAE, Norbel from Freescale had provided me the vpuenc and vpu_wrapper source files that are going to be used in the up coming Ubuntu BSP release. I re-compiled the IMX_MMCODEC_3.0.35_4.1.0_BUNDLE with them but couldn't get h264+mp4 encoded videos to playback. I tested the pipelines using gst-launch and verified the new plugin and wrapper were used because, they showed my build date in the debug messages. I used different combinations of mp4mux, ffmux_mp4, matroskamux and h264parse in the pipeline. Like before, only videos encoded with the matroskamux would playback. A notable difference I saw was with using using the vpuenc with the mp4mux. GStreamer no longer printed warnings about "no stream-format field in h264 caps" and "pad video_00 refused caps video". I had thought this was the main problem and was caused by the vpuenc not providing that field. However, since those messages are not being printed anymore I am at a bit of a loss what could be wrong.

I have not yet used Yocto but, would we certainly willing to try. Do you know if any new information from the multimedia team came out about this bug, such as if it has been fixed in Yocto or the upcoming Ubuntu BSP? If you would also like to try the source files for the Ubuntu build to verify them I can send them to you.

Sincerely,

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Thu, 23 Jan 2014 23:44:56 GMT

Leo,

I am stuck at the "repo sync" step. It is reporting "fatal: no <manifest> in /home/matthew/Desktop/ImxYocto/.repo/manifest.xml". Do you know what this could be? The <manifest> opening and closing tags are definitely in the xml file. I used "fsl-yocto-release-manifest.xml" found in the root/imx/doc (meta-fsl-bsp-release.git - Freescale i.MX Yocto BSP Release Layer) .

Also, it was not listed in the steps but, I downloaded the repo like this

$ curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo

Sincerely,

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Fri, 24 Jan 2014 00:19:08 GMT

Hi Matt,

For the fixes on this release, please ask to Norbel. I got the vpu sources you have tried, but for the moment lets focus on latest release instead.

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Fri, 24 Jan 2014 00:21:12 GMT

Matt,

make sure you have the file on ImxYocto/.repo/manifest.xml and run the repo sync under ImxYocto directory.

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

tongchunyang — Tue, 04 Feb 2014 07:48:45 GMT

Hi Leo

Now I am working on iMX6Q SabreAI(auto). And I got the same problem just like above.

Detail:

1, I download the file named "fsl-yocto-release-manifest.xml" at the url meta-fsl-bsp-release.git - Freescale i.MX Yocto BSP Release Layer .

2, I use the file download above to cover the manifest.xml at the dir "<my work dir>/.repo/manifest.xml" by command cp /<location>/fsl-yocto-release-manifest.xml /media/work/FSL-IMX-YOCTO/fsl-community-bsp/.repo/manifest.xml.

3, Then I run "repo sync" on the dir "/media/work/FSL-IMX-YOCTO/fsl-community-bsp/"

the error is:"fatal: no <manifest> in /media/work/FSL-IMX-YOCTO/fsl-community-bsp/.repo/manifest.xml"

And then I checked the file like you said, I it has exited well.

Please help me.

What wrong with it? And how can I do?

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

LeonardoSandova — Tue, 04 Feb 2014 16:51:22 GMT

Tong, I just execute these steps and it worked

2022 mkdir fsl-community-bsp-master-next
2025 cd fsl-community-bsp-master-next/
2026 repo init -u https://github.com/Freescale/fsl-community-bsp-platform -b master-next
2031 rm .repo/manifest.xml
2033 curl http://git.freescale.com/git/cgit.cgi/imx/meta-fsl-bsp-release.git/plain/imx/doc/fsl-yocto-release-manifest.xml?h=dora_3.10.17-1.0.0_beta > .repo/manifest.xml
2034 repo sync

Let me know your output on these commands.

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

matthewreynolds — Tue, 04 Feb 2014 18:21:47 GMT

Thank you Leo,

I was able to get through the "repo sync" command using those steps.

Matthew

Re: IM6Q Saber SD - vpuenc h264 with mp4 container

tongchunyang — Wed, 05 Feb 2014 00:07:46 GMT

Hi Leo

With your command as above, I finished checkout of the source master-next.

Thanks a lot.