Poor framerates on iMX6 Solo when decoding H264

ralph__ · ‎10-09-2014

I'm using gstreamer to playback some H264 videos - one is 720p60 and one is 1080p30. The framerates are much worse than expected (the iMX can do 1080p30 and 720p60 allegedly....) and I am certain the hardware is doing the decoding as it only has 1-2% CPU utilisation.

Here are the commands I run to test the video. Either:

gst-launch-0.10 filesrc location=/usr/sample.mp4 ! qtdemux ! vpudec ! mfw_isink

or:

gplay /usr/sample.mp4

Both lines give me the same end result which is this:

1080p30 clip runs at 23fps (reported)

720p60 clip runs at 27fps (reported)

Why are the clips running so slowly? If I play back either clip in VLC on Windows the framerates are as expected; i.e. 30fps for the 1080p30 and 60fps for the 1080p60.

Thanks for your help,

Ralph

igorpadykov · ‎10-09-2014

Hi ralph

I would suggest to look at below links where

all video test procedures are well summarized in one place

https://community.freescale.com/docs/DOC-93387/version/4

Testing things out

http://boundarydevices.com/mx6-video-acceleration-raring-debian/

Best regards

igor

ralph__ · ‎10-09-2014

Hi Igor,

I've looked at both those links - thanks for providing them - however none of the mp4-related commands work for me and the "Sintel" clip that they suggest to use isn't even 1080p30! Even then, the "Sintel" clip only runs at 24fps, not at the 30fps that I'd expect.

Ralph

igorpadykov · ‎10-10-2014

Hi ralph

I would suggest to test with Demo Image

L3.0.35_4.1.0_DEMO_IMAGE_BSP : i.MX 6Quad, i.MX 6Dual, i.MX 6DualLite, i.MX 6Solo and i.MX 6Sololite Linux Binary Demo Files.

One can also try kernel parameters : enable_wait_mode=off, maxcpus=0 (nosmp)

and check if SD (if video is located on it) is high class speed.

Best regards

igor

ralph__ · ‎11-07-2014

Okay, I've rebuilt my system and my results are as follows for a 720p60 clip:

echo "S:1280x720p-60" > /sys/class/graphics/fb0/mode

gst-launch filesrc location=/usr/sample720p60_40Mbps.mp4 typefind=true ! qtdemux ! vpudec framedrop=false ! queue max-size-buffers=10 ! mfw_isink sync=false disp-width=1280 disp-height=720

gplay /usr/sample720p60_40Mbps.mp4

Both the gplay and gst-launch commands give 56fps. If I then switch to a 1080p30 clip:

echo "S:1920x1080p-60" > /sys/class/graphics/fb0/mode

gplay /usr/sample1080p30_40Mbps.mp4

Then I only get 23fps still.

So I still need to get the 56fps for 720p clips up to 60fps, and I need to get the 23fps up to 30fps for 1080p clips.

I've also been trying the daisy, dizzy and master images built from scratch using Yocto. None of these give me the full frame rate results either. :'-(

By the way, if I change the sink to be mfw_v4lsink then I still get the same frame rates.

Ralph

igorpadykov · ‎11-07-2014

I think you should test on Freescale Sabre SD reference board (i.MX6DL procesor)

with latest Demo

L3.0.101_4.1.1_IM6QDLS_BUNDLE : i.MX 6Quad, i.MX 6Dual, i.MX 6DualLite, i.MX 6Solo Linux Binary Demo Files.

If you are using SabreLite board please post this on boundary devices forum

since Freescale images may not work good on Sabrelite board.

~igor

ralph__ · ‎11-07-2014

One thing I haven't mentioned is that on my Nitrogen6_lite Boundary Devices board, I have actually had it running at 1080p30 and 720p60 using the LTIB image. Here is the link where I got the image from:

http://boundarydevices.com/ltib-on-i-mx6-without-x/

gplay /nfs/sample720p60_40Mbps.mp4

[60fps]

echo "S:1920x1080p-60" > /sys/class/graphics/fb0/mode

gplay /nfs/sample1080p30_40Mbps.mp4

[30fps]

There are some strange differences between this image and my own. For LTIB, here is some interesting info:

[kernel boot messages]

Memory: 384MB = 384MB total

Memory: 381008k/381008k available, 143280k reserved, 0K highmem

Linux version 3.0.35-2026-geaaf30e-02079-g2a4aa352 (ericn@ericsam) (gcc version 4.6.2 20110630 (prerelease) (Freescale MAD -- Linaro 2011.07 -- Built at 2011/08/10 09:20) ) #4 SMP PREEMPT Tue Dec 18 08:40:53 MST 2012

CPU identified as i.MX6DL/SOLO, silicon rev 1.1

Virtual kernel memory layout:

vector : 0xffff0000 - 0xffff1000 ( 4 kB)

fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)

DMA : 0xf4600000 - 0xffe00000 ( 184 MB)

vmalloc : 0xa0800000 - 0xf2000000 (1304 MB)

lowmem : 0x80000000 - 0xa0000000 ( 512 MB)

pkmap : 0x7fe00000 - 0x80000000 ( 2 MB)

modules : 0x7f000000 - 0x7fe00000 ( 14 MB)

.init : 0x80008000 - 0x80038000 ( 192 kB)

.text : 0x80038000 - 0x806d81b0 (6785 kB)

.data : 0x806da000 - 0x8072eb20 ( 339 kB)

.bss : 0x8072eb44 - 0x8077a9a0 ( 304 kB)

root@freescale ~$ cat /proc/cpuinfo

Processor : ARMv7 Processor rev 10 (v7l)

processor : 0

BogoMIPS : 1988.28

Features : swp half thumb fastmult vfp edsp neon vfpv3

CPU implementer : 0x41

CPU architecture: 7

CPU variant : 0x2

CPU part : 0xc09

CPU revision : 10

Hardware : Freescale i.MX 6Quad Sabre-Lite Board

Revision : 61011

Serial : 0000000000000000

root@freescale ~$ cat /proc/meminfo

MemTotal: 381200 kB

MemFree: 358960 kB

...

VmallocTotal: 1335296 kB

VmallocUsed: 2964 kB

VmallocChunk: 1331744 kB

Our kernel parameters are all the same and I have CONFIG_HIGHMEM enabled in my kernel config. For my own board I have this:

[kernel boot messages]

Memory: 512MB = 512MB total

Memory: 237596k/237596k available, 286692k reserved, 0K highmem

Linux version 3.10.17 (ralphc@waldorf) (gcc version 4.8.1 (Sourcery CodeBench Lite 2013.11-33) ) #4 SMP Fri Nov 7 11:44:20 GMT 2014

CPU identified as i.MX6DL, silicon rev 1.1

vector : 0xffff0000 - 0xffff1000 ( 4 kB)

fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)

vmalloc : 0xe0800000 - 0xff000000 ( 488 MB) ***LTIB is 1304MB***

lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)

modules : 0xbf000000 - 0xc0000000 ( 16 MB)

.text : 0xc0008000 - 0xc0790f9c (7716 kB)

.init : 0xc0791000 - 0xc18e6200 (17749 kB)

.data : 0xc18e8000 - 0xc1924b60 ( 243 kB)

.bss : 0xc1924b60 - 0xc1983dbc ( 381 kB)

root@freescale ~$ cat /proc/cpuinfo

processor : 0

model name : ARMv7 Processor rev 10 (v7l)

BogoMIPS : 790.52

Features : swp half thumb fastmult vfp edsp neon vfpv3 tls

CPU implementer : 0x41

CPU architecture: 7

CPU variant : 0x2

CPU part : 0xc09

CPU revision : 10

Hardware : Freescale i.MX6 Quad/DualLite (Device Tree)

Revision : 61011

Serial : 0000000000000000

# cat /proc/meminfo

MemTotal: 510728 kB

MemFree: 342616 kB

...

VmallocTotal: 499712 kB

VmallocUsed: 7212 kB

VmallocChunk: 316008 kB

I notice that I have no DMA (2MB) or pkmap allocated on mine (184MB) compared to the LTIB board. Also, the LTIB image has more vmalloc memory than there even is on the board!!! (I only have 512MB according to the datasheet). Could the memory allocation setup be the cause of my problems? There is one other difference and that is in the vpu.imx gstreamer component that is used for decoding mp4 is version 3.0.2 for the LTIB board and 3.0.11 for my board.

Ralph

igorpadykov · ‎11-08-2014

please post this on boundary devices forum

ralph__ · ‎11-10-2014

They don't have a forum.

The issue I have here is with the Freescale graphic libraries and firmware and the fact that neither me nor anyone else has managed to get it decoding at 720p60 or 1080p30.

igorpadykov · ‎11-10-2014

software for this board requires optimized, specially prepared

BSP, it is provided and supported by boundary devices directly.

You can try meta-fsl-arm mailing list

https://lists.yoctoproject.org/listinfo/meta-freescale

ralph__ · ‎12-04-2014

I've already posted to the meta-fsl-arm mailing list but didn't get any replies.

I am already using patches from Boundary Devices. I have tried increasing the VPU frequency but this hasn't fixed things either.

Do you have any ideas what I could investigate to improve the decoding speed on my system? I was thinking it might be something to do with my memory but since this is set up the same way each time by my U-Boot I don't think this is the problem.

Thanks,
Ralph

igorpadykov · ‎12-04-2014

Suggest to ask on meta-fsl-arm mailing who is maintainer of this board

and direct questions to him.

There are many boards with i.MX6, but they are maintained and supported

by its vendors. Also note that i.MX6 perfomance given in RM/Datasheet is given without

OS. For obtaining good performance one needs to fine tune BSP.

Freescale provides own reference designs i.MX6 Sabre SD/AI and BSPs for them,

you can try them

i.MX6S|i.MX 6Solo Processors|Multimedia|Freescale

Best regards

igor

ralph__ · ‎10-14-2014

Hi, I tried those kernel parameters but no luck.

I'll try the demo image and report back.

Thanks,
Ralph

ralph__ · ‎10-20-2014

Sadly those images are all for the Sabre board (whatever that is) and not the IMX6Solo which is what I have. Here is the boot trace before it hangs:

Linux version 3.0.35-2666-gbdde708 (r65388@shlinux3) (gcc version 4.6.2 20110630 (prerelease) (Freescale MAD -- Linaro 2011.07 -- Built at 2011/08/10 09:20) ) #1 SMP PREEMPT Fri Aug 16 10:43:44 CST 2013

CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c53c7d

CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache

Machine: Freescale i.MX 6Quad Sabre-Lite Board

Memory policy: ECC disabled, Data cache writealloc

CPU identified as i.MX6DL/SOLO, silicon rev 1.1

igorpadykov · ‎10-20-2014

Hi ralph

if you are using Sabre-Lite then it is better to

check boundary images, like below

http://boundarydevices.com/running-linaro-ubuntu-on-i-mx6-sabre-lite-and-nitrogen6x/

~igor

ralph__ · ‎10-21-2014

After a reasonably massive struggle, I managed to get the original DEMO image up and running. Sadly, there are no H264 or mp4 files on that image and the gstreamer framework fails when I try to get one of my own mp4 files working that I know work on my own system.

Thanks for the Ubuntu link. I'll try that and see if I have better luck.

xizhouwang · ‎10-21-2014

Can you try a different sink like

gst-launch playbin2 uri=file:///video-file.mp4 video-sink="mfw_v4lsink"

to play your clips (change the file name) .

ralph__ · ‎10-21-2014

I've tried this but all I get is:

gst-launch playbin2 uri=file:///nfs/sample.mp4 video-sink="mf

MFW_GST_V4LSINK_PLUGIN 3.0.10 build on Mar 2 2014 17:33:04.

Setting pipeline to PAUSED ...

Pipeline is PREROLLING ...

Aiur: 3.0.10

Core: MPEG4PARSER_06.07.03 build on Nov 15 2013 01:36:39

mime: video/quicktime; audio/x-m4a; application/x-3gp

file: /usr/lib/imx-mm/parser/lib_mp4_parser_arm11_elinux.so.3.2

Content Info:

URI:

file:///nfs/sample.mp4

Idx File:

/home/root/.aiur/.nfs.sample.mp4.aidx

Seekable : Yes

Size(byte): 14621544

Movie Info:

Seekable : Yes

Live : No

Duration : 0:00:52.209000000

ReadMode : Track

Track : 2

Track 00 [video_000000] Enabled

Duration: 0:00:52.208000000

Language: und

Mime:

video/x-h264, parsed=(boolean)true, width=(int)1920, height=(int)1080, framerate

=(fraction)24/1

[ERR] Error in opening firmware binary file

[ERR] Please put bin file to /lib/firmware/vpu folder or export VPU_FW_PATH env

H264D_ARM11_02.06.03 build on Apr 16 2013 13:30:59.

MFW_GST_H264_DECODER_PLUGIN 3.0.10 build on Mar 2 2014 17:33:34.

Track 01 [audio_000000] Enabled

Duration: 0:00:51.946000000

Language: und

Mime:

audio/mpeg, mpegversion=(int)4, channels=(int)2, rate=(int)48000, bitrate=(int)0

, framed=(boolean)true, stream-format=(string)raw, codec_data=(buffer)1190

ERROR: from element /GstPlayBin2:playbin20/GstURIDecodeBin:uridecodebin0/GstDecodeBin2:decodebin20/MFW_GST_H264DEC_INFO_T:mfw_gst_h264dec_info_t0: fatal error

Additional debug info:

non recoverable error while decoding the H.264 Data

ERROR: pipeline doesn't want to preroll.

Setting pipeline to NULL ...

Total rendered:0

Freeing pipeline ...

[--->FINALIZE v4l_sink

In fact, I always get the error message "Error in opening firmware binary file" and I've tried loads of different command lines with gstreamer. :-( Here are some of the environment variables I have set up:

export GST_PLUGIN_PATH="/usr/lib/gstreamer-0.10/:/usr/lib"

export LD_LIBRARY_PATH="/usr/lib/gstreamer-0.10/:/usr/lib"

export VPU_FW_PATH="/usr/lib/gstreamer-0.10/:/usr/lib:/lib/firmware/vpu"

Ralph

xizhouwang · ‎10-21-2014

do this

gst-inspect |grep mfw

to check what sink are on your system. Here is from one of my fs.

# gst-inspect |grep mfw

ipucsc.imx: mfw_ipucsc: IPU-based video converter

v4lsrc.imx: mfw_v4lsrc: v4l2 based camera src

mpeg2dec.imx: mfw_mpeg2decoder: mpeg2 video decoder

amrdec.imx: mfw_amrdecoder: amr audio decoder

aacdec.imx: mfw_aacdecoder: aac audio decoder

v4lsink.imx: mfw_v4lsink: v4l2 video sink

mpeg4dec.imx: mfw_mpeg4aspdecoder: mpeg4 video decoder

audiopeq.imx: mfw_audio_pp: audio post equalizer

h264.imx: mfw_h264decoder: h264 video decoder

isink.imx: mfw_isink: IPU-based video sink

mp3enc.imx: mfw_mp3encoder: mp3 audio encoder

vorbisdec.imx: mfw_vorbisdecoder: vorbis audio decoder

ralph__ · ‎10-21-2014

Here you go:

root@nitrogen6x:~# gst-inspect | grep mfw

ipucsc.imx: mfw_ipucsc: IPU-based video converter

v4lsrc.imx: mfw_v4lsrc: v4l2 based camera src

h264.imx: mfw_h264decoder: h264 video decoder

mp3enc.imx: mfw_mp3encoder: mp3 audio encoder

isink.imx: mfw_isink: IPU-based video sink

amrdec.imx: mfw_amrdecoder: amr audio decoder

v4lsink.imx: mfw_v4lsink: v4l2 video sink

audiopeq.imx: mfw_audio_pp: audio post equalizer

mpeg2dec.imx: mfw_mpeg2decoder: mpeg2 video decoder

mpeg4dec.imx: mfw_mpeg4aspdecoder: mpeg4 video decoder

I tried using mfw_v4lsink and mfw_mpeg4aspdecoder in various combinations but no luck.

Ralph

xizhouwang · ‎10-22-2014

This link could be helpful for your firmware ERR.

Error in opening firmware binary file for i.mx6 solo wandboard

Poor framerates on iMX6 Solo when decoding H264

Poor framerates on iMX6 Solo when decoding H264

Graphics & Display

i.MX6_All

i.MX6S

Linux

Multimedia