i.MX6 VPU mem allocation failed

[davidthompson]

I'm looking for information on the correct way to reserve memory for the VPU to use for decoding. We are encountering errors where, while the board is under heavy load, video decoding cannot be started. We see the following error message:

[ERR] mem allocation failed!

What we're attempting to do, is test performance of video, while other processes are also using a lot of system resources. We start a video decoding, and everything is working well. Then we launch a separate process that consumes a lot of memory and cpu. Video decode continues to work correctly though. When the first video has completed, we attempt to launch another video. However, this time, it appears that the VPU cannot allocate enough memory for decoding, and fails immediately.

I have found similar issues posted here from the last year or two, although I haven't seen any definitive fix. Typically, the fix involves editting the board file and changing various reserved memory values, or changing kernel parameters. I believe I have tried nearly all of these fixes, but I haven't had much success. Is there a specific option to guarantee that certain memory will only be used for the VPU?

Here is a simplified version of the gstreamer pipeline I'm using to launch the decode.

gst-launch filesrc location=/home/root/test.h264 ! vpudec ! fakesink

The text of the kernel error is below

VPU interrupt received.

VPU interrupt received.

source-queue:sr: page allocation failure: order:11, mode:0xd1

[<8004a304>] (unwind_backtrace+0x0/0xf4) from [<800e8268>] (warn_alloc_failed+0xd4/0x10c)

[<800e8268>] (warn_alloc_failed+0xd4/0x10c) from [<800eb01c>] (__alloc_pages_nodemask+0x5f4/0x78c)

[<800eb01c>] (__alloc_pages_nodemask+0x5f4/0x78c) from [<8004d018>] (__dma_alloc+0x9c/0x2fc)

[<8004d018>] (__dma_alloc+0x9c/0x2fc) from [<8004d5b0>] (dma_alloc_coherent+0x60/0x68)

[<8004d5b0>] (dma_alloc_coherent+0x60/0x68) from [<803ecc7c>] (vpu_alloc_dma_buffer+0x2c/0x54)

[<803ecc7c>] (vpu_alloc_dma_buffer+0x2c/0x54) from [<803ed0cc>] (vpu_ioctl+0x428/0x8c4)

[<803ed0cc>] (vpu_ioctl+0x428/0x8c4) from [<801301bc>] (do_vfs_ioctl+0x3b4/0x530)

[<801301bc>] (do_vfs_ioctl+0x3b4/0x530) from [<8013036c>] (sys_ioctl+0x34/0x60)

[<8013036c>] (sys_ioctl+0x34/0x60) from [<80043840>] (ret_fast_syscall+0x0/0x30)

Mem-info:

DMA per-cpu:

CPU    0: hi:   90, btch:  15 usd:  14

CPU    1: hi:   90, btch:  15 usd:   4

CPU    2: hi:   90, btch:  15 usd:  75

CPU    3: hi:   90, btch:  15 usd:   0

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:  58

CPU    1: hi:  186, btch:  31 usd:  30

CPU    2: hi:  186, btch:  31 usd: 163

CPU    3: hi:  186, btch:  31 usd:   0

active_anon:5222 inactive_anon:2012 isolated_anon:0

active_file:32654 inactive_file:118028 isolated_file:0

unevictable:0 dirty:4232 writeback:19 unstable:0

free:19417 slab_reclaimable:2524 slab_unreclaimable:1995

mapped:6132 shmem:2104 pagetables:441 bounce:0

DMA free:93640kB min:844kB low:1052kB high:1264kB active_anon:4992kB inactive_anon:2704kB active_file:428kB inactive_file:50252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:186944kB mlocked:0kB dirty:6004kB writeback:0kB mapped:2888kB shmem:2812kB slab_reclaimable:1380kB slab_unreclaimable:96kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no

lowmem_reserve[]: 0 577 577 577

Normal free:3288kB min:2680kB low:3348kB high:4020kB active_anon:18196kB inactive_anon:5344kB active_file:130188kB inactive_file:400232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:591296kB mlocked:0kB dirty:11892kB writeback:84kB mapped:21640kB shmem:5604kB slab_reclaimable:8716kB slab_unreclaimable:7884kB kernel_stack:976kB pagetables:1764kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:26 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1773*4kB 1674*8kB 1224*16kB 818*32kB 444*64kB 144*128kB 50*256kB 8*512kB 3*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 133060kB

Normal: 48*4kB 2563*8kB 297*16kB 36*32kB 16*64kB 9*128kB 7*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 30568kB

125865 total pagecache pages

0 pages in swap cache

Swap cache stats: add 0, delete 0, find 0/0

Free swap  = 0kB

Total swap = 0kB

262144 pages of RAM

57374 free pages

71094 reserved pages

2797 slab pages

56553 pages shared

0 pages swap cached

Physical memory allocation error!

Physical memory allocation error!

Kernel: linux-imx 3.0.35-4.1.0 yocto

Board: imx6q Sabre SD

Thanks for any advice

David

[dmitriykornoush]

HI, have you solve problem?  I have same. Pls share solution

[dilshad_alam]

Hi , You need to migrate to new Kernel 3.14.52 and new user space GPU drivers compatible with that, This has fixed the issue.

[dilshad_alam]

SunnySun​

We are facing similar issue, I tried opening the Link but unable to open.

iMX6 VPU: Use a DMA allocation pool, instead of kernel default allocator - Gitorious

Could you please send me on my email ID: dilshad.alam@harman.com

Thanks,

Dilshad

[marcomadrigal]

Hi,

Not sure if this can help you, I haven't tried it yet but I am facing the same issue:

iMX6 VPU: Use a DMA allocation pool, instead of kernel default allocator - Gitorious

Regards,

Marco

[jimheath]

Marco,

I recompiled the kernel with that patch: iMX6 VPU: Use a DMA allocation pool, instead of kernel default allocator - Gitorious and I'm still seeing the memory allocation errors. Looking at the out of dmesg, I'm seeing new errors I haven't seen before:

o free invalid coherent area:  (null)

[<800a2e04>] (unwind_backtrace+0x0/0xf4) from [<800a65a8>] (dma_free_coherent+0x1a0/0x1d0)

[<800a65a8>] (dma_free_coherent+0x1a0/0x1d0) from [<803fb0b0>] (vpu_release+0x200/0x2a0)

[<803fb0b0>] (vpu_release+0x200/0x2a0) from [<80152a3c>] (fput+0xb8/0x1e4)

[<80152a3c>] (fput+0xb8/0x1e4) from [<8014f35c>] (filp_close+0x64/0x88)

[<8014f35c>] (filp_close+0x64/0x88) from [<8014f424>] (sys_close+0xa4/0xdc)

[<8014f424>] (sys_close+0xa4/0xdc) from [<8009bf00>] (ret_fast_syscall+0x0/0x30)

__dma_free_remap: trying to free invalid coherent area:  (null)

[<800a2e04>] (unwind_backtrace+0x0/0xf4) from [<800a65a8>] (dma_free_coherent+0x1a0/0x1d0)

[<800a65a8>] (dma_free_coherent+0x1a0/0x1d0) from [<803fb0b0>] (vpu_release+0x200/0x2a0)

[<803fb0b0>] (vpu_release+0x200/0x2a0) from [<80152a3c>] (fput+0xb8/0x1e4)

[<80152a3c>] (fput+0xb8/0x1e4) from [<8014f35c>] (filp_close+0x64/0x88)

[<8014f35c>] (filp_close+0x64/0x88) from [<8014f424>] (sys_close+0xa4/0xdc)

[<8014f424>] (sys_close+0xa4/0xdc) from [<8009bf00>] (ret_fast_syscall+0x0/0x30)

__dma_free_remap: trying to free invalid coherent area:  (null)

[<800a2e04>] (unwind_backtrace+0x0/0xf4) from [<800a65a8>] (dma_free_coherent+0x1a0/0x1d0)

[<800a65a8>] (dma_free_coherent+0x1a0/0x1d0) from [<803fb0b0>] (vpu_release+0x200/0x2a0)

[<803fb0b0>] (vpu_release+0x200/0x2a0) from [<80152a3c>] (fput+0xb8/0x1e4)

[<80152a3c>] (fput+0xb8/0x1e4) from [<8014f35c>] (filp_close+0x64/0x88)

[<8014f35c>] (filp_close+0x64/0x88) from [<8014f424>] (sys_close+0xa4/0xdc)

[<8014f424>] (sys_close+0xa4/0xdc) from [<8009bf00>] (ret_fast_syscall+0x0/0x30)

I was really hoping this patch would work, the description seemed promising. Anybody else have ideas?

Thanks

Jim

[gavinbao]

I have the same question,you have slove this question?

[jimheath]

Hi Marco,

Thanks for your response, I'll try this out and let you know...

Jim

[jimheath]

David or Michael, have either of you resolved this issue? We're still having the memory allocation error and as I posted earlier, the drop caches command is a workaround but we're now having instances where even this doesn't help and have to resort to restarting the device.

If not, does anyone else have a solution to this problem?

Thanks,

Jim

[Tarek]

Hi Jim,

Are you using mfw_isink ?

Thanks

[jimheath]

I'm using Gstreamer 0.10.36 and the playbin2 element. I think it uses mfw_v4lsink. Should I be using mfw_isink? If so, how can I do this?

Thanks,

Jim

[Tarek]

Hi Jim,

In my case I found that mfw_isink is allocating too many HW buffer so I wanted to know if you have the same problem. playbin2 will use mfw_v4lsink. I'm not sure if mfw_v4lsink will allocate the same amount of buffer or not but I don't think in your case you should switch to isink.

thanks,

Tarek

[michaelb_]

Seems, I'm experiencing the same problem :-(

Playing videos via gstreamer via qt5, I get this error after some several hours:

Apr 24 17:47:00 cubox-i user.warn kernel: alloc_contig_range test_pages_isolated(7d800, 7dcff) failed

Apr 24 17:47:00 cubox-i user.warn kernel: alloc_contig_range test_pages_isolated(7d800, 7ddff) failed

Apr 24 17:47:00 cubox-i user.warn kernel: alloc_contig_range test_pages_isolated(7d800, 7deff) failed

Apr 24 17:47:00 cubox-i user.warn kernel: alloc_contig_range test_pages_isolated(7d800, 7dfff) failed

Apr 24 17:47:00 cubox-i user.err kernel: mxc_vpu 2040000.vpu: Physical memory allocation error!

Apr 24 17:47:00 cubox-i user.err kernel: mxc_vpu 2040000.vpu: Physical memory allocation error!

Apr 24 17:47:01 cubox-i user.warn kernel: Alignment trap: multiqueue120:s (1424) PC=0x6764afc8 Instr=0xe8810011 Address=0x00002f0f FSR 0x801

Qt5 then gets a segmentation fault.

[ERR]    mem allocation failed!

395 Segmentation fault

System is:

cubox-i4pro, kernel 3.10.30, yocto "master-next"

vpudec versions :smileyhappy:

    plugin: 3.0.10

    wrapper: 1.0.45(VPUWRAPPER_ARM_LINUX Build on Apr 15 2014 09:52:34)

    vpulib: 5.4.20

    firmware: 2.3.10.40778

I will try next to monitor memory usage and maybe try the idea of issuing "echo 3 > /proc/sys/vm/drop_caches" on a regular basis.

[michaelb_]

Hmm, "echo 3 > /proc/sys/vm/drop_caches" doesn't help.

I have a long running encode job working (encoding stream from webcam). This seems to be (at least part of) the problem.

I can only "reactivate" my vpu memory, when this happens, by killing all vpu involved processes and restarting them.

[Tarek]

Hi Michael,

Try to reduce the amount of memory allocated by VPUDEC by setting the "frame-plus" property to 2.

[jimheath]

Hi David,

You've probably seen my thread (GStreamer crashing on i.MX6 (Boundary Devices Nitrogen6x) describing this same issue. We're still experiencing the VPU allocation failure. The workaround that I've been using is; whenever the video fails to play, I execute "echo 3 > /proc/sys/vm/drop_caches" to drop all caches. This frees up enough contiguous memory to allow the video to begin playing.

I don't like having to rely on workarounds but that's been the best way to keep our software consistently playing videos. Hopefully this problem is resolved in the kernel soon...

Jim

[Danube]

Hi David,

Could you please share your test.h264 file to me for test again ?

[davidthompson]

That file, and that gstreamer pipeline, were just used as an example of the minimal pipeline needed to reproduce the error. This removes the need to include demuxers, audio decoders, and sinks. This problem appears to exist with MPEG videos as well, not just h.264. There isn't really a 'test.h264' file, but I can reproduce the problem using any media file I have access to. I will attach one for reference.

Here is the pipeline that I use to test the video, although through additional testing, I've definitively narrowed the cause of my issue down to the vpudec plugin.

gst-launch filesrc location=/home/root/IPTV-cnbc.ts ! tsdemux name=demux demux. ! queue max-size-buffers=0 max-size-time=0 ! vpudec ! mfw_v4lsink demux. ! queue max-size-buffers=0 max-size-time=0 ! a52dec ! audioconvert ! alsasink

Hope that is useful to you.

David