i.MX6Q: VPU memory allocation "fragmentation" error

[Tarek]

Hi,

I'm working on Nitrogen board, BSP version L3.0.35_1.1.0_121218_source, codecs version IMX_MMCODEC_3.0.5_Bundle and LTIB build.  The application is surveillance system which displays up to 16 cameras  on HDMI screen. There is no problem when I first bring up the application in the 16 way mode. All the cameras are displayed fine. When I try to switch to a different set of 16 cameras by shutting down all streams and starting new connections "new Gstreamer pipelines" I get the following ERROR from some streams. The error is due to VPU trying to allocate memory but fails to do so. I'm sure there is enough memory to handle the application requirement because the first time it works without problem. My guess it's fragmentation problem due to the nature of dynamically allocating  and freeing resources.

Gstreamer pipeline: appsrc -> typefinder -> vpudec -> mfw_isink

Is there any VPU garbage collection mechanism to avoid this problem?

console: page allocation failure: order:11, mode:0xd1

[<800477f4>] (unwind_backtrace+0x0/0xf8) from [<800be9a0>] (warn_alloc_failed+0xc8/0x100)

[<800be9a0>] (warn_alloc_failed+0xc8/0x100) from [<800c0ed8>] (__alloc_pages_nodemask+0x4c8/0x6cc)

[<800c0ed8>] (__alloc_pages_nodemask+0x4c8/0x6cc) from [<8004a760>] (__dma_alloc+0xa4/0x300)

[<8004a760>] (__dma_alloc+0xa4/0x300) from [<8004af98>] (dma_alloc_coherent+0x54/0x60)

[<8004af98>] (dma_alloc_coherent+0x54/0x60) from [<803a0964>] (vpu_alloc_dma_buffer+0x2c/0x54)

[<803a0964>] (vpu_alloc_dma_buffer+0x2c/0x54) from [<803a0aac>] (vpu_ioctl+0x120/0x864)

[<803a0aac>] (vpu_ioctl+0x120/0x864) from [<800ff51c>] (do_vfs_ioctl+0x80/0x54c)

[<800ff51c>] (do_vfs_ioctl+0x80/0x54c) from [<800ffa20>] (sys_ioctl+0x38/0x5c)

[<800ffa20>] (sys_ioctl+0x38/0x5c) from [<80040f80>] (ret_fast_syscall+0x0/0x30)

Mem-info:

DMA per-cpu:

CPU    0: hi:   90, btch:  15 usd:  79

CPU    1: hi:   90, btch:  15 usd:  75

CPU    2: hi:   90, btch:  15 usd:  84

CPU    3: hi:   90, btch:  15 usd:  83

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:  23

CPU    1: hi:  186, btch:  31 usd: 170

CPU    2: hi:  186, btch:  31 usd: 148

CPU    3: hi:  186, btch:  31 usd:  78

active_anon:14250 inactive_anon:16 isolated_anon:0

active_file:936 inactive_file:8191 isolated_file:0

unevictable:0 dirty:0 writeback:0 unstable:12

free:172332 slab_reclaimable:339 slab_unreclaimable:1799

mapped:2711 shmem:17 pagetables:192 bounce:0

DMA free:78980kB min:780kB low:972kB high:1168kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:186944kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 705 705 705

Normal free:610348kB min:3028kB low:3784kB high:4540kB active_anon:57000kB inactive_anon:64kB active_file:3744kB inactive_file:32764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:722368kB mlocked:0kB dirty:0kB writeback:0kB mapped:10844kB shmem:68kB slab_reclaimable:1356kB slab_unreclaimable:7196kB kernel_stack:1024kB pagetables:768kB unstable:48kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 103*4kB 85*8kB 86*16kB 61*32kB 37*64kB 6*128kB 1*256kB 3*512kB 10*1024kB 9*2048kB 10*4096kB 0*8192kB 0*16384kB 0*32768kB = 78980kB

Normal: 3*4kB 4*8kB 11*16kB 26*32kB 18*64kB 5*128kB 5*256kB 2*512kB 1*1024kB 1*2048kB 1*4096kB 3*8192kB 3*16384kB 16*32768kB = 610332kB

9144 total pagecache pages

0 pages in swap cache

Swap cache stats: add 0, delete 0, find 0/0

Free swap  = 0kB

Total swap = 0kB

262144 pages of RAM

173240 free pages

37873 reserved pages

1116 slab pages

4788 pages shared

0 pages swap cached

Physical memory allocation error!

Physical memory allocation error!

[yossishukron]

hello Tarek

Can you create a simple test program that communicate directly with the VPU driver. you can based your program on the VPU_TEST that we supply together with our BSP.

in this simple program you can try to allocate and Free VPU Memory and see if you are getting error after many iterations.

[daiane_angolini]

How did you turn your application off?

[Tarek]

Hi Daiane,

I'm not turning off the application. I'm terminating the gstreamer threads

and restarting new ones.

I use the following:

g_main_loop_quit (ctx->loop);

gst_element_set_state (ctx->pipeline, GST_STATE_NULL);

gst_object_unref (GST_OBJECT (ctx->pipeline));

What is the correct method of termination?

[timothybean]

Hi Tarek,

First, you may want to go from PLAYING to PAUSED and then to NULL. That is the way I do it. I think there may be some things in Gstreamer that need to go through that transition for cleaning up resources.

You could look at the gst-fls-plugins at the vpudec plug in and see in the source what the plugin does during those transitions.

Also, I am wondering what is your video encoding, resolution, and frame rate for you to do 16 at once? Are these live streams or from files?

Thanks

Tim

[Tarek]

Hi Tim,

It's life stream using mpeg low res.

Thanks,

Tarek

[daiane_angolini]

Take a look on gplay source code, how it handle the quit command. Do you know where to find it?

I feel like your memory leakage has been caused by unclear termination of vpu stuff.

[ieio]

Hi Daiane,

pls point me to the source code of gplay.

Thanks in advance,

i.

[daiane_angolini]

gplay source code is inside gst-fsl-plugin tar ball :smileywink:

[Tarek]

I looked at the gplay source code but couldn't spot anything obvious.

I guess gplay is terminating the application on a quite command but this is a different situation. I need to clear  gstreamer pipeline and reset the vpu without terminating the application.

Is there a way to inform the VPU that the stream source "compression" will change? ie. a different stream will be used as the source with a different compression (changing the decoder from mpeg4 to H.264 on the fly).

What is the gstreamer API for flushing the pipeline?

What is the gstreamer API for resetting the VPU decoder?

Thanks

[daiane_angolini]

I don´t know it. karinavalencia who could help with this VPU question?