i.MX ProcessorsのトピックRe: i.MX6Q: VPU memory allocation "fragmentation" error

i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Fri, 22 Feb 2013 10:33:15 GMT

Hi,

I'm working on Nitrogen board, BSP version L3.0.35_1.1.0_121218_source, codecs version IMX_MMCODEC_3.0.5_Bundle and LTIB build. The application is surveillance system which displays up to 16 cameras on HDMI screen. There is no problem when I first bring up the application in the 16 way mode. All the cameras are displayed fine. When I try to switch to a different set of 16 cameras by shutting down all streams and starting new connections "new Gstreamer pipelines" I get the following ERROR from some streams. The error is due to VPU trying to allocate memory but fails to do so. I'm sure there is enough memory to handle the application requirement because the first time it works without problem. My guess it's fragmentation problem due to the nature of dynamically allocating and freeing resources.

Gstreamer pipeline: appsrc -> typefinder -> vpudec -> mfw_isink

Is there any VPU garbage collection mechanism to avoid this problem?

console: page allocation failure: order:11, mode:0xd1

[<800477f4>] (unwind_backtrace+0x0/0xf8) from [<800be9a0>] (warn_alloc_failed+0xc8/0x100)

[<800be9a0>] (warn_alloc_failed+0xc8/0x100) from [<800c0ed8>] (__alloc_pages_nodemask+0x4c8/0x6cc)

[<800c0ed8>] (__alloc_pages_nodemask+0x4c8/0x6cc) from [<8004a760>] (__dma_alloc+0xa4/0x300)

[<8004a760>] (__dma_alloc+0xa4/0x300) from [<8004af98>] (dma_alloc_coherent+0x54/0x60)

[<8004af98>] (dma_alloc_coherent+0x54/0x60) from [<803a0964>] (vpu_alloc_dma_buffer+0x2c/0x54)

[<803a0964>] (vpu_alloc_dma_buffer+0x2c/0x54) from [<803a0aac>] (vpu_ioctl+0x120/0x864)

[<803a0aac>] (vpu_ioctl+0x120/0x864) from [<800ff51c>] (do_vfs_ioctl+0x80/0x54c)

[<800ff51c>] (do_vfs_ioctl+0x80/0x54c) from [<800ffa20>] (sys_ioctl+0x38/0x5c)

[<800ffa20>] (sys_ioctl+0x38/0x5c) from [<80040f80>] (ret_fast_syscall+0x0/0x30)

Mem-info:

DMA per-cpu:

CPU 0: hi: 90, btch: 15 usd: 79

CPU 1: hi: 90, btch: 15 usd: 75

CPU 2: hi: 90, btch: 15 usd: 84

CPU 3: hi: 90, btch: 15 usd: 83

Normal per-cpu:

CPU 0: hi: 186, btch: 31 usd: 23

CPU 1: hi: 186, btch: 31 usd: 170

CPU 2: hi: 186, btch: 31 usd: 148

CPU 3: hi: 186, btch: 31 usd: 78

active_anon:14250 inactive_anon:16 isolated_anon:0

active_file:936 inactive_file:8191 isolated_file:0

unevictable:0 dirty:0 writeback:0 unstable:12

free:172332 slab_reclaimable:339 slab_unreclaimable:1799

mapped:2711 shmem:17 pagetables:192 bounce:0

DMA free:78980kB min:780kB low:972kB high:1168kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:186944kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 705 705 705

Normal free:610348kB min:3028kB low:3784kB high:4540kB active_anon:57000kB inactive_anon:64kB active_file:3744kB inactive_file:32764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:722368kB mlocked:0kB dirty:0kB writeback:0kB mapped:10844kB shmem:68kB slab_reclaimable:1356kB slab_unreclaimable:7196kB kernel_stack:1024kB pagetables:768kB unstable:48kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 103*4kB 85*8kB 86*16kB 61*32kB 37*64kB 6*128kB 1*256kB 3*512kB 10*1024kB 9*2048kB 10*4096kB 0*8192kB 0*16384kB 0*32768kB = 78980kB

Normal: 3*4kB 4*8kB 11*16kB 26*32kB 18*64kB 5*128kB 5*256kB 2*512kB 1*1024kB 1*2048kB 1*4096kB 3*8192kB 3*16384kB 16*32768kB = 610332kB

9144 total pagecache pages

0 pages in swap cache

Swap cache stats: add 0, delete 0, find 0/0

Free swap = 0kB

Total swap = 0kB

262144 pages of RAM

173240 free pages

37873 reserved pages

1116 slab pages

4788 pages shared

0 pages swap cached

Physical memory allocation error!

Re: i.MX6Q: VPU memory allocation "fragmentation" error

daiane_angolini — Tue, 26 Feb 2013 14:16:03 GMT

How did you turn your application off?

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Tue, 26 Feb 2013 20:09:14 GMT

Hi Daiane,

I'm not turning off the application. I'm terminating the gstreamer threads

and restarting new ones.

I use the following:

g_main_loop_quit (ctx->loop);

gst_element_set_state (ctx->pipeline, GST_STATE_NULL);

gst_object_unref (GST_OBJECT (ctx->pipeline));

What is the correct method of termination?

Re: i.MX6Q: VPU memory allocation "fragmentation" error

daiane_angolini — Wed, 27 Feb 2013 14:25:36 GMT

Take a look on gplay source code, how it handle the quit command. Do you know where to find it?

I feel like your memory leakage has been caused by unclear termination of vpu stuff.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Fri, 01 Mar 2013 09:00:42 GMT

I looked at the gplay source code but couldn't spot anything obvious.

I guess gplay is terminating the application on a quite command but this is a different situation. I need to clear gstreamer pipeline and reset the vpu without terminating the application.

Is there a way to inform the VPU that the stream source "compression" will change? ie. a different stream will be used as the source with a different compression (changing the decoder from mpeg4 to H.264 on the fly).

What is the gstreamer API for flushing the pipeline?

What is the gstreamer API for resetting the VPU decoder?

Thanks

Re: i.MX6Q: VPU memory allocation "fragmentation" error

daiane_angolini — Fri, 01 Mar 2013 14:02:02 GMT

I don´t know it. karinavalencia who could help with this VPU question?

Re: i.MX6Q: VPU memory allocation "fragmentation" error

timothybean — Thu, 07 Mar 2013 16:34:20 GMT

Hi Tarek,

First, you may want to go from PLAYING to PAUSED and then to NULL. That is the way I do it. I think there may be some things in Gstreamer that need to go through that transition for cleaning up resources.

You could look at the gst-fls-plugins at the vpudec plug in and see in the source what the plugin does during those transitions.

Also, I am wondering what is your video encoding, resolution, and frame rate for you to do 16 at once? Are these live streams or from files?

Thanks

Tim

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Thu, 07 Mar 2013 17:54:38 GMT

Hi Tim,

It's life stream using mpeg low res.

Thanks,

Tarek

Re: i.MX6Q: VPU memory allocation "fragmentation" error

ieio — Tue, 30 Apr 2013 07:27:47 GMT

Hi all,

I am experiencing the exact same problem on a different board based on imx6s.

Have u solved this problem?

Regards.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Tue, 30 Apr 2013 14:14:39 GMT

Hi ieio,

No not really. I changed my strategy, now I'm allocating all the VPU decoders I may need at start up so I don't need to allocate/deallocate dynamically.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

ieio — Tue, 30 Apr 2013 14:49:24 GMT

Hi Tarek,

may I ask how many VPU decoders are u using? Is each VPU decoder used just once in the pipeline or dynamically linked to several pipeline? I tried to use a limited number or VPU decoders, but I experienced problems dynamically changing the pipeline. My input is an mpeg stream as well and I need to change stream dynamically.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Tue, 30 Apr 2013 15:12:21 GMT

Hi ieio,

I have 16 VPU decoders for 16 pipelines. So each pipeline is using one VPU. I couldn't changed dynamically so I did it this way.

What exactly are you changing?

Re: i.MX6Q: VPU memory allocation "fragmentation" error

ieio — Tue, 30 Apr 2013 16:23:10 GMT

Hi Tarek,

I created a simple loop that switch from one stream to another, I unref a pipeline and I create a new one each time I switch from one flow to the another.

May I ask how u select which pipeline should be displayed in case you are running them concurrently?

thanks,

Re: i.MX6Q: VPU memory allocation "fragmentation" error

ieio — Tue, 30 Apr 2013 16:28:20 GMT

Hi Daiane,

pls point me to the source code of gplay.

Thanks in advance,

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Tue, 30 Apr 2013 19:18:16 GMT

Hi ieio,

You can set the pipeline state to NULL if you don't want it to display.

Also you can set the pipeline state to NULL change the stream then set it to PLAYING. For example if your source is filesrc, set state to NULL change the filesrc property "location" to the new file then set it to PLAYING.

This way you can avoid the need for multiple pipelines.

In my case I had to have 16 pipelines because the program may display the 16 at the same time.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

yossishukron — Wed, 01 May 2013 22:10:03 GMT

hello Tarek

Can you create a simple test program that communicate directly with the VPU driver. you can based your program on the VPU_TEST that we supply together with our BSP.

in this simple program you can try to allocate and Free VPU Memory and see if you are getting error after many iterations.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

ieio — Thu, 02 May 2013 09:57:30 GMT

Thanks Tarek,

your suggestion works.

Do you run some pipelines in parallel?

I am now using just a pipeline and I put it in PAUSED and then NULL state and then I change the "location" property before PLAYING again.

Anyway if I run 2 instances of my code I run in the same problem.

q1 and q2 are two processes running in parallel, and using different portions of screen.

q1: page allocation failure: order:11, mode:0xd1
[<8003d0dc>] (unwind_backtrace+0x0/0xfc) from [<800b5c5c>] (warn_alloc_failed+0x9c/0x118)
[<800b5c5c>] (warn_alloc_failed+0x9c/0x118) from [<800b6948>] (__alloc_pages_nodemask+0x494/0x6ec)
[<800b6948>] (__alloc_pages_nodemask+0x494/0x6ec) from [<80040434>] (__dma_alloc+0xd4/0x2fc)
[<80040434>] (__dma_alloc+0xd4/0x2fc) from [<80040730>] (dma_alloc_coherent+0x54/0x60)
[<80040730>] (dma_alloc_coherent+0x54/0x60) from [<803b3ba0>] (vpu_alloc_dma_buffer+0x2c/0x64)
[<803b3ba0>] (vpu_alloc_dma_buffer+0x2c/0x64) from [<803b439c>] (vpu_ioctl+0x7c4/0x8c8)
[<803b439c>] (vpu_ioctl+0x7c4/0x8c8) from [<800f69a0>] (do_vfs_ioctl+0x80/0x5e0)
[<800f69a0>] (do_vfs_ioctl+0x80/0x5e0) from [<800f6f38>] (sys_ioctl+0x38/0x60)
[<800f6f38>] (sys_ioctl+0x38/0x60) from [<8003769c>] (__sys_trace_return+0x0/0x24)
Mem-info:
DMA per-cpu:
CPU 0: hi: 90, btch: 15 usd: 88
Normal per-cpu:
CPU 0: hi: 6, btch: 1 usd: 0
active_anon:1139 inactive_anon:9 isolated_anon:0
active_file:3 inactive_file:94 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:46144 slab_reclaimable:274 slab_unreclaimable:1563
mapped:103 shmem:14 pagetables:57 bounce:0
DMA free:160012kB min:1564kB low:1952kB high:2344kB active_anon:948kB inactive_anon:0kB active_file:0kB inactive_file:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:186944kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:60kB slab_unreclaimable:140kB kernel_stack:96kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:93 all_unreclaimable? yes
lowmem_reserve[]: 0 39 39 39
Normal free:24368kB min:336kB low:420kB high:504kB active_anon:3608kB inactive_anon:36kB active_file:12kB inactive_file:484kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:40384kB mlocked:0kB dirty:0kB writeback:0kB mapped:432kB shmem:56kB slab_reclaimable:1036kB slab_unreclaimable:6128kB kernel_stack:328kB pagetables:224kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 191*4kB 182*8kB 158*16kB 102*32kB 89*64kB 55*128kB 32*256kB 16*512kB 22*1024kB 13*2048kB 16*4096kB 1*8192kB 0*16384kB 0*32768kB = 160012kB
Normal: 502*4kB 581*8kB 416*16kB 207*32kB 54*64kB 5*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 24032kB
315 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
65536 pages of RAM
46031 free pages
11564 reserved pages
1283 slab pages
270 pages shared
0 pages swap cached
Physical memory allocation error!
Physical memory allocation error!

I have strace logs:

hwbuf allocator zone(614400) destroied.

hwbuf allocator zone(614400) created

hwbuf allocator zone(614400) destroied.

hwbuf allocator zone(614400) created

hwbuf allocator zone(614400) destroied.

[1;34mVS1 destroyed, force=0!

[0mhwbuf allocator zone(462848) destroied.

[INFO] Product Info: i.MX6Q/D/S

[1;32mvpudec versions :smileyhappy:

[0m) = 669

write(1, "\33[1;32m\tplugin: 3.0.5\n\33[0m", 26 [1;32m plugin: 3.0.5

[0m) = 26

write(1, "\33[1;32m\twrapper: 1.0.28(VPUWRAPP"..., 80 [1;32m wrapper: 1.0.28(VPUWRAPPER_ARM_LINUX Build on Apr 12 2013 17:28:04)

[0m) = 80

write(1, "\33[1;32m\tvpulib: 5.4.10\n\33[0m", 27 [1;32m vpulib: 5.4.10

[0m) = 27

write(1, "\33[1;32m\tfirmware: 2.1.8.34588\n\33["..., 34 [1;32m firmware: 2.1.8.34588

[0m) = 34

ioctl(3, VIDIOC_QUERYCAP or VT_OPENQRY, 0x7ec0a878) = -1 EPERM (Operation not permitted)

write(2, "Unable to set the pipeline to th"..., 49Unable to set the pipeline to the playing state.

) = 49

write(1, "[ERR]\tmem allocation failed!\n", 29[ERR] mem allocation failed!

) = 29

it seems that the ioctl VIDIOC_QUERYCAP or VT_OPENQRY fails.

When it works correctly the log is:

hwbuf allocator zone(614400) destroied.

hwbuf allocator zone(614400) created

hwbuf allocator zone(614400) destroied.

hwbuf allocator zone(614400) created

hwbuf allocator zone(614400) destroied.

[1;34mVS1 destroyed, force=0!

[0mhwbuf allocator zone(462848) destroied.

[INFO] Product Info: i.MX6Q/D/S

[1;32mvpudec versions :smileyhappy:

[0m) = 669

write(1, "\33[1;32m\tplugin: 3.0.5\n\33[0m", 26 [1;32m plugin: 3.0.5

[0m) = 26

write(1, "\33[1;32m\twrapper: 1.0.28(VPUWRAPP"..., 80 [1;32m wrapper: 1.0.28(VPUWRAPPER_ARM_LINUX Build on Apr 12 2013 17:28:04)

[0m) = 80

write(1, "\33[1;32m\tvpulib: 5.4.10\n\33[0m", 27 [1;32m vpulib: 5.4.10

[0m) = 27

write(1, "\33[1;32m\tfirmware: 2.1.8.34588\n\33["..., 34 [1;32m firmware: 2.1.8.34588

[0m) = 34

ioctl(3, VIDIOC_QUERYCAP or VT_OPENQRY, 0x7ec0a878) = 0

mmap2(NULL, 5236743, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x19000) = 0x2b414000

SYS_288(0x1, 0x1, 0, 0x7ec0a860, 0x194c8) = 0

fcntl64(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0

fcntl64(8, F_SETFL, O_RDONLY|O_NONBLOCK) = 0

timer_delete(0x1) = 0

mq_notify(16, ptrace: umoven: Input/output error

{...}) = 9

mq_getsetattr(9, {mq_flags=O_RDONLY|0x10, mq_maxmsg=0, mq_msgsize=0, mq_curmsg=12}, ptrace: umoven: Input/output error

{...}) = 0

SYS_286(0x9, 0x7ec0a658, 0x7ec0a664, 0x10, 0x9) = 0

gettimeofday({265634, 766627}, NULL) = 0

SYS_290(0x9, 0x7ec0a600, 0x14, 0, 0x7ec0a614) = 20

SYS_297(0x9, 0x7ec0a5e4, 0, 0, 0x7ec0a796) = 108

SYS_297(0x9, 0x7ec0a5e4, 0, 0, 0x2) = 20

close(9) = 0

mq_notify(16, ptrace: umoven: Input/output error

{...}) = 9

mq_getsetattr(9, {mq_flags=O_RDONLY|0x10, mq_maxmsg=0, mq_msgsize=0, mq_curmsg=12}, ptrace: umoven: Input/output error

{...}) = 0

SYS_286(0x9, 0x7ec0a628, 0x7ec0a634, 0x10, 0x9) = 0

Yossi I think the interaction that fails with the VPU is ioctl(3, VIDIOC_QUERYCAP or VT_OPENQRY,

where fd 3 is /dev/mxc_vpu

lrwx------ 1 root root 64 Jan 4 02:42 3 -> /dev/mxc_vpu

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Thu, 02 May 2013 10:32:02 GMT

Hi ieio,

I have 32 concurrent pipelines but only 16 can be active at the same time.

Try first to use the gst-launch command line. run 2 or more pipelines at the same time and see if that's any good.

Re: i.MX6Q: VPU memory allocation "fragmentation" error

ieio — Thu, 02 May 2013 11:29:57 GMT

I can run 4 pipelines at the same time using gst-launch, but I do not stop them and I do not change the src property each 3 second.

Are u running the concurrent pipelines in the same process? are u using different threads?

Thanks,

Re: i.MX6Q: VPU memory allocation "fragmentation" error

Tarek — Thu, 02 May 2013 13:19:01 GMT

I don't think it's related to threads/process. Try to increase the 3 seconds. Make it 3 minutes or so. I think it takes much more than 3 seconds to start a single stream.