P4080 Cannot allocate large buffers of 768 MB with contiguous memory

nmeriane · ‎05-30-2022

Hello folks,

I'm porting an application from QNX to Linux on a customized QorIQ P4080 revision 3 board. And, I'm using Dunfell Yocto release.

I'm having trouble finding the right way to allocate two 768MB buffers with contiguous memory. Under QNX such an allocation is easily done and works fine, but under Linux this allocation fails.

Here are the two ways I used under Linux to configure the two buffers required by the application:
1. reserve a CMA memory size at kernel boot so that I can use it later by the application. I tried to allocate the CMA using one of the following three methods:

a) command line argument: cma=1536MB
b) adding the cma allocation in the device tree
c) the CONFIG_CMA_SIZE_MBYTES parameter in the .config file.

No one of these three methods describe above worked. I even tried to allocate cma with a much little size cma=16MB.

2. setting the FORCE_MAX_ZONE_ORDER parameter to "17" as follow:
range 17 17 if PPC64 && !PPC_64K_PAGES
default "17" if PPC64 && !PPC_64K_PAGES

According to the explanation in arch/powerpc/Kconfig:809, a value of 17 means that the largest free memory bloc is 2^(17-1).

Any help would be very much appreciated.

Details follow below:

Allocation of a buffer of 256MB fails:
root@p4080ds:~/tmp#insmod u-dma-buf.ko udmabuf0=268435456

Log:
[ 144.908948] u_dma_buf: loading out-of-tree module taints kernel.
[ 144.930262] insmod: page allocation failure: order:16, mode:0xcc4(GFP_KERNEL|GFP_DMA32), nodemask=(null),cpuset=/,mems_allowed=0
[ 144.930276] CPU: 1 PID: 942 Comm: insmod Tainted: G O 5.4.3+g134788b16485 #1
[ 144.930278] Call Trace:
[ 144.930287] [d17cf858] [c0aa105c] dump_stack+0x90/0xc4 (unreliable)
[ 144.930293] [d17cf878] [c01ee8a8] warn_alloc+0x128/0x1a0
[ 144.930297] [d17cf8d8] [c01ef900] __alloc_pages_nodemask+0xfe0/0x1050
[ 144.930305] [d17cf9c8] [c00ddf44] __dma_direct_alloc_pages+0xb4/0x210
[ 144.930310] [d17cf9e8] [c00de0d4] dma_direct_alloc_pages+0x34/0x1f0
[ 144.930314] [d17cfa18] [c00dca24] dma_alloc_attrs+0x94/0x130
[ 144.930328] [d17cfa58] [fbd45ca0] udmabuf_platform_driver_probe+0x5e0/0xbd0 [u_dma_buf]
[ 144.930336] [d17cfab8] [c05e6fc0] platform_drv_probe+0x60/0xd0
[ 144.930340] [d17cfad8] [c05e414c] really_probe+0x10c/0x3b0
[ 144.930344] [d17cfb08] [c05e4618] driver_probe_device+0x68/0x1d0
[ 144.930347] [d17cfb28] [c05e1b9c] bus_for_each_drv+0x8c/0x100
[ 144.930351] [d17cfb58] [c05e3fc0] __device_attach+0x110/0x180
[ 144.930355] [d17cfb88] [c05e2e84] bus_probe_device+0xc4/0xe0
[ 144.930363] [d17cfba8] [c05de864] device_add+0x424/0x770
[ 144.930367] [d17cfc08] [c05e6c1c] platform_device_add+0x13c/0x320
[ 144.930373] [d17cfc48] [fbd442f4] udmabuf_platform_device_create+0x1b4/0x330 [u_dma_buf]
[ 144.930379] [d17cfce8] [fbd4b2cc] u_dma_buf_init+0x2cc/0x4f8 [u_dma_buf]
[ 144.930385] [d17cfd18] [c0002d68] do_one_initcall+0x58/0x258
[ 144.930392] [d17cfd88] [c010c650] do_init_module+0x60/0x250
[ 144.930396] [d17cfdb8] [c010eaec] load_module+0x21ac/0x27b0
[ 144.930401] [d17cfea8] [c010f360] sys_finit_module+0xe0/0x110
[ 144.930407] [d17cff38] [c00131d8] ret_from_syscall+0x0/0x38
[ 144.930411] --- interrupt: c01 at 0xfec9cc0
LR = 0x10014abc
[ 144.930413] Mem-Info:
[ 144.930422] active_anon:1009 inactive_anon:29 isolated_anon:0
active_file:914 inactive_file:535 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:1634 slab_unreclaimable:3456
mapped:868 shmem:66 pagetables:74 bounce:0
free:991880 free_pcp:424 free_cma:0
[ 144.930428] Node 0 active_anon:4036kB inactive_anon:116kB active_file:3656kB inactive_file:2140kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3472kB dirty:0kB writeback:0kB shmem:264kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 144.930436] Normal free:629164kB min:3252kB low:4064kB high:4876kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:786432kB managed:662836kB mlocked:0kB kernel_stack:872kB pagetables:296kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 144.930437] lowmem_reserve[]: 0 26207 26207
[ 144.930447] HighMem free:3338356kB min:512kB low:4628kB high:8744kB active_anon:3364kB inactive_anon:116kB active_file:3600kB inactive_file:2420kB unevictable:0kB writepending:0kB present:3407872kB managed:3354620kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:1696kB local_pcp:0kB free_cma:0kB
[ 144.930448] lowmem_reserve[]: 0 0 0
[ 144.930451] Normal: 328*4kB (UME) 231*8kB (UME) 152*16kB (UME) 92*32kB (UME) 45*64kB (UME) 13*128kB (UM) 4*256kB (ME) 2*512kB (M) 4*1024kB (ME) 2*2048kB (UM) 2*4096kB (M) 3*8192kB (UME) 5*16384kB (UME) 1*32768kB (E) 3*65536kB (UME) 2*131072kB (M) 0*262144kB = 629528kB
[ 144.930480] HighMem: 363*4kB (UM) 383*8kB (UM) 212*16kB (UM) 108*32kB (UM) 39*64kB (UM) 17*128kB (UM) 10*256kB (UM) 1*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB 1*8192kB (U) 2*16384kB (UM) 2*32768kB (UM) 1*65536kB (U) 2*131072kB (UM) 11*262144kB (M) = 3338916kB
[ 144.930509] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=4096kB
[ 144.930511] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16384kB
[ 144.930514] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=65536kB
[ 144.930517] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=262144kB
[ 144.930519] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 144.930521] 1490 total pagecache pages
[ 144.930523] 0 pages in swap cache
[ 144.930525] Swap cache stats: add 0, delete 0, find 0/0
[ 144.930527] Free swap = 0kB
[ 144.930528] Total swap = 0kB
[ 144.930529] 1048576 pages RAM
[ 144.930531] 851968 pages HighMem/MovableOnly
[ 144.930532] 44212 pages reserved
[ 144.930534] 0 pages cma reserved
[ 144.930537] dma_alloc_coherent(size=268435456) failed. return(0)
[ 144.930542] u-dma-buf u-dma-buf.0: driver setup failed. return=-12
[ 144.930697] u-dma-buf u-dma-buf.0: driver installed.
[ 144.930706] u-dma-buf: probe of u-dma-buf.0 failed with error -12