P4040 (e500-mc core) Kernel memory zone

amrutkulkarni · ‎06-30-2014

Hi All,

We have P4040 Processor which has e500-mc core with 4GB of DDR3 RAM. I have some questions regarding the kernel memory zone

According to my understanding kernel divides memory into following different zones :

1. DMA ZONE : Permanently mapped into the kernel address space (limited Space)

2. NORMAL ZONE : Permanently mapped into the kernel address space (upto 896MB )

3. HIGHMEM ZONE : not permanently mapped into the kernel's address space (above 896MB)

SDK1.1 source with 3.0.6 kernel.

When i look at the buddyinfo command i got following output :

# cat /proc/buddyinfo

Node 0, zone DMA 24 127 92 38 8 4 1 1 3 1 64

Node 0, zone HighMem 1348 208 183 396 114 54 28 30 11 13 257

Following is the boot-up log :

COMX-P4080 board from Emerson

arch: exit

Zone PFN ranges:

DMA 0x00000000 -> 0x00030000

Normal empty

HighMem 0x00030000 -> 0x00100000

Movable zone start PFN for each node

I did not observed Kernel memory normal zone in buddyinfo and Normal zone shows empty in bootup log as above.

Does this mean that NORMAL memory zone has been allocated in DMA ? or it is has been allocated in HIGH memory zone which can cause performance issue due to page translation as this memory is not directly mapped into the kernel ?

I get following error message while performing ftp session :

kswapd0: page allocation failure: order:2, mode:0x20

Call Trace:

[dfe4dcf0] [b0007944] show_stack+0x44/0x154 (unreliable)

[dfe4dd30] [b0096604] warn_alloc_failed+0xd8/0x130

[dfe4dd80] [b00994b8] __alloc_pages_nodemask+0x360/0x57c

[dfe4de10] [b00c4444] cache_alloc_refill+0x374/0x5fc

[dfe4de60] [b00c47ac] __kmalloc+0xe0/0x108

[dfe4de80] [b04e8790] __alloc_skb+0x60/0x118

[dfe4dea0] [b04e98ec] dev_alloc_skb+0x20/0x50

[dfe4deb0] [b03fa900] dpa_bp_add_8+0x64/0x21c

[dfe4df40] [b03fcf0c] dpaa_eth_poll+0x78/0x98

[dfe4df60] [b04f1dbc] net_rx_action+0xc0/0x178

[dfe4dfa0] [b0043cf4] __do_softirq+0xe8/0x174

[dfe4dff0] [b000cfc8] call_do_softirq+0x14/0x24

[dbe9bb70] [b0004684] do_softirq+0x98/0xc4

[dbe9bb90] [b0044048] irq_exit+0x9c/0xa0

[dbe9bba0] [b0004374] do_IRQ+0x94/0x188

[dbe9bbd0] [b000e708] ret_from_except+0x0/0x18

--- Exception: 501 at get_pageblock_flags_group+0x5c/0xa8

LR = free_hot_cold_page+0x58/0x1f0

[dbe9bc90] [b0098884] free_hot_cold_page+0x2c/0x1f0 (unreliable)

[dbe9bcb0] [b0098c74] __pagevec_free+0x38/0x54

[dbe9bcd0] [b009e534] free_page_list+0xac/0x104

[dbe9bd30] [b009fa04] shrink_page_list+0x15c/0x8a4

[dbe9bde0] [b00a05b8] shrink_inactive_list+0x228/0x3b8

[dbe9be30] [b00a0a04] shrink_zone+0x2bc/0x510

[dbe9bed0] [b00a1190] kswapd+0x538/0x838

[dbe9bfb0] [b005b5b8] kthread+0x7c/0x80

[dbe9bff0] [b000d414] kernel_thread+0x4c/0x68

Mem-Info:

DMA per-cpu:

CPU 0: hi: 186, btch: 31 usd: 162

CPU 1: hi: 186, btch: 31 usd: 172

CPU 2: hi: 186, btch: 31 usd: 183

CPU 3: hi: 186, btch: 31 usd: 131

HighMem per-cpu:

CPU 0: hi: 186, btch: 31 usd: 35

CPU 1: hi: 186, btch: 31 usd: 177

CPU 2: hi: 186, btch: 31 usd: 181

CPU 3: hi: 186, btch: 31 usd: 25

active_anon:156890 inactive_anon:34301 isolated_anon:0

active_file:111601 inactive_file:638189 isolated_file:0

unevictable:21694 dirty:46 writeback:0 unstable:0

free:4840 slab_reclaimable:17894 slab_unreclaimable:7875

mapped:31061 shmem:25091 pagetables:2867 bounce:0

Call Trace:

[dfe49cf0] [b0007944] show_stack+0x44/0x154 (unreliable)

[dfe49d30] [b0096604] warn_alloc_failed+0xd8/0x130

[dfe49d80] [b00994b8] __alloc_pages_nodemask+0x360/0x57c

[dfe49e10] [b00c4444] cache_alloc_refill+0x374/0x5fc

[dfe49e60] [b00c47ac] __kmalloc+0xe0/0x108

[dfe49e80] [b04e8790] __alloc_skb+0x60/0x118

Can anyone please help me with this questions

Thank you for your time !!!

Regards,

Amrut

yipingwang · ‎07-10-2014

Hello Amrut,

According to the Kernel boot log, it seems that you didn't configure 4G RAM in Kernel configuration, please configure "CONFIG_PHYS_64BIT=y" and "CONFIG_HIGHMEM=y" in Kernel configuration.

In addition, you could use the command "free -m -l -s 2 &" to show low and high memory allocation.

Have a great day,
Yiping Wang

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

amrutkulkarni · ‎07-10-2014

Hi Yiping,

I made sure that both the kernel config CONFIG_HIGHMEM=y & CONFIG_PHYS_64BIT=y are enabled.

meminfo shows 4G ram has been configured.

cat /proc/meminfo

MemTotal: 4023752 kB

MemFree: 26616 kB

Buffers: 10228 kB

Cached: 2564488 kB

SwapCached: 0 kB

Active: 2330260 kB

Inactive: 1434028 kB

Active(anon): 1153164 kB

Inactive(anon): 248028 kB

Active(file): 1177096 kB

Inactive(file): 1186000 kB

Unevictable: 0 kB

Mlocked: 0 kB

HighTotal: 3407868 kB

HighFree: 8036 kB

LowTotal: 615884 kB

LowFree: 18580 kB

SwapTotal: 7839684 kB

SwapFree: 7839684 kB

Dirty: 512 kB

Writeback: 0 kB

AnonPages: 1189648 kB

Mapped: 109000 kB

Shmem: 211424 kB

Slab: 166328 kB

SReclaimable: 125428 kB

SUnreclaim: 40900 kB

KernelStack: 7568 kB

PageTables: 14116 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 9851560 kB

Committed_AS: 6049140 kB

VmallocTotal: 503720 kB

VmallocUsed: 335712 kB

VmallocChunk: 148968 kB

Below is the output of free command which also shows 4G ram is being used

free -m -l -

total used free shared buffers cached

Mem: 3929 3892 37 0 8 2497

Low: 601 579 22

High: 3327 3313 14

-/+ buffers/cache: 1385 2543

Swap: 7655 0 7655

The error i mentioned earlier happens during ftp session and this causes some of the transferred file to be corrupted due to dev_alloc_skb error.

I tried to increase the /proc/sys/vm/min_free_kbytes minimum free kernel from 3533 KB to 24000 KB but still i get the same error.

Is this something related to the memory issue that during high network traffic kernel is not able to allocate the SKB in RAM as this memory cannot be swapped out due to GFP_ATOMIC flag.

Following error is being also observed with vsftpd server :

vsftpd: page allocation failure: order:2, mode:0x20

Please let me know is any way to fix/workaround this issue.

Again thank you for your time & support !!

Regards,

Amrut

yipingwang · ‎07-11-2014

Hello Amrut,

Please check whether the following is included in Linux Kernel file

if (likely(skb_is_recycleable(skb, dpa_bp->size) &&

+ (skb_end_pointer(skb) - skb->head <= DPA_BP_MAX_BUF_SIZE)

If Jumbo frames are not configured, please define as the following.

#define DPA_BP_MAX_BUF_SIZE

1984

Have a great day,
Yiping Wang

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

amrutkulkarni · ‎07-11-2014

Hi Yiping,

I did not find the the same lines of codes as mentioned above. This is what i found in SDK1.1 base source code in DPAA Ethernet network driver :

fd.length20 = skb->len;

fd.offset = DPA_BP_HEAD; /* This is now guaranteed */

if (likely(skb_is_recycleable(skb, dpa_bp->size + pad)

&& (*percpu_priv->dpa_bp_count + 1 <= dpa_bp->count))) {

fd.cmd |= FM_FD_CMD_FCO;

fd.bpid = dpa_bp->bpid;

skb_recycle(skb);

skb = NULL;

(*percpu_priv->dpa_bp_count)++;

percpu_priv->tx_returned++;

}

#define DPA_BP_HEAD (DPA_PRIV_DATA_SIZE + DPA_PARSE_RESULTS_SIZE + \

DPA_HASH_RESULTS_SIZE)

#define DPA_BP_SIZE(s) (DPA_BP_HEAD + (s))

Due to think this is related to some configuration issue ??

Thank you

Regards,

Amrut

yipingwang · ‎07-14-2014

Hello Amrut,

Please check whether the attached patch would be helpful for this problem.

If this problem remains, please feel free to let me know.

Thanks,

Yiping

lunminliang · ‎07-09-2014

Could you try the latest SDK 1.6SDKLINUX: Linux® SDK for QorIQ Processors and see the result?