dpdk on iMX8M-PLUS

NG_R3 · ‎10-26-2022

Hi there,

I'm working on a i.MX8M Plus custom board on which I'm running a Yocto produced Linux OS.
Yocto version is hardknott, kernel version is 5.10.109.

I'm interested in using dpdk.

Kernel config by default enables the following:

CONFIG_FEC=y
CONFIG_FEC_UIO=y

CONFIG_UIO=y
CONFIG_UIO_PCI_GENERIC=y

CONFIG_UIO_IVSHMEM=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO=y
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
# CONFIG_VFIO_PLATFORM is not set
# CONFIG_VFIO_MDEV is not set
CONFIG_VFIO_FSL_MC=y

In order to bind eth0 to dpdk I added into the device tree the following code:

+&fec {
+        compatible = "fsl,imx8mm-fec-uio";
+};

And indeed "ip link" shows only 1 eth ports instead of 2.

I installed the yocto produced packages dpdk_20.11-r0, dpdk-tools and dpdk-examples on the image, and followed the links:
3. Compiling and testing a PMD for a NIC — Data Plane Development Kit 22.11.0-rc1 documentation (dpd...
7. Linux Drivers — Data Plane Development Kit 22.11.0-rc1 documentation (dpdk.org)

But some errors come up...

When printing the status I get

# dpdk-devbind.py --status

Other Network devices
=====================
0000:01:00.0 'Device 2b43' unused=vfio-pci,uio_pci_generic
0000:01:00.1 'Device 2b44' unused=vfio-pci,uio_pci_generic

And I'm not entirely sure these devices represent any of the 2 ENET ports I have.

Trying to bind the first port to vfio-pci fails:

# dpdk-devbind.py --bind=vfio-pci 0000:01:00.0
Error: bind failed for 0000:01:00.0 - Cannot bind to driver vfio-pci
# dmesg
...
[  896.790092] vfio-pci: probe of 0000:01:00.0 failed with error -22
[  896.790222] vfio-pci: probe of 0000:01:00.0 failed with error -22

Trying to bind to uio_pci_generic is successful:

# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'Device 2b43' drv=uio_pci_generic unused=vfio-pci

Other Network devices
=====================
0000:01:00.1 'Device 2b44' unused=vfio-pci,uio_pci_generic

Finally, when trying to invoke dpdk-testpmd

# dpdk-testpmd -l 0-3 -n 4 -- -i
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-32768kB
EAL: No available hugepages reported in hugepages-64kB
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
Interactive-mode selected
Killed
#

And a wonderful kernel dump comes along:

[ 1287.229463] dpdk-testpmd invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[ 1287.229484] CPU: 0 PID: 571 Comm: dpdk-testpmd Not tainted 5.10.109-tq+g34fe5b1b3aab-rt65-tq #1
[ 1287.229492] Hardware name: TQ-Systems i.MX8MPlus TQMa8MPxL on MBa8MPxL (DT)
[ 1287.229495] Call trace:
[ 1287.229496]  dump_backtrace+0x0/0x1b0
[ 1287.229508]  show_stack+0x18/0x24
[ 1287.229513]  dump_stack+0xd0/0x12c
[ 1287.229519]  dump_header+0x48/0x1f8
[ 1287.229523]  oom_kill_process+0x21c/0x220
[ 1287.229531]  out_of_memory+0x174/0x54c
[ 1287.229538]  __alloc_pages_nodemask+0xe34/0xf5c
[ 1287.229547]  alloc_pages_vma+0x104/0x1dc
[ 1287.229554]  handle_mm_fault+0x61c/0xd90
[ 1287.229560]  __get_user_pages+0x204/0x3b0
[ 1287.229564]  populate_vma_page_range+0x5c/0x70
[ 1287.229569]  __mm_populate+0xbc/0x184
[ 1287.229574]  __arm64_sys_mlockall+0xf0/0x170
[ 1287.229583]  el0_svc_common.constprop.0+0x78/0x1c4
[ 1287.229590]  do_el0_svc+0x24/0x8c
[ 1287.229597]  el0_svc+0x14/0x20
[ 1287.229603]  el0_sync_handler+0x1a4/0x1b0
[ 1287.229608]  el0_sync+0x180/0x1c0
[ 1287.229657] Mem-Info:
[ 1287.229661] active_anon:97 inactive_anon:6730 isolated_anon:0
                active_file:3 inactive_file:0 isolated_file:0
                unevictable:14690 dirty:0 writeback:0
                slab_reclaimable:4977 slab_unreclaimable:6250
                mapped:4210 shmem:2711 pagetables:270 bounce:0
                free:1310 free_pcp:62 free_cma:0
[ 1287.229673] Node 0 active_anon:388kB inactive_anon:26920kB active_file:12kB inactive_file:0kB unevictable:58760kB isolated(anon):0kB isolated(file):0kB mapped:16840kB dirty:0kB writeback:0kB shmem:10844kB writeback_tmp:0kB kernel_stack:2800kB all_unreclaimable? yes
[ 1287.229683] Node 0 DMA free:5240kB min:5248kB low:6968kB high:8688kB reserved_highatomic:0KB active_anon:388kB inactive_anon:26920kB active_file:652kB inactive_file:0kB unevictable:58616kB writepending:0kB present:2097152kB managed:1727516kB mlocked:58616kB pagetables:1080kB bounce:0kB free_pcp:248kB local_pcp:248kB free_cma:0kB
[ 1287.229699] lowmem_reserve[]: 0 0 0 0
[ 1287.229706] Node 0 DMA: 189*4kB (UEC) 133*8kB (UEC) 63*16kB (UE) 47*32kB (UEC) 13*64kB (U) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 5548kB
[ 1287.229742] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1287.229746] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB
[ 1287.229750] Node 0 hugepages_total=769 hugepages_free=765 hugepages_surp=0 hugepages_size=2048kB
[ 1287.229754] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB
[ 1287.229757] 6386 total pagecache pages
[ 1287.229760] 0 pages in swap cache
[ 1287.229761] Swap cache stats: add 0, delete 0, find 0/0
[ 1287.229764] Free swap  = 0kB
[ 1287.229765] Total swap = 0kB
[ 1287.229767] 524288 pages RAM
[ 1287.229768] 0 pages HighMem/MovableOnly
[ 1287.229770] 92409 pages reserved
[ 1287.229771] 229376 pages cma reserved
[ 1287.229773] 0 pages hwpoisoned
[ 1287.229775] Tasks state (memory values in pages):
[ 1287.229776] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1287.229814] [    291]     0   291     1959      774    53248        0             0 haveged
[ 1287.229821] [    292]     0   292     3951      319    53248        0          -250 systemd-journal
[ 1287.229828] [    293]     0   293     3769      334    61440        0         -1000 systemd-udevd
[ 1287.229835] [    310]   993   310    20436      115    65536        0             0 systemd-timesyn
[ 1287.229847] [    420]     0   420      517       26    45056        0             0 atd
[ 1287.229854] [    421]     0   421      629       63    40960        0             0 crond
[ 1287.229862] [    422]   998   422     1137      108    53248        0          -900 dbus-daemon
[ 1287.229869] [    423]   997   423      684      121    40960        0             0 dhcpcd
[ 1287.229878] [    424]     0   424     1271       28    40960        0             0 agetty
[ 1287.229886] [    427]     0   427      808       87    45056        0             0 dhcpcd
[ 1287.229895] [    428]   997   428      605       66    40960        0             0 dhcpcd
[ 1287.229904] [    429]   997   429      605       66    40960        0             0 dhcpcd
[ 1287.229910] [    430]     0   430     1208      355    45056        0             0 bash
[ 1287.229919] [    431]     0   431     1895      107    49152        0             0 systemd-logind
[ 1287.229928] [    446]   995   446     2207      140    57344        0             0 systemd-network
[ 1287.229936] [    452]   994   452     1934       93    53248        0             0 systemd-resolve
[ 1287.229945] [    455] 61563   455      495       19    40960        0             0 rdisc
[ 1287.229954] [    457]     0   457     1776       51    49152        0             0 xinetd
[ 1287.229964] [    468]   997   468      805       85    45056        0             0 dhcpcd
[ 1287.229972] [    469]   997   469      805       85    45056        0             0 dhcpcd
[ 1287.229980] [    485]     0   485     1605      135    53248        0             0 sshd
[ 1287.229987] [    487]     0   487     1203      358    45056        0             0 bash
[ 1287.229995] [    509]     0   509     2427      897    53248        0             0 vi
[ 1287.230006] [    571]     0   571 16828604    14684   208896        0             0 dpdk-testpmd
[ 1287.230015] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-sshd.slice/sshd@0-10.4.20.39:22-10.4.20.38:62756.service,task=dpdk-testpmd,pid=571,uid=0
[ 1287.230060] Out of memory: Killed process 571 (dpdk-testpmd) total-vm:67314416kB, anon-rss:42608kB, file-rss:14364kB, shmem-rss:1764kB, UID:0 pgtables:204kB oom_score_adj:0
[ 1287.230630] oom_reaper: reaped process 571 (dpdk-testpmd), now anon-rss:42848kB, file-rss:14360kB, shmem-rss:1764kB
[ 1287.261435] systemd-journal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-250
[ 1287.261447] CPU: 3 PID: 292 Comm: systemd-journal Not tainted 5.10.109-tq+g34fe5b1b3aab-rt65-tq #1
[ 1287.261454] Hardware name: TQ-Systems i.MX8MPlus TQMa8MPxL on MBa8MPxL (DT)
[ 1287.261457] Call trace:
[ 1287.261458]  dump_backtrace+0x0/0x1b0
[ 1287.261469]  show_stack+0x18/0x24
[ 1287.261475]  dump_stack+0xd0/0x12c
[ 1287.261482]  dump_header+0x48/0x1f8
[ 1287.261486]  oom_kill_process+0x21c/0x220
[ 1287.261494]  out_of_memory+0x174/0x54c
[ 1287.261501]  __alloc_pages_nodemask+0xe34/0xf5c
[ 1287.261508]  alloc_pages_current+0x98/0x110
[ 1287.261515]  pagecache_get_page+0x200/0x360
[ 1287.261519]  filemap_fault+0x644/0x890
[ 1287.261525]  ext4_filemap_fault+0x34/0x870
[ 1287.261530]  __do_fault+0x3c/0x1d0
[ 1287.261538]  handle_mm_fault+0x84c/0xd90
[ 1287.261543]  do_page_fault+0x140/0x400
[ 1287.261551]  do_translation_fault+0xb0/0xd0
[ 1287.261558]  do_mem_abort+0x44/0xa4
[ 1287.261565]  el0_ia+0x64/0xbc
[ 1287.261571]  el0_sync_handler+0x180/0x1b0
[ 1287.261579]  el0_sync+0x180/0x1c0
[ 1287.261602] Mem-Info:
[ 1287.261606] active_anon:97 inactive_anon:6879 isolated_anon:0
                active_file:0 inactive_file:29 isolated_file:0
                unevictable:14567 dirty:0 writeback:0
                slab_reclaimable:4977 slab_unreclaimable:6251
                mapped:4206 shmem:2711 pagetables:271 bounce:0
                free:1247 free_pcp:131 free_cma:0
[ 1287.261617] Node 0 active_anon:388kB inactive_anon:27516kB active_file:0kB inactive_file:116kB unevictable:58268kB isolated(anon):0kB isolated(file):0kB mapped:16824kB dirty:0kB writeback:0kB shmem:10844kB writeback_tmp:0kB kernel_stack:2736kB all_unreclaimable? yes
[ 1287.261629] Node 0 DMA free:4988kB min:5248kB low:6968kB high:8688kB reserved_highatomic:0KB active_anon:388kB inactive_anon:27328kB active_file:324kB inactive_file:232kB unevictable:58204kB writepending:0kB present:2097152kB managed:1727516kB mlocked:58188kB pagetables:1084kB bounce:0kB free_pcp:524kB local_pcp:0kB free_cma:0kB
[ 1287.261647] lowmem_reserve[]: 0 0 0 0
[ 1287.261658] Node 0 DMA: 126*4kB (UEC) 133*8kB (UEC) 63*16kB (UE) 47*32kB (UEC) 13*64kB (U) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 5296kB
[ 1287.261697] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1287.261702] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB
[ 1287.261705] Node 0 hugepages_total=769 hugepages_free=765 hugepages_surp=0 hugepages_size=2048kB
[ 1287.261709] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB
[ 1287.261713] 6363 total pagecache pages
[ 1287.261716] 0 pages in swap cache
[ 1287.261717] Swap cache stats: add 0, delete 0, find 0/0
[ 1287.261720] Free swap  = 0kB
[ 1287.261721] Total swap = 0kB
[ 1287.261723] 524288 pages RAM
[ 1287.261725] 0 pages HighMem/MovableOnly
[ 1287.261726] 92409 pages reserved
[ 1287.261727] 229376 pages cma reserved
[ 1287.261729] 0 pages hwpoisoned
[ 1287.261731] Tasks state (memory values in pages):
[ 1287.261732] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1287.261765] [    291]     0   291     1959      774    53248        0             0 haveged
[ 1287.261774] [    292]     0   292     3951      319    53248        0          -250 systemd-journal
[ 1287.261783] [    293]     0   293     3769      334    61440        0         -1000 systemd-udevd
[ 1287.261792] [    310]   993   310    20436      115    65536        0             0 systemd-timesyn
[ 1287.261805] [    420]     0   420      517       26    45056        0             0 atd
[ 1287.261813] [    421]     0   421      629       63    40960        0             0 crond
[ 1287.261822] [    422]   998   422     1137      108    53248        0          -900 dbus-daemon
[ 1287.261830] [    423]   997   423      684      121    40960        0             0 dhcpcd
[ 1287.261838] [    424]     0   424     1271       28    40960        0             0 agetty
[ 1287.261847] [    427]     0   427      808       87    45056        0             0 dhcpcd
[ 1287.261855] [    428]   997   428      605       66    40960        0             0 dhcpcd
[ 1287.261864] [    429]   997   429      605       66    40960        0             0 dhcpcd
[ 1287.261872] [    430]     0   430     1208      355    45056        0             0 bash
[ 1287.261880] [    431]     0   431     1895      107    49152        0             0 systemd-logind
[ 1287.261889] [    446]   995   446     2207      140    57344        0             0 systemd-network
[ 1287.261897] [    452]   994   452     1934       93    53248        0             0 systemd-resolve
[ 1287.261906] [    455] 61563   455      495       19    40960        0             0 rdisc
[ 1287.261914] [    457]     0   457     1776       51    49152        0             0 xinetd
[ 1287.261923] [    468]   997   468      805       85    45056        0             0 dhcpcd
[ 1287.261932] [    469]   997   469      805       85    45056        0             0 dhcpcd
[ 1287.261942] [    485]     0   485     1605      135    53248        0             0 sshd
[ 1287.261952] [    487]     0   487     1203      358    45056        0             0 bash
[ 1287.261960] [    509]     0   509     2427      897    53248        0             0 vi
[ 1287.261971] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-serial\x2dgetty.slice/serial-getty@ttymxc3.service,task=vi,pid=509,uid=0
[ 1287.262002] Out of memory: Killed process 509 (vi) total-vm:9708kB, anon-rss:3584kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:52kB oom_score_adj:0
[ 1287.263712] oom_reaper: reaped process 509 (vi), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So, how can I proceed? Is there a guide to deploying dpdk on imx8 platform?

btw, under /sys/module/vfio_pci there is no "parameters" directory.

Thanks a lot,

Nir

Zhiming_Liu · ‎11-01-2022

Please refer this dts:

https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-d...