ls1046a rdb run dpdk l3fwd problem

[yi-linhsieh]

Hi all, 

I try to run lsdk 18.12 dpdk  l3fwd example on ls1046ardb.

I followed 9.2.4.4 DPDK example applications in LSDK-KC-REV18.12.

l3fwd -c 0x6 -n 1 -- -p 0x3 --config="(0,0,1),(1,0,2)"

It worked!.

But I tried to run 4 core - 2 Port, 2 queues per port:

l3fwd -c 0xF -n 1 -- -p 0x3 -P --config="(0,0,0),(0,1,1),(1,0,2),(1,1,3)"
EAL: Detected 4 lcore(s)
EAL: DPAA Bus Detected
EAL: Probing VFIO support...
EAL: VFIO support initialized
PMD: net: dpaa: fm1-mac3: 00:04:9f:05:5b:76
PMD: net: dpaa: fm1-mac4: 00:04:9f:05:5b:77
PMD: net: dpaa: fm1-mac5: 00:04:9f:05:5b:72
PMD: net: dpaa: fm1-mac6: 00:04:9f:05:5b:73
PMD: net: dpaa: fm1-mac9: 00:04:9f:05:5b:75
PMD: net: dpaa: fm1-mac10: 00:04:9f:05:5b:74
PMD: dpaa_sec-6 cryptodev init
PMD: dpaa_sec-7 cryptodev init
PMD: dpaa_sec-8 cryptodev init
PMD: dpaa_sec-9 cryptodev init
L3FWD: Promiscuous mode selected
L3FWD: LPM or EM none selected, default LPM on
Initializing port 0 ... Creating queues: nb_rxq=2 nb_txq=4... EAL: Error - exiting with code: 1
Cause: Cannot configure device: err=-22, port=0
[ 90.652132] USDPAA process leaking 16422 FQIDs
[ 91.096213] USDPAA process leaking 6 CGRIDs

Does any one know how to fix it?

Regards,

Yi-Lin

[hemantagrawal]

Just to add - you can do $export DPAA_PUSH_QUEUES_NUMBER=0 

before running your DPDK application (even without these bootargs). 

This will reduce the number of portal requirements - but performance will be bit lower.

[hemantagrawal]

try setting these extra parameters in othbootargs

If it is still not working, please send a mail to nxp support 

[yi-linhsieh]

Yes! It works after add the extra parameters in othbootargs.

[yi-linhsieh]

I had try modify RX_QUEUES setting:

export DPAA_NUM_RX_QUEUES=2
root@localhost:/usr/local/dpdk/dpaa# export DPAA_FMC_MODE=1
root@localhost:/usr/local/dpdk/dpaa# fmc -c  usdpaa_config_ls1046.xml -p usdpaa_policy_hash_ipv4_2queue.xml -a
root@localhost:/usr/local/dpdk/dpaa# cd
root@localhost:~#
root@localhost:~#
root@localhost:~# mount -t hugetlbfs none /mnt/hugepages
_hugepages

ages@localhost:~# echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepa
root@localhost:~#
root@localhost:~#
root@localhost:~#
2),(1,1,3)"ost:~# l3fwd -c 0xF -n 1 -- -p 0x3 -P --config="(0,0,0),(0,1,1),(1,0,2
EAL: Detected 4 lcore(s)
EAL: DPAA Bus Detected
EAL: Probing VFIO support...
EAL: VFIO support initialized
PMD: net: dpaa: fm1-mac3: 00:04:9f:05:5b:76
PMD: net: dpaa: fm1-mac4: 00:04:9f:05:5b:77
PMD: net: dpaa: fm1-mac5: 00:04:9f:05:5b:72
PMD: net: dpaa: fm1-mac6: 00:04:9f:05:5b:73
PMD: net: dpaa: fm1-mac9: 00:04:9f:05:5b:75
PMD: net: dpaa: fm1-mac10: 00:04:9f:05:5b:74
PMD: dpaa_sec-6 cryptodev init
PMD: dpaa_sec-7 cryptodev init
PMD: dpaa_sec-8 cryptodev init
PMD: dpaa_sec-9 cryptodev init
L3FWD: Promiscuous mode selected
L3FWD: LPM or EM none selected, default LPM on
Initializing port 0 ... Creating queues: nb_rxq=2 nb_txq=4... Address:00:04:9F:05:5B:76, Destination:02:00:00:00:00:00, Allocated mbuf pool on socket 0
LPM: Adding route 0x01010100 / 24 (0)
LPM: Adding route 0x02010100 / 24 (1)
LPM: Adding route IPV6 / 48 (0)
LPM: Adding route IPV6 / 48 (1)
txq=0,0,0 txq=1,1,0 txq=2,2,0 txq=3,3,0
Initializing port 1 ... Creating queues: nb_rxq=2 nb_txq=4... Address:00:04:9F:05:5B:77, Destination:02:00:00:00:00:01, Allocated mbuf pool on socket 0
txq=0,0,0 txq=1,1,0 txq=2,2,0 txq=3,3,0

Skipping disabled port 2

Skipping disabled port 3

Skipping disabled port 4

Skipping disabled port 5

Initializing rx queues on lcore 0 ... rxq=0,0,0
Initializing rx queues on lcore 1 ... rxq=0,1,0
Initializing rx queues on lcore 2 ... rxq=1,0,0
Initializing rx queues on lcore 3 ... rxq=1,1,0

Checking link statusdone
Port0 Link Up. Speed 1000 Mbps -full-duplex
Port1 Link Up. Speed 1000 Mbps -full-duplex
L3FWD: entering main loop on lcore 1
L3FWD: -- lcoreid=1 portid=0 rxqueueid=1
L3FWD: entering main loop on lcore 3
L3FWD: -- lcoreid=3 portid=1 rxqueueid=1
L3FWD: entering main loop on lcore 0
L3FWD: -- lcoreid=0 portid=0 rxqueueid=0
L3FWD: entering main loop on lcore 2
L3FWD: -- lcoreid=2 portid=1 rxqueueid=0
ioctl(DPAA_IOCTL_PORTAL_MAP): No such device
l3fwd: process_portal_map(): Unknown error -1
dpaa: qman_thread_init failed on core 2 (lcore=2) with ret: -1
dpaa: portal initialization failure
dpaa_eth_queue_portal_rx(): Failure in affining portal -1
ioctl(DPAA_IOCTL_PORTAL_MAP): No such device
l3fwd: process_portal_map(): Unknown error -1
dpaa: qman_thread_init failed on core 2 (lcore=2) with ret: -1

....

[hemantagrawal]

Is your issue resolved?

Can you share the output for following:

#cat /proc/cmdline

#

[yi-linhsieh]

I had returned the RDB.

I run you command with  our ls1046a board:

root@localhost:~# cat /proc/cmdline
console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=PARTUUID=c46fcc0e-03 rw rootwait

[hemantagrawal]

dpaa_eth_queue_portal_rx(): Failure in affining portal -1
ioctl(DPAA_IOCTL_PORTAL_MAP): No such device

The error is related to in-sufficient number of IO Portals for DPDK. 

Please refer to the LSDK documentation chapter for DPDK. (9.2 Data Plane Development Kit (DPDK))

There are some pre-requirements for running DPDK

1. you should be adding following to your bootargs -  default_hugepagesz=2MB hugepagesz=2MB hugepages=448 isolcpus=1-3 bportals=s0 qportals=s0 iommu.passthrough=1

    - hugepages for dpdk usages

    - isolcpus - so kernel does not schedule any task for these cores - dedicated for user-space

    - limiting the portals in kernel so that userspace dpdk can use it. 

[yi-linhsieh]

Hi Hemant,

Thanks for your reply.

I did add the parameter at bootargs. But it seems not show in /proc/cmdline.

=> print bootargs
bootargs=console=ttyS0,115200 root=/dev/ram0 earlycon=uart8250,mmio,0x21c0500 mtdparts=1550000.quadspi:1m(rcw),15m(u-boot),48m(kernel.itb);7e800000.flash:16m(nand_uboot),48m(nand_kernel),448m(nand_free) default_hugepagesz=2MB hugepagesz=2MB hugepages=448 isolcpus=1-3 bportals=s0 qportals=s0

root@localhost:~# cat /proc/cmdline
console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=PARTUUID=c46fcc0e-03 rw rootwait

Is any thing wrong?

Regards,

Yi-Lin