qman Error while running DPDK application

[Ghouse]

Hi,

We are using a custom board based on LS1046A, we are facing "qman_retire_fq failed" error when we are running L2_Crypto application in one of our Device.

Please note that we have tested the same device for dpdk-l2fwd and we have not seen such error, the error is coming only when we run dpdk-l2crypto application.

Please find attached the completed dmesg report after the error has occurred, also attached is the complete boot log for the same device.

Please let me know what can be done to resolve this error.

[yipingwang]

Please pay attention to the following procedure to run DPDK dpdk-l2fwd-crypto application.

1. Please boot the target board with USDPAA dtb fsl-ls1046a-rdb-usdpaa.dtb.

2. Please add the following bootargs in u-boot, then run "boot" command.

=>setenv othbootargs bportals=s0 qportals=s0 isolcpus=1-3 default_hugepagesz=1024m hugepagesz=1024m hugepages=2 nmi_watchdog=0 rcupdate.rcu_cpu_stall_suppress=1

 3. After boot up the system, please run the following command

# modprobe caam_jr

4. Please run the following commands sequence before run dpdk-l2fwd-crypto.

 fmc -x

mkdir -p /mnt/hugepages

mount -t hugetlbfs none /mnt/hugepages

export DPAA_FMC_MODE=1

fmc -c /usr/local/dpdk/dpaa/usdpaa_config_ls1046.xml -p /usr/local/dpdk/dpaa/usdpaa_policy_hash_ipv4_4queue.xml -a

export DPAA_NUM_RX_QUEUES=1 

dpdk-l2fwd-crypto --vdev "crypto_openssl0" -v -c 0x8 -n 1   -- -p 0x20 -q 1 --chain CIPHER_ONLY --cryptodev_mask 0x10 --cipher_algo aes-cbc --cipher_op ENCRYPT --cipher_key 01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f:10

 If your problem persists, please capture your console log running dpdk-l2fwd-crypto to me to do more investigation.

[Ghouse]

@yipingwang, thank you for the reply,

Please note that in our custom board we are using 4GB DDR4 (The ram section is copied from LS1046AFRWY, i.e MT40A512M16LY is being used)

And in our board fm1-mac3 and fm1-mac4 is dedicated to DPDK(for Traffic handling) and fm1-mac5 and fm1-mac6 is dedicated to Kernel(For Configuration purpose)

With above setup, please let me know what is the optimal settings for hugepage configurations, if below settings can be used

setenv othbootargs bportals=s0 qportals=s0 isolcpus=1-3 default_hugepagesz=1024m hugepagesz=1024m hugepages=2 nmi_watchdog=0 rcupdate.rcu_cpu_stall_suppress=1

or below settings can be used

setenv othbootargs bportals=s0 qportals=s0 isolcpus=1-3 default_hugepagesz=2MB hugepagesz=2MB hugepages=1024 nmi_watchdog=0 rcupdate.rcu_cpu_stall_suppress=1

or it needs to be modified.

Also, we referred DPDK programers manual to run below comands, but we got vdev creation error,

dpdk-l2fwd-crypto -l 0-1 --vdev "crypto_aesni_mb0" \
--vdev "crypto_aesni_mb1" -- -p 0x3 --chain CIPHER_HASH \
--cipher_op ENCRYPT --cipher_algo aes-cbc \
--cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f \
--auth_op GENERATE --auth_algo aes-xcbc-mac \
--auth_key 10:11:12:13:14:15:16:17:18:19:1a:1b:1c:1d:1e:1f

so, can we use below commands to Encrypt in one device

dpdk-l2fwd-crypto --vdev "crypto_openssl0" -v -c 0x8 -n 1   -- -p 0x20 -q 1 --chain CIPHER_ONLY --cryptodev_mask 0x10 --cipher_algo aes-cbc --cipher_op ENCRYPT --cipher_key 01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f:10

And below commands in another device to Decrypt

dpdk-l2fwd-crypto --vdev "crypto_openssl0" -v -c 0x8 -n 1   -- -p 0x20 -q 1 --chain CIPHER_ONLY --cryptodev_mask 0x10 --cipher_algo aes-cbc --cipher_op DECRYPT --cipher_key 01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f:10

Also as per my understanding above commands use the S/W engine to encrypt, what commnads can be used to make use of the Sec Hardware of LS1046A.

Please provide your help.

Thank u.

[yipingwang]

You could use the following bootargs.

bportals=s0 qportals=s0 isolcpus=1-3 default_hugepagesz=2m hugepagesz=2m hugepages=448 nmi_watchdog=0 rcupdate.rcu_cpu_stall_suppress=1

About l2fwd-crypto we only have ENCRYPT case.

ENCRYPT can be specified as the following.

For caam_jr mode:
  use  --vdev 'crypto_caam_jr1'
For openssl mode:
  use --vdev 'crypto_openssl'
For armv8_crypto mode:
  use --vdev 'crypto_armv8'

Please refer to the following command.

root@localhost:~# /root/dpdkfiles/l2fwd-crypto -v -c 0x8 -n 1 -- -p 0x20 -q 1 --chain CIPHER_HASH --cipher_algo aes-cbc --cipher_op ENCRYPT --cipher_key 01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f:10 --auth_algo sha1-hmac --auth_op GENERATE --auth_key_random_size 64

[Ghouse]

@yipingwangthank u for the reply,

what do u mean by "About l2fwd-crypto we only have ENCRYPT case" does it mean that the l2fwd-crypto does not support DECRYPT ??

Also we are using the caam_jr mode for Encryption and we are using KNI (virtual interface of the DPDK), are there any prerequisites in the OS before loading the CAAM module (i.e before running modprobe caam_jr ) and the KNI module (i.e insert KNI)

Please provide your help.

Thank u.

[yipingwang]

I checked l2fwd-crypto source code, DECRYPT  is supported, however we only tested ENCRYPT in LSDK.

Please refer to the following test steps for dpdk-kni.

Test steps:

fmc -x

mkdir -p /mnt/hugepages

mount -t hugetlbfs none /mnt/hugepages

export DPAA_FMC_MODE=1

fmc -c /usr/local/dpdk/dpaa/usdpaa_config_ls1043.xml -p /usr/local/dpdk/dpaa/usdpaa_policy_hash_ipv4_1queue.xml -a

insmod /usr/local/dpdk/rte_kni.ko carrier=on

taskset -pc 1 `pgrep -fl kni_single | awk '{print $1}'`

dpdk-kni -c 0x3 -n 1 – -p 0x03 --config="(0,0,0),(1,0,1)"

[hemantagrawal]

Though caam_jr is supported on ls1043 but caam_dpaa is better in terms of functionality and performance. (It is default crypto)

l2fwd_crypto do not support proper IPSEC protocol, it only demonstrate getting a packet and using crypto for it. If you want to use decryption with it. - you need to put two system back to back, encrypt on one and decrypt on other. 

ipsec-gw is a better suitable example to test. 

check: 
https://docs.nxp.com/bundle/GUID-C3A436DA-E944-4F73-9811-2335DEBD04D6/page/GUID-9B14728A-B012-48BC-8...

https://doc.dpdk.org/guides/cryptodevs/dpaa_sec.html

[Ghouse]

Hi @hemantagrawal 

Thank u for the reply,

In our setup we are using 2 custom boards based on LS41046A (Refer Attached block diagram), however in the Diagram Board No1 should Encrypt Uplink traffic and Decrypt Downlink traffic, the entire network is in same L2 Network.

Now, with this setup we are running the dpdk-l2crypto example code, will there be any feasibility issue??

Also, how do i enable caam_dpaa after OS boots ? 

we are using "modprobe caam" is this enough to invoke caam_dpaa?

Please provide your help

Thank u.

[hemantagrawal]

dpdk-l2fwd is not IPSEC. It is networking + raw crypto. 

DPDK on DPAA1 by default enables the crypto_dpaa (caam) - unless you are having non-crypto parts. check the dpdk application prints for it's enablement/config.

It is completely indenpendent of the kernel caam driver. No relationship at all. 

You can initally run the following test on your board

(https://doc.dpdk.org/guides/contributing/unit_test.html)

./dpdk-test  

than run the cryptodev_dpaa_sec_autotest

if you see too many logs - just run dpdk-test with --log-level=6