topic Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun in QorIQ

[LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Thu, 04 May 2023 09:20:53 GMT

Hi,

We have encountered an issue with IPSec. When using IPSec to receive downlink packets, after a period of time, the core layer cannot receive the packets.

We are attempting to debug this issue
I conducted the following checks (before and after the problem occurred, the rate of downlink packet filling remained unchanged)
1. Check if there are no data packets on the network port. Through packet capture inspection, it was found that the network port can receive downstream ESP packets
2. Check the rx port of fman and find that the number of counter values increasing within 1 second before and after the problem is similar

3. Check the esp interrupt and performance monitor registers, and after checking the issue, the decryption times in 1 second have increased by about the same amount

4. I tried to check the FMan offline port, but the counter was always 0 and there was no valid information available

May I ask if there is any other information that can help us debug the issue of the core layer not receiving packets?

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

peweban424 — Tue, 09 May 2023 04:27:22 GMT

Hi, We have encountered an issue with IPSec. When using IPSec to receive downlink packets, after a period of time, the core layer cannot. Check Here

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

yipingwang — Tue, 09 May 2023 05:04:23 GMT

Let me be more clear about your status,
1. Can I know IPsec run in user space or kernel space? DPDK in user space?
2. Fman rx port can continue receive ESP packets, right? Did you know where these received ESP packets go?
3. Can you see 'ob_bytes_encrypted' and 'ib_bytes_decrypted' number increase when in problematic setup?

You can have a try below commands to debug,
1. If you know to-SEC fqid, you can query this FQ to check tis status, e.g fqid 318 is to SEC,
root@:~# echo 318 > /sys/kernel/debug/qman/query_fq_np_fields
root@:~# cat /sys/kernel/debug/qman/query_fq_np_fields
root@:~# cat /sys/kernel/debug/qman/query_fq_fields

2. Stop traffic, check Bman buffer counter with below command,
cat /sys/devices/ffe000000.soc/ffe31a000.bman/pool_count/?

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Tue, 09 May 2023 05:51:05 GMT

1.ipsec in kernel , i did not use DPDK

2.Fman rx port can continue receive ESP packets, these ESP packets should enter the SEC for decryption

3.Only tested the downlink， ib_ bytes_ decrypted number consistently increase when in problematic setup

For your suggestion for debug

1.I have checked fpids, but did not see SEC's pid , just get Rx PCD,Rx PCD High Priority, RX default and RX error. May I ask if there are any other methods to determine the SEC's pid ？

2.i will try , thank you

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

yipingwang — Tue, 09 May 2023 06:28:40 GMT

Please refer to the following update from the AE team.

Can I know which LSDK customer use?

Good to know 'ib_ bytes_ decrypted' number consistently increase when in problematic setup, it indicates SEC hardware still work with ESP packet.
ipsec in kernel, customer need confirm which mode is used for caam, JR mode or QI? Customer can check this with below command, root@localhost:~# cat /proc/interrupts | grep jr

76: 125 0 0 0 GICv2 103 Level 1710000.jr

77: 3113307 0 0 0 GICv2 104 Level 1720000.jr

78: 781273 0 0 0 GICv2 105 Level 1730000.jr

root@localhost:~#

root@localhost:~# cat /proc/interrupts | grep QMan

32: 0 0 0 0 GICv2 216 Level QMan portal 3

33: 0 0 0 0 GICv2 218 Level QMan portal 2

34: 0 0 0 0 GICv2 220 Level QMan portal 1

35: 50 0 0 0 GICv2 222 Level QMan portal 0

If customer use JR mode, they needn't query FQ status.

Did customer find any error message when SEC finish ipsec decryption? For JR mode, they can check in this function,

caam_jr_dequeue() in driver/crypto/caam/jr.c userstatus = caam32_to_cpu(jr_outentry_jrstatus(jrp->outring,

hw_idx));

"userstatus" should be zero if SEC run successful. Please check this status when using JR mode.

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Fri, 12 May 2023 09:08:21 GMT

Hi, I'm not sure, for 2 cat/proc/interrupts | grep jr, how can I confirm which mode is used for caam, JR mode or QI through this command? thanks

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

yipingwang — Tue, 16 May 2023 03:06:24 GMT

You can check if interrupts number increase for jr, it means JR mode is enabled.
Or you can check whether "CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI=y" enabled in your kenrel config. 'y' means QI mode is enabled. Meanwhile "CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API=n", JR mode should be disabled if QI mode enabled.

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Tue, 16 May 2023 12:00:15 GMT

ok, so my JR mode is enabled

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Wed, 24 May 2023 12:30:17 GMT

for 3, tested and userstatus appears to be consistently 0; I will test multiple times and report the results later.
Is there any other way to check the problem?

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

yipingwang — Tue, 30 May 2023 01:59:27 GMT

As my previous answer, did customer find any error message when SEC finish ipsec decryption?
For JR mode, they can check in linux caam driver, in caam_jr_dequeue() in driver/crypto/caam/jr.c

userstatus = caam32_to_cpu(jr_outentry_jrstatus(jrp->outring,
hw_idx));
"userstatus" should be zero if SEC run successful. Please check this status when using JR mode.

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Tue, 30 May 2023 02:33:44 GMT

1. Yes，I used JR mode

2."userstatus" is zero when IPsec downlink data impassability, so SEC run successful.

3.You asked me if there is anything other than a "userstatus" error message when the SEC completes the ipsec decryption?

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Fri, 09 Jun 2023 02:50:33 GMT

Hi ，

The previous reply was nested in too many layers, so a new reply is being opened at this layer.
Regarding the previous question, my test results indicate that it is not a problem with SEC decryption, what should I check next? Thank you.

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

yipingwang — Mon, 19 Jun 2023 08:33:10 GMT

Good to know JR mode is used and SEC work well for IPsec decryption.
If SEC can return decrypted packet successful, can core receive these packets from SEC?
If core can receive these packets, can customer trace these packets? Be dropped when route to output port?

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Mon, 19 Jun 2023 09:55:18 GMT

Sorry, but the only thing I can confirm so far is that no packets are received on the USER side; is there any way to count the packets between SEC and core?

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

yipingwang — Fri, 30 Jun 2023 09:07:45 GMT

You could check in two ways,
1. add printk in kernel caam driver, caam_jr_dequeue() in caam_jr_dequeue drivers\crypto\caam\jr.c
2. check JR interupt count increase or not
root@localhost:~# cat /proc/interrupts | grep jr
77: 13 0 0 0 GICv2 103 Level 1710000.jr
78: 0 0 0 0 GICv2 104 Level 1720000.jr
79: 0 0 0 0 GICv2 105 Level fsl-jr0
root@localhost:~#

Re: [LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

Raymond_Cen — Fri, 30 Jun 2023 10:48:08 GMT

thanks, we will try then.