[LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

[Raymond_Cen]

Hi,

  We have encountered an issue with IPSec. When using IPSec to receive downlink packets, after a period of time, the core layer cannot receive the packets.

We are attempting to debug this issue
I conducted the following checks (before and after the problem occurred, the rate of downlink packet filling remained unchanged)
1. Check if there are no data packets on the network port. Through packet capture inspection, it was found that the network port can receive downstream ESP packets
2. Check the rx port of fman and find that the number of counter values increasing within 1 second before and after the problem is similar

3. Check the esp interrupt and performance monitor registers, and after checking the issue, the decryption times in 1 second have increased by about the same amount

4. I tried to check the FMan offline port, but the counter was always 0 and there was no valid information available

May I ask if there is any other information that can help us debug the issue of the core layer not receiving packets?

[peweban424]

Hi, We have encountered an issue with IPSec. When using IPSec to receive downlink packets, after a period of time, the core layer cannot. Check Here

[yipingwang]

Let me be more clear about your status,
1. Can I know IPsec run in user space or kernel space? DPDK in user space?
2. Fman rx port can continue receive ESP packets, right? Did you know where these received ESP packets go?
3. Can you see 'ob_bytes_encrypted' and 'ib_bytes_decrypted' number increase when in problematic setup?

You can have a try below commands to debug,
1. If you know to-SEC fqid, you can query this FQ to check tis status, e.g fqid 318 is to SEC,
root@:~# echo 318 > /sys/kernel/debug/qman/query_fq_np_fields
root@:~# cat /sys/kernel/debug/qman/query_fq_np_fields
root@:~# cat /sys/kernel/debug/qman/query_fq_fields

2. Stop traffic, check Bman buffer counter with below command,
cat /sys/devices/ffe000000.soc/ffe31a000.bman/pool_count/?

[Raymond_Cen]

Hi ，

  The previous reply was nested in too many layers, so a new reply is being opened at this layer.
Regarding the previous question, my test results indicate that it is not a problem with SEC decryption, what should I check next? Thank you.

[yipingwang]

Good to know JR mode is used and SEC work well for IPsec decryption.
If SEC can return decrypted packet successful, can core receive these packets from SEC?
If core can receive these packets, can customer trace these packets? Be dropped when route to output port?

[Raymond_Cen]

Sorry, but the only thing I can confirm so far is that no packets are received on the USER side; is there any way to count the packets between SEC and core?

[yipingwang]

You could check in two ways,
1. add printk in kernel caam driver, caam_jr_dequeue() in caam_jr_dequeue drivers\crypto\caam\jr.c
2. check JR interupt count increase or not
root@localhost:~# cat /proc/interrupts | grep jr
77: 13 0 0 0 GICv2 103 Level 1710000.jr
78: 0 0 0 0 GICv2 104 Level 1720000.jr
79: 0 0 0 0 GICv2 105 Level fsl-jr0
root@localhost:~#

[Raymond_Cen]

thanks, we will try then.

[Raymond_Cen]

1.ipsec in kernel , i did not use DPDK

2.Fman rx port can continue receive ESP packets, these ESP packets should enter the SEC for decryption

3.Only tested the downlink， ib_ bytes_ decrypted number consistently increase when in problematic setup

For your suggestion for debug

1.I have checked fpids, but did not see SEC's pid , just get Rx PCD,Rx PCD High Priority, RX default and RX error. May I ask if there are any other methods to determine the SEC's pid ？

2.i will try , thank you

[yipingwang]

Please refer to the following update from the AE team.

Can I know which LSDK customer use?

1. Good to know 'ib_ bytes_ decrypted' number consistently increase when in problematic setup, it indicates SEC hardware still work with ESP packet.
2. ipsec in kernel, customer need confirm which mode is used for caam, JR mode or QI? Customer can check this with below command, root@localhost:~# cat /proc/interrupts | grep jr

 76:        125          0          0          0     GICv2 103 Level     1710000.jr

 77:    3113307          0          0          0     GICv2 104 Level     1720000.jr

 78:     781273          0          0          0     GICv2 105 Level     1730000.jr

root@localhost:~#

root@localhost:~# cat /proc/interrupts | grep QMan

 32:          0          0          0          0     GICv2 216 Level     QMan portal 3

 33:          0          0          0          0     GICv2 218 Level     QMan portal 2

 34:          0          0          0          0     GICv2 220 Level     QMan portal 1

 35:         50          0          0          0     GICv2 222 Level     QMan portal 0

If customer use JR mode, they needn't query FQ status.

3. Did customer find any error message when SEC finish ipsec decryption? For JR mode, they can check in this function,

caam_jr_dequeue() in driver/crypto/caam/jr.c userstatus = caam32_to_cpu(jr_outentry_jrstatus(jrp->outring,

                                                                   hw_idx));

"userstatus" should be zero if SEC run successful. Please check this status when using JR mode.

[Raymond_Cen]

for 3, tested and userstatus appears to be consistently 0; I will test multiple times and report the results later.
Is there any other way to check the problem?

[yipingwang]

As my previous answer, did customer find any error message when SEC finish ipsec decryption?
For JR mode, they can check in linux caam driver, in caam_jr_dequeue() in driver/crypto/caam/jr.c

userstatus = caam32_to_cpu(jr_outentry_jrstatus(jrp->outring,
hw_idx));
"userstatus" should be zero if SEC run successful. Please check this status when using JR mode.

[Raymond_Cen]

1. Yes，I used JR mode

2."userstatus" is zero when IPsec downlink data impassability, so SEC run successful.

3.You asked me if there is anything other than a "userstatus" error message when the SEC completes the ipsec decryption?

[Raymond_Cen]

Hi, I'm not sure, for 2 cat/proc/interrupts | grep jr, how can I confirm which mode is used for caam, JR mode or QI through this command? thanks

[yipingwang]

You can check if interrupts number increase for jr, it means JR mode is enabled.
Or you can check whether "CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI=y" enabled in your kenrel config. 'y' means QI mode is enabled. Meanwhile "CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API=n", JR mode should be disabled if QI mode enabled.

[Raymond_Cen]

ok, so my JR mode is enabled