[LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

[LS1046A][LSDK 20.04] IPsec downlink data impassability issue after longrun

1,648 Views
Raymond_Cen
Contributor I

Hi,

  We have encountered an issue with IPSec. When using IPSec to receive downlink packets, after a period of time, the core layer cannot receive the packets.

Raymond_Cen_4-1683191984649.png

 

We are attempting to debug this issue
I conducted the following checks (before and after the problem occurred, the rate of downlink packet filling remained unchanged)
1. Check if there are no data packets on the network port. Through packet capture inspection, it was found that the network port can receive downstream ESP packets
2. Check the rx port of fman and find that the number of counter values increasing within 1 second before and after the problem is similar

Raymond_Cen_0-1683191251269.png

3. Check the esp interrupt and performance monitor registers, and after checking the issue, the decryption times in 1 second have increased by about the same amount

Raymond_Cen_1-1683191623564.png
Raymond_Cen_2-1683191632604.png

4. I tried to check the FMan offline port, but the counter was always 0 and there was no valid information available

Raymond_Cen_3-1683191678151.png

May I ask if there is any other information that can help us debug the issue of the core layer not receiving packets?

 

0 Kudos
15 Replies

1,592 Views
peweban424
Contributor I

Hi, We have encountered an issue with IPSec. When using IPSec to receive downlink packets, after a period of time, the core layer cannot. Check Here

0 Kudos

1,589 Views
yipingwang
NXP TechSupport
NXP TechSupport

Let me be more clear about your status,
1. Can I know IPsec run in user space or kernel space? DPDK in user space?
2. Fman rx port can continue receive ESP packets, right? Did you know where these received ESP packets go?
3. Can you see 'ob_bytes_encrypted' and 'ib_bytes_decrypted' number increase when in problematic setup?

You can have a try below commands to debug,
1. If you know to-SEC fqid, you can query this FQ to check tis status, e.g fqid 318 is to SEC,
root@:~# echo 318 > /sys/kernel/debug/qman/query_fq_np_fields
root@:~# cat /sys/kernel/debug/qman/query_fq_np_fields
root@:~# cat /sys/kernel/debug/qman/query_fq_fields

2. Stop traffic, check Bman buffer counter with below command,
cat /sys/devices/ffe000000.soc/ffe31a000.bman/pool_count/?

0 Kudos

1,025 Views
Raymond_Cen
Contributor I

Hi ,

  The previous reply was nested in too many layers, so a new reply is being opened at this layer.
Regarding the previous question, my test results indicate that it is not a problem with SEC decryption, what should I check next? Thank you.

0 Kudos

902 Views
yipingwang
NXP TechSupport
NXP TechSupport

Good to know JR mode is used and SEC work well for IPsec decryption.
If SEC can return decrypted packet successful, can core receive these packets from SEC?
If core can receive these packets, can customer trace these packets? Be dropped when route to output port?

0 Kudos

894 Views
Raymond_Cen
Contributor I

Sorry, but the only thing I can confirm so far is that no packets are received on the USER side; is there any way to count the packets between SEC and core?

0 Kudos

768 Views
yipingwang
NXP TechSupport
NXP TechSupport

You could check in two ways,
1. add printk in kernel caam driver, caam_jr_dequeue() in caam_jr_dequeue drivers\crypto\caam\jr.c
2. check JR interupt count increase or not
root@localhost:~# cat /proc/interrupts | grep jr
77: 13 0 0 0 GICv2 103 Level 1710000.jr
78: 0 0 0 0 GICv2 104 Level 1720000.jr
79: 0 0 0 0 GICv2 105 Level fsl-jr0
root@localhost:~#

0 Kudos

763 Views
Raymond_Cen
Contributor I

thanks, we will try then.

0 Kudos

1,580 Views
Raymond_Cen
Contributor I

1.ipsec in kernel , i did not use DPDK

2.Fman rx port can continue receive ESP packets, these ESP packets should enter the SEC for decryption

3.Only tested the downlink, ib_ bytes_ decrypted number consistently increase when in problematic setup

For your suggestion for debug

1.I have checked fpids, but did not see SEC's pid , just get Rx PCD,Rx PCD High Priority, RX default and RX error. May I ask if there are any other methods to determine the SEC's pid ?

2.i will try , thank you

0 Kudos

1,573 Views
yipingwang
NXP TechSupport
NXP TechSupport

Please refer to the following update from the AE team.

Can I know which LSDK customer use?

 

  1. Good to know 'ib_ bytes_ decrypted' number consistently increase when in problematic setup, it indicates SEC hardware still work with ESP packet.
  2. ipsec in kernel, customer need confirm which mode is used for caam, JR mode or QI? Customer can check this with below command, root@localhost:~# cat /proc/interrupts | grep jr

 76:        125          0          0          0     GICv2 103 Level     1710000.jr

 77:    3113307          0          0          0     GICv2 104 Level     1720000.jr

 78:     781273          0          0          0     GICv2 105 Level     1730000.jr

root@localhost:~#

root@localhost:~# cat /proc/interrupts | grep QMan

 32:          0          0          0          0     GICv2 216 Level     QMan portal 3

 33:          0          0          0          0     GICv2 218 Level     QMan portal 2

 34:          0          0          0          0     GICv2 220 Level     QMan portal 1

 35:         50          0          0          0     GICv2 222 Level     QMan portal 0

If customer use JR mode, they needn't query FQ status.

  1. Did customer find any error message when SEC finish ipsec decryption? For JR mode, they can check in this function,

caam_jr_dequeue() in driver/crypto/caam/jr.c userstatus = caam32_to_cpu(jr_outentry_jrstatus(jrp->outring,

                                                                   hw_idx));

"userstatus" should be zero if SEC run successful. Please check this status when using JR mode.

0 Kudos

1,343 Views
Raymond_Cen
Contributor I
for 3, tested and userstatus appears to be consistently 0; I will test multiple times and report the results later.
Is there any other way to check the problem?
0 Kudos

1,233 Views
yipingwang
NXP TechSupport
NXP TechSupport

As my previous answer, did customer find any error message when SEC finish ipsec decryption?
For JR mode, they can check in linux caam driver, in caam_jr_dequeue() in driver/crypto/caam/jr.c

userstatus = caam32_to_cpu(jr_outentry_jrstatus(jrp->outring,
hw_idx));
"userstatus" should be zero if SEC run successful. Please check this status when using JR mode.

0 Kudos

1,226 Views
Raymond_Cen
Contributor I

1. Yes,I used JR mode

2."userstatus" is zero when IPsec downlink data impassability, so SEC run successful.

3.You asked me if there is anything other than a "userstatus" error message when the SEC completes the ipsec decryption?

0 Kudos

1,527 Views
Raymond_Cen
Contributor I
Hi, I'm not sure, for 2 cat/proc/interrupts | grep jr, how can I confirm which mode is used for caam, JR mode or QI through this command? thanks
0 Kudos

1,461 Views
yipingwang
NXP TechSupport
NXP TechSupport

You can check if interrupts number increase for jr, it means JR mode is enabled.
Or you can check whether "CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI=y" enabled in your kenrel config. 'y' means QI mode is enabled. Meanwhile "CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API=n", JR mode should be disabled if QI mode enabled.

0 Kudos

1,443 Views
Raymond_Cen
Contributor I
ok, so my JR mode is enabled
0 Kudos