kernel BUG at /home/xxx/IoT-gateway-platform-Release-V0.1-20170120-yocto/build_ls1021aiot/tmp/work-shared/ls1021aiot/kernel-source/include/linux/skbuff.h:1707!

[yuqianli]

We are testing Strongswan with  IoT-LS1021-SDK-V0.1, as there is a lot of compiling missing header files issue in the default SDK with strongswan_5.1.3, we move to Strongswan 5.2.3 which got from here meta-networking/recipes-support/strongswan · 1dbb1ff9b15e42ace9ac12c8e0630b9e520e17cd · yocto / meta...  

Now we can get StrongSwan running without a problem. but when try to testing its performance with  Iperf, we always got the following issue, prompted it is a kernel bug

[ 5] local 192[ 1796.659536] ------------[ cut here ]------------
[ 1796.665203] kernel BUG at /home/liyq/IoT-gateway-platform-Release-V0.1-20170120-yocto/build_ls1021aiot/tmp/work-shared/ls1021aiot/kernel-source/include/linux/skbuff.h:1707!
[ 1796.680506] Internal error: Oops - BUG: 0 [#1] SMP ARM
[ 1796.685614] Modules linked in:
[ 1796.688657] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.30-rt34+g73b5a4e #44
[ 1796.695836] Hardware name: Freescale LS1021A
[ 1796.700080] task: c13d9a30 ti: c13ca000 task.ti: c13ca000
[ 1796.705455] PC is at skb_pull+0x40/0x44
[ 1796.709270] LR is at gfar_clean_rx_ring+0x3b0/0x780
[ 1796.714120] pc : [<c0b3da7c>] lr : [<c07f6d34>] psr: 000b0113
[ 1796.714120] sp : c13cbde8 ip : 00000000 fp : ee548000
[ 1796.725536] r10: 00000046 r9 : ed7a3000 r8 : 00000450
[ 1796.730729] r7 : ed1eb840 r6 : ed083000 r5 : ee5484c0 r4 : ee51cd80
[ 1796.737219] r3 : 000005f6 r2 : fffffffe r1 : 00000008 r0 : ed083000
[ 1796.743710] Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 1796.750977] Control: 30c5387d Table: ae642c40 DAC: fffffffd
[ 1796.756689] Process swapper/0 (pid: 0, stack limit = 0xc13ca220)
[ 1796.762660] Stack: (0xc13cbde8 to 0xc13cc000)
[ 1796.766993] bde0: fffffffe 00000800 00000001 ed0ba800 00000002 00000002
[ 1796.775127] be00: 18000602 08000000 c13daedc 00000040 000005fe 00000001 ee5487d8 ee5487f8
[ 1796.783263] be20: 00000000 c13ce604 ee548548 f00da000 ee51cd80 00000040 c13c8640 c13cbe78
[ 1796.791398] be40: 00049146 c07f7374 ee548548 c07f732c c13cc140 0000012c 00000040 c0b4ae90
[ 1796.799533] be60: 51388160 eefd1640 2dc09000 c14d964b c13cff78 c13cff78 c13cbe78 c13cbe78
[ 1796.807669] be80: c13cbe80 c13cbe80 00000000 c13cc08c 00000000 c13cc080 c13ca000 00000100
[ 1796.815805] bea0: c13cbeb0 00000003 0000000c c0253884 00000000 ee51e9c0 ee51ea28 c14dfec0
[ 1796.823940] bec0: 0000000a 00049145 c13cc140 00200000 00000001 c13c5ff8 00000000 00000000
[ 1796.832075] bee0: 00000001 ee008000 c13cbf80 c0da73d4 00000001 c0253fe4 c13c5ff8 c028c364
[ 1796.840209] bf00: f000200c c13cea18 c13cbf30 f0002000 00000000 c020a6b8 c0211148 600b0013
[ 1796.848344] bf20: ffffffff c13cbf64 00000000 c0214680 00000001 00000000 00000000 c0222a40
[ 1796.856480] bf40: c13ca000 c13ce6d0 c13ce670 00000000 00000000 c13cbf80 c0da73d4 00000001
[ 1796.864615] bf60: 01000000 c13cbf78 c0211144 c0211148 600b0013 ffffffff c13ca000 c0281548
[ 1796.872749] bf80: 00000000 c13c439c c13c7138 c0d9b020 c13c7e00 c14d93c9 c13ce6d8 c0d9b3bc
[ 1796.880884] bfa0: c136edd0 c13ce5c0 c136edd0 c12d4c80 ffffffff ffffffff c12d4678 00000000
[ 1796.889019] bfc0: ffffffff 00000000 00000000 c136edd0 00000000 c14dc5d4 c13ce640 c136edcc
[ 1796.897154] bfe0: c13dabc0 80207000 410fc075 00000000 00000000 8020808c 00000000 00000000
[ 1796.905300] [<c0b3da7c>] (skb_pull) from [<c07f6d34>] (gfar_clean_rx_ring+0x3b0/0x780)
[ 1796.913181] [<c07f6d34>] (gfar_clean_rx_ring) from [<c07f7374>] (gfar_poll_rx_sq+0x48/0xe4)
[ 1796.921493] [<c07f7374>] (gfar_poll_rx_sq) from [<c0b4ae90>] (net_rx_action+0x1d4/0x2d0)
[ 1796.929547] [<c0b4ae90>] (net_rx_action) from [<c0253884>] (__do_softirq+0x100/0x21c)
[ 1796.937339] [<c0253884>] (__do_softirq) from [<c0253fe4>] (irq_exit+0xb4/0x11c)
[ 1796.944614] [<c0253fe4>] (irq_exit) from [<c028c364>] (__handle_domain_irq+0x60/0xb4)
[ 1796.952407] [<c028c364>] (__handle_domain_irq) from [<c020a6b8>] (gic_handle_irq+0x30/0x6c)
[ 1796.960716] [<c020a6b8>] (gic_handle_irq) from [<c0214680>] (__irq_svc+0x40/0x54)
[ 1796.968156] Exception stack(0xc13cbf30 to 0xc13cbf78)
[ 1796.973178] bf20: 00000001 00000000 00000000 c0222a40
[ 1796.981312] bf40: c13ca000 c13ce6d0 c13ce670 00000000 00000000 c13cbf80 c0da73d4 00000001
[ 1796.989447] bf60: 01000000 c13cbf78 c0211144 c0211148 600b0013 ffffffff
[ 1796.996031] [<c0214680>] (__irq_svc) from [<c0211148>] (arch_cpu_idle+0x38/0x3c)
[ 1797.003393] [<c0211148>] (arch_cpu_idle) from [<c0281548>] (cpu_startup_entry+0x268/0x2d0)
[ 1797.011620] [<c0281548>] (cpu_startup_entry) from [<c12d4c80>] (start_kernel+0x3c0/0x3cc)
[ 1797.019756] Code: e12fff1e e3a01000 e1a00001 e12fff1e (e7f001f2)
[ 1797.025824] ---[ end trace d9575ad44e6e6c5f ]---
[ 1797.030414] Kernel panic - not syncing: Fatal exception in interrupt
[ 1797.036739] CPU1: stopping
[ 1797.039436] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 4.1.30-rt34+g73b5a4e #44
[ 1797.047825] Hardware name: Freescale LS1021A
[ 1797.052081] [<c02183a4>] (unwind_backtrace) from [<c0213b14>] (show_stack+0x10/0x14)
[ 1797.059789] [<c0213b14>] (show_stack) from [<c0d956d4>] (dump_stack+0x98/0xac)
[ 1797.066978] [<c0d956d4>] (dump_stack) from [<c0216b98>] (handle_IPI+0x150/0x170)
[ 1797.074337] [<c0216b98>] (handle_IPI) from [<c020a6f0>] (gic_handle_irq+0x68/0x6c)
[ 1797.081868] [<c020a6f0>] (gic_handle_irq) from [<c0214680>] (__irq_svc+0x40/0x54)
[ 1797.089307] Exception stack(0xee07df78 to 0xee07dfc0)
[ 1797.094328] df60: 00000001 00000000
[ 1797.102463] df80: 00000000 c0222a40 ee07c000 c13ce6d0 c13ce670 00000000 00000000 ee07dfc8
[ 1797.110599] dfa0: c0da73d4 00000001 01000000 ee07dfc0 c0211144 c0211148 60080013 ffffffff
[ 1797.118735] [<c0214680>] (__irq_svc) from [<c0211148>] (arch_cpu_idle+0x38/0x3c)
[ 1797.126096] [<c0211148>] (arch_cpu_idle) from [<c0281548>] (cpu_startup_entry+0x268/0x2d0)
[ 1797.134318] [<c0281548>] (cpu_startup_entry) from [<8020a82c>] (0x8020a82c)
[ 1797.141245] Rebooting in 3 seconds..

Does anyone get same issue with new SDK, thank you.

[yipingwang]

Hello Yuqian,

I suspect this problem is cause the tool Strongswan. According to your Kernel information, this SDK release for IoT should be based on QorIQ SDK 2.0 1611 release, we have performed full IPSEC test on LS1021ATWR in Kernel 4.1.30. However in the testing, we don't use Strongswan to do IPSEC test, we use setkey(the static key generation method) to do IPSEC performance testing, because Strongswan will bring in the performance data downgrading.

I suggest you do IPSEC test with setkey firset, please refer to the attached log on LS1021ATWR.

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yuqianli]

Hi Yiping,

I appreciate your suggestion. Do you mind run an Iperf test on both sides of LS1021ATWR? I want to see how the performance which is without Strongswan :smileyhappy: . The Iperf is easy to run, like: 

B1 side

iperf -s

B2 side

iperf -c x.x.x.x (ip address of B1)

But please run this test with Gigabyte Ethernet switch connected.

Also,  can I use this link Submit Form for reference to set up the test scenario? 

Thank you,

Have a good day.

[yipingwang · Accepted solution]

Hello Yuqian,

You could refer to the IPsec setup procedure from Submit Form , it seems this is the most detailed IPSEC procedure in this document. Please ignore the ASF module inserting section, in addition this procedure can be used for DPAA and non DPAA target boards, please open use the non DPAA parts. Please use setkey method to configure IPsec. You could refer to the testing logs which I provided previously.

You could refer to the following 2 cores 2 OF ((IPv4_SEC_AES_SHA1_SMP_2C2OF_eTSEC_Perf) ) performance data on LS1021ATWR SDK2.0 GA 1611 release from the testing team.

Througput(Mbps)            FrameSize(Bytes)            Throughput(%)

81.88                                82                                     4.09

342.91                              408                                    17.15

1145.35                            1442                                   57.27

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yuqianli]

Hi Yiping,

When I try to load the CAAM module in our LS1021A-IOT orange box, it always shows me

[ 60.608875] caam 1700000.crypto: can't identify CAAM ipg clk: -2
[ 60.614919] caam: probe of 1700000.crypto failed with error -2

Is there some special configure need to done in dts file of ls1021a.dtsi?

Thanks again.

[yipingwang]

Hello Yuqian,

Let's continue to discuss this new problem in the thread https://community.nxp.com/thread/449509 , would you please attach your dts file in that thread?

Thanks,

Yiping

[yuqianli]

Hi Yiping,

Thanks for fast reply. I had post the contents of dts file to that thread.