Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

cancel
Showing results for 
Search instead for 
Did you mean: 

Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

4,869 Views
h_eznt
Contributor III

Hardware impacted:

- imx6qsabresd  BSP Board (MCIMX6Q-SDB)

- imx6 Solo custom board

 

Linux  kernel versions:

- imx_4.1.15_1.0.0_ga,

- imx_3.10.17_1.0.0_ga

- and probably all others

 

Description of the Problem:

Encryption of IPSEC ESP-packets coming from eth0 ("fec" driver)  fails with CAAM when encryption algorithm is AES or AES256 on Linux platforms.

 

The sample test setup is as follow:

 

  • an i.mx6 BSP board (MCIMX6Q-SDB) with 2 network interfaces:
    •    eth0:  attached to the processor, handled in Linux by the fec driver
    •    eth1:  a digitus USB2.0 Ethernet  Adapter (Pegasus/Pegasus II USB Ethernet driver) connected to the OTG port.
  • The board runs the latest linux 4.1.15_1.0.0_ga yocto software with "ipsec-tools" and the necessary ipsec kernel modules .i.e.

CONFIG_XFRM_IPCOMP=m

CONFIG_NET_KEY=y

CONFIG_INET_AH=m

CONFIG_INET_ESP=m

CONFIG_INET_IPCOMP=m

CONFIG_INET_XFRM_TUNNEL=m

CONFIG_INET_TUNNEL=y

CONFIG_INET_XFRM_MODE_TRANSPORT=m

CONFIG_INET_XFRM_MODE_TUNNEL=m

CONFIG_INET_XFRM_MODE_BEET=m

  • The ipsec client on the board  (ipsec-tools) is configured to encrypt traffic in an IPSEC-tunnel from eth0 to a VPN-server behind eth1: the ip traffic comes unencrypted via eth0 and goes out encrypted over eth1. This is a typical "road-warrior" scenario.
  • The esp encryption algorithm chosen is aes (or aes256)
  • In this case CAAM does not encrypt correctly the ESP packets and they are discarded by the VPN-server.
  • It doesn't matter which VPN-server is used, we have been trying with CISCO ASA, various Juniper gateways, PfSense, racoon, strongSwan: problem is always there.
  • We have been replacing ipsec-tools by strongSwan: the problem remains
  • We have been investigating this issue quiet intensively these last 6 months and found out that:
    •   changing the encryption algorithm from aes to 3des solves this issue
    •   deactivating caam also solves this issue
    •   switching cable and ip addresses of eth0 and eth1 also solves this issue
    •   using VLAN tagging on eth0 solves this issue

 

For different (good) reasons non of these workarounds are acceptable for us: we have a customer that needs to perform IPSEC AES encryption from eth0 to eth1. This issue seems to be present on all i.MX6 platforms.

 

Are there any workarounds other than the ones listed above for this issue ?

 

We can provide additional logging/configuration files/images if needed.

Attached is a short description of the IPSEC  test-setup on the i.MX6 test board.

Original Attachment has been moved to: ipsec.sh

Original Attachment has been moved to: racoon.setkey.zip

Original Attachment has been moved to: racoon.conf.aes.zip

Original Attachment has been moved to: run-logs.tgz

Original Attachment has been moved to: setup.log.zip

Labels (2)
29 Replies

2,055 Views
utkarsh_gupta
NXP Employee
NXP Employee

Hi ko-hey,

Have you tried to patch the patches attached by me in the thread above. They may not patch cleanly. Let me know if they resolve the issue. We do not have 4.1.15 specific patches currently for this fix.

Regards,

Utkarsh

0 Kudos

2,055 Views
karina_valencia
NXP Apps Support
NXP Apps Support

Ragan_Dunham‌ can you   help to provide the status of this thread to Ko-hey?

0 Kudos

2,055 Views
h_eznt
Contributor III

Anything new there ? Is still somebody working on this issue ? The crypto interface in kernel 4.1 has changed a lot since kernel 3.x . Therefore It is impossible for me to up-port the 3.14 CAAM-patches into 4.1. If I cannot make the encryption engine work properly for IPSEC  in 4.1.15, I will need to move my customers to a different hardware.

2,055 Views
Ragan_Dunham
NXP Employee
NXP Employee

Hi Hubert,

As Utkarsh mentioned at this time there are no plans to port these patches to the 4.1.x series. We will review internally to determine the effort and provide additional feedback at a later time.

If this is a firm requirement on your side I would encourage you to escalate through your local NXP sales representative.

Best Regards,

Ragan

0 Kudos

2,055 Views
ko-hey
Senior Contributor II

Hi Ragan_Dunham

My customer have same issue in i.MX7D with L4.1.15 and let me confirm the status of this issue.

According to the communication, the issue related to hardware and make the patch file to resolve by sofware.

Am I correct ?

Please let me know the latest status of the issue.

Ko-hey

0 Kudos

2,055 Views
utkarsh_gupta
NXP Employee
NXP Employee

Hi,

Sorry I have higher priority issues I am stuck with due to which I am unable to spend time on this issue. I will update you as soon as I get time to look into this issue.

Regards,

Utkarsh

0 Kudos

2,055 Views
h_eznt
Contributor III

Any news on 4.1.15 port of the patches? I would be glad to test!

0 Kudos

2,055 Views
h_eznt
Contributor III

OK, Just let me know when the patches are committed in the 4.1.15 branch, I will test them right away then.

0 Kudos

2,058 Views
h_eznt
Contributor III

The log extract above was taken from our platform. Now I did try on the mcimx6q BSP board without software changes and the result is similar to it.  The kernel crashes badly and hangs. 

See the log here under: 

2016-09-01 10:23:07: INFO: IPsec-SA established: ESP/Tunnel 192.168.3.77[500]->192.168.3.1[500] spi=7872688)
2016-09-01 10:23:07: DEBUG: ===


Unable to handle kernel paging request at virtual address 70000000
pgd = 80004000
[70000000] *pgd=00000000
Internal error: Oops: 805 [#1] PREEMPT SMP ARM
Modules linked in: esp4 xfrm4_mode_tunnel hmac ov5642_camera mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still g
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.15-1.1.1+gd5d7c02 #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: 80b038f0 ti: 80afe000 task.ti: 80afe000
PC is at v7_dma_clean_range+0x20/0x38
LR is at __dma_page_cpu_to_dev+0x28/0x94
pc : [<8001ee3c>] lr : [<8001ad90>] psr: 000f0113
sp : 80affc48 ip : ab757000 fp : 00000000
r10: a89e8b40 r9 : a876b410 r8 : 00000000
r7 : 00000001 r6 : 00000000 r5 : ab557000 r4 : 00000000
r3 : 0000001f r2 : 00000020 r1 : 70000000 r0 : 70000000
Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 389b004a DAC: 00000015
Process swapper/0 (pid: 0, stack limit = 0x80afe210)
Stack: (0x80affc48 to 0x80b00000)
fc40: 8001ee8c 00000000 a8c3a040 80b04bc4 8001adfc a8c3a040
fc60: 00000000 8057778c 00000001 00000000 00000002 00000003 00000000 38ae647e
fc80: 00000010 00000000 00000000 a8775000 00000001 00000000 a8c3a080 80bb81c0
fca0: a8c3a040 a89e8000 a8c3a040 a8c3a000 a8ae647e a8c3a0d0 00000000 a8acca00
fcc0: a8c3a0c0 7f092a1c 0000000c 00000060 a8ae6440 a8c3a0d0 a8c3a000 a8c3a000
fce0: 00000000 00000036 00000008 00000000 00010000 00000000 0201010a a89e8000
fd00: 00000008 00000054 00000000 a89e8000 a8acca00 00000000 a829405c 80b0128c
Unable to handle kernel paging request at virtual address 72617493

Additionally, I think that the patches that you provided are quite problematic because they cannot be applied directly on the NXP sources. So what are they supposed to be applied on ? Parts of some patches seem to have been committed already !  So I had to sort out manually the parts that had been committed already and reformed the patches accordingly. May be you could provide a patch set that can be directly applied on the NXP git source tree ?  

For info: here is  the applicable patch that I could prepare out of your patches.:

0 Kudos

2,055 Views
utkarsh_gupta
NXP Employee
NXP Employee

At the time while preparing the patches I was working on imx_3.14.52_1.1.0_ga branch. Those patches can be clearly applied to this branch and the setup works with it too.

I see that few of the patches have still not been applied to current 4.1.15 branch and I am working with BSP team in incorporating those patches in next release.

Let me know your tests.

Thanks,

Utkarsh

0 Kudos

2,065 Views
h_eznt
Contributor III

I have tried these patches, at least those which were not in the 4.1.15 nxp branch already.

as a result , I get following kernel Oops when configuring aes in ipsec phase 2.

Internal error: Oops: 805 [#1] ARM
Modules linked in: xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 ipv6 ath9k_htc(O)b
CPU: 0 PID: 0 Comm: swapper Tainted: G O 4.1.27 #3
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: 8064fe38 ti: 80648000 task.ti: 80648000
PC is at v7_dma_clean_range+0x1c/0x34
LR is at dma_cache_maint_page+0xb4/0x160
pc : [<800184c4>] lr : [<800148e0>] psr: 00000113
sp : 80649ba0 ip : 80018510 fp : 8064a04c
r10: 8fb77000 r9 : 00000414 r8 : 00000000
r7 : 00000001 r6 : 00000000 r5 : 00000000 r4 : 00000000
r3 : 0000001f r2 : 00000020 r1 : 70000000 r0 : 70000000
Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 18f44059 DAC: 00000015
Process swapper (pid: 0, stack limit = 0x80648208)
Stack: (0x80649ba0 to 0x8064a000)
9ba0: 00000001 8fdb7000 00000000 88b9f940 8cf36810 00000000 8d249800 00000000
9bc0: 8d244b80 800154f8 80018510 8cf36810 800154ac 8037b9f8 00000001 00000000
9be0: 00000003 00000000 18d8627e 00000001 00000000 00000000 00000010 00000000
9c00: 88b9f980 88b9f9d0 88b9f940 8d244700 88b9f940 88d8627e 88b9f9d0 88b9f900
9c20: 88b9f9c0 8d53a800 00000036 7f2379e4 80649cd4 8cf1c380 8fda03b8 00000004
9c40: 00000060 88b9f900 0000000c 00000000 88b9f900 00000001 00000008 00000000
9c60: 80451c00 8d244700 8d244700 00000054 00000000 8d244700 8d53a800 00000001
9c80: 00000000 000005dc 8064aec8 8045a42c 80649cdc 803e05c0 8cd19380 8064b5ec
9ca0: 8d4ac800 8d244700 80649cdc 8d244700 80649cbc 8d244700 00000000 00000000
9cc0: 00000000 8d244700 8d2447b4 8044fb08 00000000 8044fcec 8064aec8 00000004
9ce0: 80000000 8064b502 00000000 890f6000 00000000 8044fad4 80649d04 00000001
9d00: 00000000 803fcaa0 88b9f800 88d8628e 88b9f800 88d8628e 00000000 803fceb4
9d20: 00000000 00000002 80000000 0381a802 8d4ac800 890f6000 00000000 803fca20
9d40: 88d86240 88d8628e 88d86240 00000000 00000000 8d244700 8d4ac800 803fae5c
9d60: 8d4ac800 8064b5cc 0000004e 88d86240 00000000 00000000 8d244700 803fb708
9d80: 00000050 00000000 80000000 00000002 8d4ac800 00000000 00000000 803fad3c
9da0: 80000000 8d244700 8064ebd4 8d4ac800 8d4ac848 00000008 8d4ac85c 803b182c
9dc0: 8064ebcc 00000000 0000000e 00000000 8d4ac800 8d244700 00000002 8d4ac85c
9de0: 00000066 8d244700 00000003 00000000 8d244700 9099e0e0 00000001 00000000
9e00: 8d4ac800 803b4528 00000001 00000000 8d244700 803b502c 00000002 00000000
9e20: 9099e0e0 802fa870 ffffffe0 00000001 061c1380 00000040 00000010 8d4ac830
9e40: 8cd62000 02000022 0000001c 00000000 00000002 00000000 00000001 00000000
9e60: 00000000 00000000 8d4accb8 00000040 8d4acc60 8d4acd30 40002710 00004c4b
9e80: 00989680 8d4acd30 00000001 00000040 0000012c 80683240 ffff9822 80649eb0
9ea0: 80649eb8 803b4c5c 80649ed8 80656290 80649eb0 80649eb0 80649eb8 80649eb8
9ec0: 00000000 00000000 80684c8c 80684c80 80684c80 80655da0 00000100 00000003
9ee0: 00000003 8002bf70 00000000 8c14e400 80652090 0000000a ffff9821 00200000
9f00: 412fc09a 8065d800 00000000 00000000 00000096 8c006000 412fc09a 90802100
9f20: 00000000 8002c324 8065d800 80050d98 9080210c 00000096 00000086 8064a350
9f40: 80649f68 80009324 8000fbdc 8000fbe0 60000013 ffffffff 80649f9c 00000001
9f60: 8fffc280 800120c0 00000007 806569c0 80649fb8 800189c0 8064a0a0 8064a0a0
9f80: 8068337b 8068337b 00000001 412fc09a 8fffc280 00000000 000af9f4 80649fb0
9fa0: 8000fbdc 8000fbe0 60000013 ffffffff 00000000 8004ab04 806834c0 80611bf0
9fc0: ffffffff ffffffff 80611670 00000000 00000000 806414f0 00000000 80683694
9fe0: 8064a050 806414ec 80650f48 10004059 00000000 10008078 00000000 00000000
[<800184c4>] (v7_dma_clean_range) from [<800148e0>] (dma_cache_maint_page+0xb4/0x160)
[<800148e0>] (dma_cache_maint_page) from [<800154f8>] (arm_dma_sync_single_for_device+0x4c/0x60)
[<800154f8>] (arm_dma_sync_single_for_device) from [<8037b9f8>] (aead_givencrypt+0x508/0xb10)
[<8037b9f8>] (aead_givencrypt) from [<7f2379e4>] (esp_output+0x348/0x4d8 [esp4])
[<7f2379e4>] (esp_output [esp4]) from [<8045a42c>] (xfrm_output_resume+0x15c/0x3cc)
[<8045a42c>] (xfrm_output_resume) from [<8044fb08>] (__xfrm4_output+0x34/0x38)
[<8044fb08>] (__xfrm4_output) from [<8044fcec>] (xfrm4_output+0x8c/0x9c)
[<8044fcec>] (xfrm4_output) from [<803fcaa0>] (ip_forward_finish+0x80/0x9c)
[<803fcaa0>] (ip_forward_finish) from [<803fceb4>] (ip_forward+0x3f8/0x484)
[<803fceb4>] (ip_forward) from [<803fae5c>] (ip_rcv_finish+0x120/0x2f4)
[<803fae5c>] (ip_rcv_finish) from [<803fb708>] (ip_rcv+0x2a4/0x3cc)
[<803fb708>] (ip_rcv) from [<803b182c>] (__netif_receive_skb_core+0x5b0/0x998)
[<803b182c>] (__netif_receive_skb_core) from [<803b4528>] (netif_receive_skb_internal+0x24/0x60)
[<803b4528>] (netif_receive_skb_internal) from [<803b502c>] (napi_gro_receive+0x78/0xa4)
[<803b502c>] (napi_gro_receive) from [<802fa870>] (fec_enet_rx_napi+0x424/0xc78)
[<802fa870>] (fec_enet_rx_napi) from [<803b4c5c>] (net_rx_action+0xe4/0x2a4)
[<803b4c5c>] (net_rx_action) from [<8002bf70>] (__do_softirq+0xd8/0x228)
[<8002bf70>] (__do_softirq) from [<8002c324>] (irq_exit+0xa8/0xf4)
[<8002c324>] (irq_exit) from [<80050d98>] (__handle_domain_irq+0x50/0xa0)
[<80050d98>] (__handle_domain_irq) from [<80009324>] (gic_handle_irq+0x20/0x58)
[<80009324>] (gic_handle_irq) from [<800120c0>] (__irq_svc+0x40/0x54)
Exception stack(0x80649f68 to 0x80649fb0)
9f60: 00000007 806569c0 80649fb8 800189c0 8064a0a0 8064a0a0
9f80: 8068337b 8068337b 00000001 412fc09a 8fffc280 00000000 000af9f4 80649fb0
9fa0: 8000fbdc 8000fbe0 60000013 ffffffff
[<800120c0>] (__irq_svc) from [<8000fbe0>] (arch_cpu_idle+0x30/0x3c)
[<8000fbe0>] (arch_cpu_idle) from [<8004ab04>] (cpu_startup_entry+0xe4/0x13c)
[<8004ab04>] (cpu_startup_entry) from [<80611bf0>] (start_kernel+0x340/0x3ac)
Code: e3a02004 e1a02312 e2423001 e1c00003 (ee070f3a)
---[ end trace 839f0387a8e09c55 ]---
Kernel panic - not syncing: Fatal exception in interrupt
---[ end Kernel panic - not syncing: Fatal exception in interrupt

2,065 Views
utkarsh_gupta
NXP Employee
NXP Employee

Have you made any other changes apart from the patches to the kernel? Is this reproducible directly after applying the patches on 4.1.15 release on 6Q board?

0 Kudos

2,065 Views
h_eznt
Contributor III

Great!

I will the patches and I give you feedback as soon as I can.

0 Kudos

2,065 Views
utkarsh_gupta
NXP Employee
NXP Employee

Hi, I was able to reproduce the problem finally with all the inputs provided and I see that there might be few patches that might need to apply to make it work.

Please apply the attached patches and let me know if that fixes the issue at your side too.

I have also attached the logs for external client, mx6q encrypting packets and vpn server.

Thanks,

Utkarsh

2,064 Views
utkarsh_gupta
NXP Employee
NXP Employee

Thank you for all the information. I burned an SD card with .sd image however I couldnt boot it on the baord. I burned it with following commands.

sudo dd if=<>.sd of=/dev/sdX bs=512 && sync

0 Kudos

2,065 Views
h_eznt
Contributor III

Regarding the  VPN-Server configuration on Ubuntu 14.04, you will need the following:

  • install racoon:  sudo apt-get install ipsec-tools
  • and stop it:  sudo /etc/racoon stop
  • configure one interface (for example eth0)  of  the VPN server with ip 192.168.3.1/24

cat /etc/interfaces
 
  auto eth0
  iface eth0 inet static
  address 192.168.3.1
  netmask 255.255.255.0

  • and restart networking: sudo service networking restart
  • activate ipv4 routing:  sudo su - ; echo 1 > /proc/sys/net/ipv4/ip_forward
  • configure another interface of the VPN server with the  ip 10.0.3.1/24

                This one can be configured manually:

                    sudo ifconfig <yourIfaceName> 10.0.3.1/24;

                    sudo ifconfig <yourIfaceName> up ;

              

  •   maybe it is wise to deactivate the NetworkManager: sudo service network-manager stop
  • move the  attached  file "racoon.conf.aes"  in the  /etc/racoon/ directory of the  VPN-Server
  • add 2 lines  in the /etc/racoon/psk.txt file  of the VPN -Server as follow

grs.garderos.com     garderos
192.168.3.1    garderos
 

 

  • start racoon in foreground with the following commands

   sudo su - ; setkey -PF ; sleep 2 ; racoon -ddddFf /etc/racoon/racoon.conf.aes

 

Policies don't need to be configured there, they are automatically generated by racoon when the VPN-client connects

Now you are done.

  • You can start to ping from client and check the packets coming in the VPN-gateway with tcpdump: tcpdump -an -i any
0 Kudos

2,065 Views
utkarsh_gupta
NXP Employee
NXP Employee

In addition can you describe if my understanding of your network configuration is correct. If not can you modify and describe it more clearly.

pastedImage_0.png

I would need help in creating a VPN server. Should it be through a linux machine running racoon (if yes I would need the configuration files) or is it a VPN through dd-wrt.

Thanks,

Utkarsh

0 Kudos

2,065 Views
h_eznt
Contributor III

Your network setup is exactly like mine and you will be able to reproduce the issue with it.

It is not mandatory to have a separate machine on the 10.0.3.0/24 network: pinging from the external test client to the ip 10.0.3.1 on the VPN gateway is sufficient. Please note that you really need an external test client, source pinging from the i.MX6Q will not reproduce the issue (because in this case the packet is directly processed by the kernel without being switched over the eth0 interface).

Basically, you can use whatever VPN-server you want. I would suggest to use an ubuntu  14.04 machine with racoon, since I am not an expert in dd-wrt.

I will post some configuration files for racoon on ubuntu 14.04  in the course of the day.

0 Kudos

2,065 Views
utkarsh_gupta
NXP Employee
NXP Employee

the sd card bootable image seems to be corrupted. I am unable to extract it. Can you send that again. thanks.

tar -xJvf imx6q-aes.sd.xz

xz: (stdin): File format not recognized

tar: Child returned status 1

tar: Error is not recoverable: exiting now

0 Kudos

2,065 Views
h_eznt
Contributor III

Ok, something went wrong with the compression. Sorry for that.
Now it has been corrected, please use the following new link:

https://extranet.garderos.com/files/?v=share/4EA09E8B02294413928527BB2CCADB06&c=1

password is: nxp-freescale

0 Kudos