i.MX ProcessorsのトピックEncryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Fri, 13 May 2016 08:57:00 GMT

Hardware impacted:

- imx6qsabresd BSP Board (MCIMX6Q-SDB)

- imx6 Solo custom board

Linux kernel versions:

- imx_4.1.15_1.0.0_ga,

- imx_3.10.17_1.0.0_ga

- and probably all others

Description of the Problem:

Encryption of IPSEC ESP-packets coming from eth0 ("fec" driver) fails with CAAM when encryption algorithm is AES or AES256 on Linux platforms.

The sample test setup is as follow:

an i.mx6 BSP board (MCIMX6Q-SDB) with 2 network interfaces:
- eth0: attached to the processor, handled in Linux by the fec driver
- eth1: a digitus USB2.0 Ethernet Adapter (Pegasus/Pegasus II USB Ethernet driver) connected to the OTG port.
The board runs the latest linux 4.1.15_1.0.0_ga yocto software with "ipsec-tools" and the necessary ipsec kernel modules .i.e.

CONFIG_XFRM_IPCOMP=m

CONFIG_NET_KEY=y

CONFIG_INET_AH=m

CONFIG_INET_ESP=m

CONFIG_INET_IPCOMP=m

CONFIG_INET_XFRM_TUNNEL=m

CONFIG_INET_TUNNEL=y

CONFIG_INET_XFRM_MODE_TRANSPORT=m

CONFIG_INET_XFRM_MODE_TUNNEL=m

CONFIG_INET_XFRM_MODE_BEET=m

The ipsec client on the board (ipsec-tools) is configured to encrypt traffic in an IPSEC-tunnel from eth0 to a VPN-server behind eth1: the ip traffic comes unencrypted via eth0 and goes out encrypted over eth1. This is a typical "road-warrior" scenario.
The esp encryption algorithm chosen is aes (or aes256)
In this case CAAM does not encrypt correctly the ESP packets and they are discarded by the VPN-server.
It doesn't matter which VPN-server is used, we have been trying with CISCO ASA, various Juniper gateways, PfSense, racoon, strongSwan: problem is always there.
We have been replacing ipsec-tools by strongSwan: the problem remains
We have been investigating this issue quiet intensively these last 6 months and found out that:
- changing the encryption algorithm from aes to 3des solves this issue
- deactivating caam also solves this issue
- switching cable and ip addresses of eth0 and eth1 also solves this issue
- using VLAN tagging on eth0 solves this issue

For different (good) reasons non of these workarounds are acceptable for us: we have a customer that needs to perform IPSEC AES encryption from eth0 to eth1. This issue seems to be present on all i.MX6 platforms.

Are there any workarounds other than the ones listed above for this issue ?

We can provide additional logging/configuration files/images if needed.

Attached is a short description of the IPSEC test-setup on the i.MX6 test board.

Original Attachment has been moved to: ipsec.sh

Original Attachment has been moved to: racoon.setkey.zip

Original Attachment has been moved to: racoon.conf.aes.zip

Original Attachment has been moved to: run-logs.tgz

Original Attachment has been moved to: setup.log.zip

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

Yuri — Tue, 17 May 2016 06:54:48 GMT

Hello,

Please try to check CAAM, using section 9 (Security) in "i.MX_Linux_User's_Guide.pdf".

http://www.nxp.com/webapp/Download?colCode=L4.1.15_1.1.0_LINUX_DOCS&Parent_nodeId=1337699481071706174845&Parent_pageType…

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Wed, 18 May 2016 07:48:10 GMT

Hi Yuri,

Thanks for your post. Unfortunately it does not help:

Chapter 9 of the "i.MX_Linux_User's_Guide.pdf" is related to the use of caam over cryptodev. cryptodev is the user-space interface to the Linux-Kernel cryptographic layers. In Linux, ESP-IPSEC encryption does not happen in user-space but only in kernel-mode. So cryptodev is not involved here.

Hubert

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

Yuri — Wed, 18 May 2016 08:04:13 GMT

Hello,

look slike You should use Scatterlist Crypto API.

https://www.kernel.org/doc/Documentation/crypto/api-intro.txt

Nevertheless, You can test it via cryprodev as mentioned yearlier.

Regards,

Yuri.

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Fri, 20 May 2016 07:10:19 GMT

Hi Yuri,

Thanks for the link. You are right that the Scatter-API is involved here.

The Linux CAAM driver is actually intensively using the Linux scatter-list API.

The Linux CAAM driver is delivered by NXP/Freescale.

The problem is that this Linux CAAM driver that you deliver fails under some conditions.

I am not the programmer of the CAAM driver, I though Freescale/NXP was. If I could fix this driver I would do it, but without deep knowledge of the CAAM hardware and debugging facilities, it is just impossible. That 's why I did report this issue, with the hope that I can get help on this.

I am addressing a very specific issue that happens only under certain conditions. Let me rephrase it briefly.

In Linux with the CAAM driver, when an unencrypted packet gets switched from eth0 (FEC) to eth1 and it matches an outgoing IPSEC policy involving AES encryption, it then gets wrongly encrypted.

So this issue involves IPSEC with AES (i.e. CAAM Hardware/CAAM driver) _AND_ packet switching (i.e. FEC ethernet controller/FEC driver) simultaneously. If you just test IPSEC (with AES encryption) without packet switching over eth0, you will no be able to reproduce the problem.

As for cryptodev, I will underline once more that these test-conditions can _NOT_ be reproduced with any cryptodev stuff because, in Linux, ipsec is always processed in kernel-mode and cryptodev is not involved. The Linux IPSEC implementation forwards the buffers to encrypt directly to the CAAM driver.

I don't know any Linux ipsec implementation based on cryptodev. Therefore, I don't understand what you expect me to do with cryptodev and user-space APIs ?

Regards,

Hubert

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Thu, 02 Jun 2016 19:53:36 GMT

Hi Hubert,

I have come close to replicating this issue. Please forward any configuration files you have and also the log of issues that you faced with accessing aes encryption using caam.

Thanks,

Utkarsh

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Fri, 03 Jun 2016 13:34:12 GMT

Added the file "Network_Schema.pdf" for a schematic description of the test setup.
Added the archive "run-logs.tgz" containing log files and traces
The ipsec racoon configuration on the i.mx6 is detailed in the attachment "setup.log.zip"
The network configuration of the i.mx6 is as follow:

root@imx6qsabresd:~# cat /etc/network/interfaces

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface

auto lo

iface lo inet loopback

# Wired or wireless interfaces

auto eth0

iface eth0 inet static

address 10.1.1.1

netmask 255.255.255.0

network 10.1.1.0

auto eth1

iface eth1 inet static

address 192.168.3.77

netmask 255.255.255.0

network 192.168.3.0

gateway 192.168.3.1

The compressed sd-card image of the MCIMX6Q-SDB BSP Board is available for download under the following URL:

https://extranet.garderos.com/files/?v=share/2B0D4DC9632B442A902AA1A482E34099

This URL is protected with a password: nxp-freescale

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Fri, 03 Jun 2016 17:37:05 GMT

I think you forgot to attach network schema and run logs files.

_________________________________________________

Sorry found it ni original post.

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Wed, 08 Jun 2016 18:56:20 GMT

Hi,

Can you also share the ipsec.sh and racoon.conf.aes scripts.

Thanks,

Utkarsh

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Thu, 09 Jun 2016 06:49:16 GMT

Added as attachment ipsec.sh, racoon.setkey and racoon.conf.aes :

/etc/racoon/ipsec.sh : ipsec start script on the i.MX6 - This script must always be called between 2 tests to ensure kernel policies are properly cleaned-up. Policies are defined in /etc/racoon/racoon.setkey
/etc/racoon/racoon.setkey : ipsec policies on i.MX6 device

Additionaly, all files of the i.MX6 device are available on the sd-card bootable image referenced above.

/etc/racoon/racoon.conf.aes : racoon configuration file on the VPN server (ip-address: 192.168.3.1)

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Thu, 09 Jun 2016 17:13:55 GMT

the sd card bootable image seems to be corrupted. I am unable to extract it. Can you send that again. thanks.

tar -xJvf imx6q-aes.sd.xz

xz: (stdin): File format not recognized

tar: Child returned status 1

tar: Error is not recoverable: exiting now

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Fri, 10 Jun 2016 04:56:49 GMT

In addition can you describe if my understanding of your network configuration is correct. If not can you modify and describe it more clearly.

I would need help in creating a VPN server. Should it be through a linux machine running racoon (if yes I would need the configuration files) or is it a VPN through dd-wrt.

Thanks,

Utkarsh

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Fri, 10 Jun 2016 07:36:46 GMT

Ok, something went wrong with the compression. Sorry for that.
Now it has been corrected, please use the following new link:

https://extranet.garderos.com/files/?v=share/4EA09E8B02294413928527BB2CCADB06&c=1

password is: nxp-freescale

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Fri, 10 Jun 2016 07:50:13 GMT

Your network setup is exactly like mine and you will be able to reproduce the issue with it.

It is not mandatory to have a separate machine on the 10.0.3.0/24 network: pinging from the external test client to the ip 10.0.3.1 on the VPN gateway is sufficient. Please note that you really need an external test client, source pinging from the i.MX6Q will not reproduce the issue (because in this case the packet is directly processed by the kernel without being switched over the eth0 interface).

Basically, you can use whatever VPN-server you want. I would suggest to use an ubuntu 14.04 machine with racoon, since I am not an expert in dd-wrt.

I will post some configuration files for racoon on ubuntu 14.04 in the course of the day.

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Fri, 10 Jun 2016 13:30:13 GMT

Regarding the VPN-Server configuration on Ubuntu 14.04, you will need the following:

install racoon: sudo apt-get install ipsec-tools
and stop it: sudo /etc/racoon stop
configure one interface (for example eth0) of the VPN server with ip 192.168.3.1/24

cat /etc/interfaces

auto eth0
iface eth0 inet static
address 192.168.3.1
netmask 255.255.255.0

and restart networking: sudo service networking restart
activate ipv4 routing: sudo su - ; echo 1 > /proc/sys/net/ipv4/ip_forward
configure another interface of the VPN server with the ip 10.0.3.1/24

This one can be configured manually:

sudo ifconfig <yourIfaceName> 10.0.3.1/24;

sudo ifconfig <yourIfaceName> up ;

maybe it is wise to deactivate the NetworkManager: sudo service network-manager stop
move the attached file "racoon.conf.aes" in the /etc/racoon/ directory of the VPN-Server
add 2 lines in the /etc/racoon/psk.txt file of the VPN -Server as follow

grs.garderos.com garderos
192.168.3.1 garderos

start racoon in foreground with the following commands

sudo su - ; setkey -PF ; sleep 2 ; racoon -ddddFf /etc/racoon/racoon.conf.aes

Policies don't need to be configured there, they are automatically generated by racoon when the VPN-client connects

Now you are done.

You can start to ping from client and check the packets coming in the VPN-gateway with tcpdump: tcpdump -an -i any

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Mon, 13 Jun 2016 22:33:19 GMT

Thank you for all the information. I burned an SD card with .sd image however I couldnt boot it on the baord. I burned it with following commands.

sudo dd if=<>.sd of=/dev/sdX bs=512 && sync

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Tue, 28 Jun 2016 21:47:40 GMT

Hi, I was able to reproduce the problem finally with all the inputs provided and I see that there might be few patches that might need to apply to make it work.

Please apply the attached patches and let me know if that fixes the issue at your side too.

I have also attached the logs for external client, mx6q encrypting packets and vpn server.

Thanks,

Utkarsh

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Wed, 29 Jun 2016 07:30:11 GMT

Great!

I will the patches and I give you feedback as soon as I can.

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

h_eznt — Mon, 01 Aug 2016 10:35:41 GMT

I have tried these patches, at least those which were not in the 4.1.15 nxp branch already.

as a result , I get following kernel Oops when configuring aes in ipsec phase 2.

Internal error: Oops: 805 [#1] ARM
Modules linked in: xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 ipv6 ath9k_htc(O)b
CPU: 0 PID: 0 Comm: swapper Tainted: G O 4.1.27 #3
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: 8064fe38 ti: 80648000 task.ti: 80648000
PC is at v7_dma_clean_range+0x1c/0x34
LR is at dma_cache_maint_page+0xb4/0x160
pc : [<800184c4>] lr : [<800148e0>] psr: 00000113
sp : 80649ba0 ip : 80018510 fp : 8064a04c
r10: 8fb77000 r9 : 00000414 r8 : 00000000
r7 : 00000001 r6 : 00000000 r5 : 00000000 r4 : 00000000
r3 : 0000001f r2 : 00000020 r1 : 70000000 r0 : 70000000
Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 18f44059 DAC: 00000015
Process swapper (pid: 0, stack limit = 0x80648208)
Stack: (0x80649ba0 to 0x8064a000)
9ba0: 00000001 8fdb7000 00000000 88b9f940 8cf36810 00000000 8d249800 00000000
9bc0: 8d244b80 800154f8 80018510 8cf36810 800154ac 8037b9f8 00000001 00000000
9be0: 00000003 00000000 18d8627e 00000001 00000000 00000000 00000010 00000000
9c00: 88b9f980 88b9f9d0 88b9f940 8d244700 88b9f940 88d8627e 88b9f9d0 88b9f900
9c20: 88b9f9c0 8d53a800 00000036 7f2379e4 80649cd4 8cf1c380 8fda03b8 00000004
9c40: 00000060 88b9f900 0000000c 00000000 88b9f900 00000001 00000008 00000000
9c60: 80451c00 8d244700 8d244700 00000054 00000000 8d244700 8d53a800 00000001
9c80: 00000000 000005dc 8064aec8 8045a42c 80649cdc 803e05c0 8cd19380 8064b5ec
9ca0: 8d4ac800 8d244700 80649cdc 8d244700 80649cbc 8d244700 00000000 00000000
9cc0: 00000000 8d244700 8d2447b4 8044fb08 00000000 8044fcec 8064aec8 00000004
9ce0: 80000000 8064b502 00000000 890f6000 00000000 8044fad4 80649d04 00000001
9d00: 00000000 803fcaa0 88b9f800 88d8628e 88b9f800 88d8628e 00000000 803fceb4
9d20: 00000000 00000002 80000000 0381a802 8d4ac800 890f6000 00000000 803fca20
9d40: 88d86240 88d8628e 88d86240 00000000 00000000 8d244700 8d4ac800 803fae5c
9d60: 8d4ac800 8064b5cc 0000004e 88d86240 00000000 00000000 8d244700 803fb708
9d80: 00000050 00000000 80000000 00000002 8d4ac800 00000000 00000000 803fad3c
9da0: 80000000 8d244700 8064ebd4 8d4ac800 8d4ac848 00000008 8d4ac85c 803b182c
9dc0: 8064ebcc 00000000 0000000e 00000000 8d4ac800 8d244700 00000002 8d4ac85c
9de0: 00000066 8d244700 00000003 00000000 8d244700 9099e0e0 00000001 00000000
9e00: 8d4ac800 803b4528 00000001 00000000 8d244700 803b502c 00000002 00000000
9e20: 9099e0e0 802fa870 ffffffe0 00000001 061c1380 00000040 00000010 8d4ac830
9e40: 8cd62000 02000022 0000001c 00000000 00000002 00000000 00000001 00000000
9e60: 00000000 00000000 8d4accb8 00000040 8d4acc60 8d4acd30 40002710 00004c4b
9e80: 00989680 8d4acd30 00000001 00000040 0000012c 80683240 ffff9822 80649eb0
9ea0: 80649eb8 803b4c5c 80649ed8 80656290 80649eb0 80649eb0 80649eb8 80649eb8
9ec0: 00000000 00000000 80684c8c 80684c80 80684c80 80655da0 00000100 00000003
9ee0: 00000003 8002bf70 00000000 8c14e400 80652090 0000000a ffff9821 00200000
9f00: 412fc09a 8065d800 00000000 00000000 00000096 8c006000 412fc09a 90802100
9f20: 00000000 8002c324 8065d800 80050d98 9080210c 00000096 00000086 8064a350
9f40: 80649f68 80009324 8000fbdc 8000fbe0 60000013 ffffffff 80649f9c 00000001
9f60: 8fffc280 800120c0 00000007 806569c0 80649fb8 800189c0 8064a0a0 8064a0a0
9f80: 8068337b 8068337b 00000001 412fc09a 8fffc280 00000000 000af9f4 80649fb0
9fa0: 8000fbdc 8000fbe0 60000013 ffffffff 00000000 8004ab04 806834c0 80611bf0
9fc0: ffffffff ffffffff 80611670 00000000 00000000 806414f0 00000000 80683694
9fe0: 8064a050 806414ec 80650f48 10004059 00000000 10008078 00000000 00000000
[<800184c4>] (v7_dma_clean_range) from [<800148e0>] (dma_cache_maint_page+0xb4/0x160)
[<800148e0>] (dma_cache_maint_page) from [<800154f8>] (arm_dma_sync_single_for_device+0x4c/0x60)
[<800154f8>] (arm_dma_sync_single_for_device) from [<8037b9f8>] (aead_givencrypt+0x508/0xb10)
[<8037b9f8>] (aead_givencrypt) from [<7f2379e4>] (esp_output+0x348/0x4d8 [esp4])
[<7f2379e4>] (esp_output [esp4]) from [<8045a42c>] (xfrm_output_resume+0x15c/0x3cc)
[<8045a42c>] (xfrm_output_resume) from [<8044fb08>] (__xfrm4_output+0x34/0x38)
[<8044fb08>] (__xfrm4_output) from [<8044fcec>] (xfrm4_output+0x8c/0x9c)
[<8044fcec>] (xfrm4_output) from [<803fcaa0>] (ip_forward_finish+0x80/0x9c)
[<803fcaa0>] (ip_forward_finish) from [<803fceb4>] (ip_forward+0x3f8/0x484)
[<803fceb4>] (ip_forward) from [<803fae5c>] (ip_rcv_finish+0x120/0x2f4)
[<803fae5c>] (ip_rcv_finish) from [<803fb708>] (ip_rcv+0x2a4/0x3cc)
[<803fb708>] (ip_rcv) from [<803b182c>] (__netif_receive_skb_core+0x5b0/0x998)
[<803b182c>] (__netif_receive_skb_core) from [<803b4528>] (netif_receive_skb_internal+0x24/0x60)
[<803b4528>] (netif_receive_skb_internal) from [<803b502c>] (napi_gro_receive+0x78/0xa4)
[<803b502c>] (napi_gro_receive) from [<802fa870>] (fec_enet_rx_napi+0x424/0xc78)
[<802fa870>] (fec_enet_rx_napi) from [<803b4c5c>] (net_rx_action+0xe4/0x2a4)
[<803b4c5c>] (net_rx_action) from [<8002bf70>] (__do_softirq+0xd8/0x228)
[<8002bf70>] (__do_softirq) from [<8002c324>] (irq_exit+0xa8/0xf4)
[<8002c324>] (irq_exit) from [<80050d98>] (__handle_domain_irq+0x50/0xa0)
[<80050d98>] (__handle_domain_irq) from [<80009324>] (gic_handle_irq+0x20/0x58)
[<80009324>] (gic_handle_irq) from [<800120c0>] (__irq_svc+0x40/0x54)
Exception stack(0x80649f68 to 0x80649fb0)
9f60: 00000007 806569c0 80649fb8 800189c0 8064a0a0 8064a0a0
9f80: 8068337b 8068337b 00000001 412fc09a 8fffc280 00000000 000af9f4 80649fb0
9fa0: 8000fbdc 8000fbe0 60000013 ffffffff
[<800120c0>] (__irq_svc) from [<8000fbe0>] (arch_cpu_idle+0x30/0x3c)
[<8000fbe0>] (arch_cpu_idle) from [<8004ab04>] (cpu_startup_entry+0xe4/0x13c)
[<8004ab04>] (cpu_startup_entry) from [<80611bf0>] (start_kernel+0x340/0x3ac)
Code: e3a02004 e1a02312 e2423001 e1c00003 (ee070f3a)
---[ end trace 839f0387a8e09c55 ]---
Kernel panic - not syncing: Fatal exception in interrupt
---[ end Kernel panic - not syncing: Fatal exception in interrupt

Re: Encryption of IPSEC ESP-packets coming from eth0 fails with CAAM when using AES

utkarsh_gupta — Thu, 11 Aug 2016 23:29:13 GMT

Have you made any other changes apart from the patches to the kernel? Is this reproducible directly after applying the patches on 4.1.15 release on 6Q board?