Hardware Offloading of NXP CAAM using OpenSSL in LS1046A

[sakthi_mgtech]

I am working with LS1046ARDB Evaluation Board to develop a network based application. I have started my project with Flex-builder(flexbuild_lsdk2108), Flex-installer(version: 1.13.2108) and Codewarrior IDE. Now i want to develop OPENSSL application with NXP CAAM Hardware Offloading feature. I have followed the steps in this document https://www.nxp.com/docs/en/user-guide/LSDKUG_Rev21.08.pdf  -> 9.1.1.3 Hardware Offloading with OpenSSL section. But i can't load the CAAM into OPENSSL so can you guide me to load and use the OPENSSL hardware offloading feature. I have posted my work below.

I have download the images from https://lsdk.github.io/components.html

boot_LS -> wget https://www.nxp.com/lgfiles/sdk/lsdk2108/boot_LS_arm64_lts_5.10.tgz

firmware image -> wget https://www.nxp.com/lgfiles/sdk/lsdk2108/firmware_ls1046ardb_sdboot.img

Rootfs -> wget https://www.nxp.com/lgfiles/sdk/lsdk2108/rootfs_lsdk2108_ubuntu_main_arm64.tgz

I have depolyed the image's in SD card through flex-installer:

$ flex-installer -i pf -d /dev/sdx (partition and format the target storage device)

$ sudo flex-installer -b boot_LS_arm64_lts_5.10.tgz -r rootfs_lsdk2108_ubuntu_main_arm64.tgz -f firmware_ls1046ardb_sdboot.img -d /dev/sdx

I have booted the board using SD card, by default caam driver is not enabled in linux kernel it was builded as a module.

NXP LSDK 2108 main (custom based on ubuntu 20.04)
localhost login: root
Password: root

root@localhost:~# dmesg | grep -i caam

root@localhost:~# modinfo caam

filename: /lib/modules/5.10.35/kernel/drivers/crypto/caam/caam.ko
author: Freescale Semiconductor - NMG/STC
description: FSL CAAM request backend
license: GPL
alias: of:N*T*Cfsl,sec4.0C*
alias: of:N*T*Cfsl,sec4.0
alias: of:N*T*Cfsl,sec-v4.0C*
alias: of:N*T*Cfsl,sec-v4.0
depends: error
intree: Y
name: caam
vermagic: 5.10.35 SMP preempt mod_unload modversions aarch64

root@localhost:~# modinfo cryptodev

filename: /lib/modules/5.10.35/extra/cryptodev.ko
license: GPL
description: CryptoDev driver
author: Nikos Mavrogiannopoulos <nmav@gnutls.org>
depends:
name: cryptodev
vermagic: 5.10.35 SMP preempt mod_unload modversions aarch64
parm: cryptodev_verbosity:0: normal, 1: verbose, 2: debug (int)

but these drivers are not available as module. It was builded as build-in drivers in kernel.

root@localhost:~# modinfo caamalg
modinfo: ERROR: Module caamalg not found.

root@localhost:~# modinfo caamhash
modinfo: ERROR: Module caamhash not found.

root@localhost:~# modinfo caam_pkc
modinfo: ERROR: Module caam_pkc not found.

I have verfied the kernel drivers below:

root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM
CONFIG_CRYPTO_DEV_FSL_CAAM=m
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_JR
CONFIG_CRYPTO_DEV_FSL_CAAM_JR=m
SH_APIocalhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_AHA
CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=y
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API
CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API=y
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API
CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API=y
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_TK_API
CONFIG_CRYPTO_DEV_FSL_CAAM_TK_API=y
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_TEST
# CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_TEST is not set
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_SM
CONFIG_CRYPTO_DEV_FSL_CAAM_SM=y
root@localhost:~# zcat /proc/config.gz | grep -iw CONFIG_CRYPTO_DEV_FSL_CAAM_JR_UIO
CONFIG_CRYPTO_DEV_FSL_CAAM_JR_UIO=m

root@localhost:~# openssl
openssl: symbol lookup error: openssl: undefined symbol: EVP_mdc2, version OPENSSL_1_1_0

By default openssl was throw an error, but if i run ./openssl in /usr/bin it was working

root@localhost:/usr/bin# ./openssl
OpenSSL> engine
(dynamic) Dynamic engine loading support
OpenSSL> version
OpenSSL 1.1.1f 31 Mar 2020
OpenSSL>

Now I load the caam module through modprobe, but it through error in caam_jr

root@localhost:~# modprobe -v caam
insmod /lib/modules/5.10.35/kernel/drivers/crypto/caam/error.ko
insmod /lib/modules/5.10.35/kernel/drivers/crypto/caam/caam.ko
root@localhost:~# [ 1790.786153] caam_jr 1730000.jr: failed to flush job ring 2

root@localhost:~# ls /dev/crypto
ls: cannot access '/dev/crypto': No such file or directory

root@localhost:~# cat /proc/interrupts | grep jr
78: 11 0 0 0 GICv2 103 Level 1710000.jr
79: 0 0 0 0 GICv2 104 Level 1720000.jr
80: 0 0 0 0 GICv2 105 Level fsl-jr0

root@localhost:/usr/bin# modprobe cryptodev

root@localhost:/usr/bin# ls /dev/crypto
/dev/crypto

root@localhost:/usr/bin# ./openssl engine
(dynamic) Dynamic engine loading support

root@localhost:/usr/bin# ./openssl engine
(dynamic) Dynamic engine loading support

I have faced few error in caam and cryptodev in dmesg after loading caam and cryptodev module, due to this may be CAAM offload is not happening or else, can you guide me to offload.

root@localhost:~# dmesg
[ 1790.522486] caam 1700000.crypto: Linux CAAM Queue I/F driver initialised
[ 1790.523182] caam 1700000.crypto: device ID = 0x0a11030100000000 (Era
[ 1790.523190] caam 1700000.crypto: job rings = 3, qi = 1
[ 1790.780457] caam algorithms registered in /proc/crypto
[ 1790.781646] caam 1700000.crypto: caam pkc algorithms registered in /proc/crypto
[ 1790.784495] caam 1700000.crypto: algorithms registered in /proc/crypto
[ 1790.784503] caam 1700000.crypto: registering rng-caam
[ 1790.784718] Device caam-keygen registered
[ 1790.786153] caam_jr 1730000.jr: failed to flush job ring 2
[ 1790.791749] caam_jr: probe of 1730000.jr failed with error -5
[ 1790.802356] fsl-jr-uio 1730000.jr: UIO device full name fsl-jr0 initialized
[ 1872.560845] cryptodev: loading out-of-tree module taints kernel.
[ 1872.562580] cryptodev: driver 1.12 loaded.

I am not getting the cryptodev engine in OPENSSL like below

Can you provide a neccessary details to fix this issue. Looking forward for your response.

Thanks in advance.

[sakthi_mgtech · Accepted solution]

Hi  Hector_Villarruel,

This OpenSSL hardware offloading issue is due to linker path. I have updated the file: /etc/ld.so.conf.

By default, openssl through below error in LSDK_21.08

user@localhost:~$ openssl  

openssl: symbol lookup error: openssl: undefined symbol: EVP_mdc2, version OPENSSL_1_1_0

Check the linker path’s:

user@localhost:~$ ldd /usr/local/bin/openssl

linux-vdso.so.1 (0x0000ffff860cf000) 

libssl.so.1.1 => /lib/aarch64-linux-gnu/libssl.so.1.1 (0x0000ffff85f33000) 

libcrypto.so.1.1 => /lib/aarch64-linux-gnu/libcrypto.so.1.1 (0x0000ffff85ca5000) 

libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff85c74000) 

libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff85b01000) 

/lib/ld-linux-aarch64.so.1 (0x0000ffff8609f000) 

libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff85aed000)

If the binary is linking with the original library from /usr/lib, then it may be necessary to adjust the linker paths.  

Put /usr/local/lib in a line before /usr/lib inside /etc/ld.so.conf and then update the linker cache. 

user@localhost:~$ cat /etc/ld.so.conf 

include /etc/ld.so.conf.d/*.conf

Update the above /etc/ld.so.conf as below:

user@localhost:~$ cat /etc/ld.so.conf 

# libc default configuration 

/usr/local/lib  

# Multiarch support 

/usr/local/lib/aarch64-linux-gnu 

/lib/aarch64-linux-gnu 

/usr/lib/aarch64-linux-gnu  

/usr/lib/aarch64-linux-gnu/libfakeroot 

user@localhost:~$ sudo ldconfig

user@localhost:~$ ldd /usr/local/bin/openssl 

linux-vdso.so.1 (0x0000ffff990c3000) 

libssl.so.1.1 => /usr/local/lib/libssl.so.1.1 (0x0000ffff98f23000) 

libcrypto.so.1.1 => /usr/local/lib/libcrypto.so.1.1 (0x0000ffff98c71000) 

libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff98c40000) 

libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff98acd000) 

/lib/ld-linux-aarch64.so.1 (0x0000ffff99093000) 

libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff98ab9000)

Now CAAM hardware will offloaded into OpenSSL:

user@localhost:~$ sudo modprobe cryptodev
[sudo] password for user:

user@localhost:~$ ls /dev/crypto
/dev/crypto

user@localhost:~$ openssl engine
(devcrypto) /dev/crypto engine
(dynamic) Dynamic engine loading support

Verify the CAAM offloading:

Hardware operations can be monitored with the interrupt counters for CAAM JR and QI (DPAA1 and DPAA2) interfaces.

user@localhost:~$ cat /proc/interrupts | grep jr
78: 20 0 0 0 GICv2 103 Level 1710000.jr
79: 0 0 0 0 GICv2 104 Level 1720000.jr
80: 0 0 0 0 GICv2 105 Level fsl-jr0

user@localhost:~$ openssl enc -aes-256-cfb -k secret -P -md sha1
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
salt=233157F098D1B0E8
key=D2B0C0C088AA7DAA6CF37CC856E283E3B0C3DD2BE155688DE18593EB3F997776
iv =A4E2C86E0DBA1211B5A4B02EBEFF6DC4

user@localhost:~$ cat /proc/interrupts | grep jr
78: 54 0 0 0 GICv2 103 Level 1710000.jr
79: 69 0 0 0 GICv2 104 Level 1720000.jr
80: 0 0 0 0 GICv2 105 Level fsl-jr0

[Hector_Villarruel]

Hello,

The application that you want to do it requires an Application Solutions Kit which is available to purchase, In the following link you can find the "Layerscape 1046A Processor Broadband Home Router Application Solutions Kit"

​​​​​​​https://www.nxp.com/support/support/nxp-engineering-services/vortiqa-application-solutions-ki...

Best Regards,

Hector Villarruel