Can't load SEC Device Drivers using LSDK 20.12 on FRWY-LS1046A board

[PedroRamos]

I'm trying to follow this tutorial  to to hardware offloading with OpenSSL on my FRWY-LS1046A board. My goal is to test the difference in performance for crypto operations with SEC drivers loaded.

For all my attempts, I used LSDK 20.12 to build the images and write them into the SD card that goes inside the board.

Attempts:

1. Tried following the documentation talking about automatically building both OpenSSL and cryptodev . However, upon trying to load "caamalg", the first step to test the hardware offload, I received:

$ modprobe caamalg
modprobe: FATAL: Module caamalg not found in directory /lib/modules/5.4.47

2. Then, I read the LSDK 20.12 documentation section about SEC , and made sure that the default kernel config file that the LSDK generates for the kernel match the options that need to be set according to the "Kernel Configuration" subsection. After relating these kernel options to their respective "CONFIG_*" fields in the .config file, I concluded that the only missing options were: "CONFIG_COMPILE_TEST=y" (prerequisite) and "CONFIG_CRYPTO_DEV_FSL_DPAA2_CAAM=m". In order to not mess up any other important kernel options, I added these options as a fragment to be added to the default kernel configuration. The building process followed as such:

source setup.env
flex-builder docker
source setup.env
flex-builder -c linux -a arm64 -m ls1046afrwy -B fragment:mod_fragment.config
flex-builder -c cryptodev_linux -a arm64 # This does nothing as cryptodev was already built by the previous instruction, but still
flex-builder -c openssl -a arm64
flex-builder -i merge-component -a arm64
flex-builder -i mkbootpartition -a arm64
flex-builder -i mkfw -m ls1046afrwy -b sd

After exiting docker, and as root, I wrote the images on my micro-SD as such:

source setup.env
flex-installer -i pf -d /dev/sdc
cd build/images
flex-installer -r rootfs_lsdk2012_ubuntu_main_arm64.tgz -b bootpartition_LS_arm64_lts_5.4.tgz -f firmware_ls1046afrwy_uboot_sdboot.img -d /dev/sdc

But even so, after booting the OS on my board, "caamalg" was still not found.

Questions:

1. What am I doing wrong?

2. I read [here](https://community.nxp.com/t5/Layerscape/LSDK-Sec-Engine-Integration/m-p/1195386) that:

in the default Linux Kernel configuration file, these SEC related Kernel options have already been configured as built-in mode, so "modprobe caamalg" is not needed.

While it is true that in my first attempt I managed to "modprobe cryptodev" and test openssl using the engine, I noticed almost no performance difference when doing "openssl test" before and after loading "cryptodev". I also could not run the suggested openssl test:

$ openssl speed -evp AES128-SHA -elapsed
speed: AES128-SHA is an unknown cipher or digest

What is happening in this case? Do I have the optimization already built-in and this openssl command is not correct, or is it something else?

Attached to this question is the ".config" file generated on my second attempt and the last one I used.

[lucasgm]

I'm having the same issue and I don't think the solution in this thread is the correct one. The last command proposed by the solution post doesn't use the -elapsed argument. Check what openssl says about this parameter:

-elapsed

Measure time in real time instead of CPU time. It can be useful when testing speed of hardware engines.

Using the -elapsed parameters makes openssl speed report the lower performance again.

Is there any other solution for this?

Best regards,

[yipingwang]

I checked your Kernel configuration file, the following Linux Kernel options have been configured as built-in mode.

CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API=y

CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=y

CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API=y

No need to insert these Linux Kernel modules caamalg, caamhash, caam_pkc.

Please check openssl engine before and after inserting module cryptodev.

# openssl engine

(dynamic) Dynamic engine loading support

# modprobe cryptodev

# openssl engine

(cryptodev) BSD cryptodev engine

(dynamic) Dynamic engine loading support

It seems that the algorithm name addressed in the document is incorrect, please use the following command.

# openssl speed -evp aes-128-cbc -elapsed

[PedroRamos]

Thanks for the clarification about the necessity for the modules. I checked the openssl engine as you suggested, but the output was different. Is this OK?

root@localhost:~# openssl engine
(dynamic) Dynamic engine loading support
root@localhost:~# modprobe cryptodev
root@localhost:~# openssl engine
(devcrypto) /dev/crypto engine
(dynamic) Dynamic engine loading support

 Also, while running the test before and after the module loading, the results with cryptodev on were a little bit worse (?). At least according to openssl stating that:

The 'numbers' are in 1000s of bytes per second processed.

root@localhost:~# openssl engine
(dynamic) Dynamic engine loading support
root@localhost:~# openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 59126833 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 33542510 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 12082639 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 3341674 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 439326 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 220081 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d  10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr) 
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600e
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc     315343.11k   715573.55k  1031051.86k  1140624.73k  1199652.86k  1201935.70k
root@localhost:~# modprobe cryptodev
root@localhost:~# openssl engine
(devcrypto) /dev/crypto engine
(dynamic) Dynamic engine loading support
root@localhost:~# openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 252918 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 253014 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 241738 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 198395 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 78924 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 50127 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d  10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr) 
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600e
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc       1348.90k     5397.63k    20628.31k    67718.83k   215515.14k   273760.26k

[yipingwang · Accepted solution]

Please refer to my result on LS1046ARDB.

root@localhost:~# openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 65562785 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 37205043 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 13419270 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 3713783 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 483317 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 244278 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d 10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -I/home/jenkins/ci/lsdk/master/all/build/apps/components_LS_arm64_ubuntu/usr/local/include
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 349668.19k 793707.58k 1145111.04k 1267637.93k 1319777.62k 1334083.58k
root@localhost:~# find / -name cryptodev.ko
/boot/modules/5.4.47/extra/cryptodev.ko
root@localhost:~# insmod /boot/modules/5.4.47/extra/cryptodev.ko
root@localhost:~# openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 296559 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 299671 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 266576 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 210612 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 92367 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 55655 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d 10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -I/home/jenkins/ci/lsdk/master/all/build/apps/components_LS_arm64_ubuntu/usr/local/include
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 1581.65k 6392.98k 22747.82k 71888.90k 252223.49k 303950.51k
root@localhost:~# openssl speed -evp aes-128-cbc -engine cryptodev
invalid engine "cryptodev"
281473327998592:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(/usr/local/lib/engines-1.1/cryptodev.so): /usr/local/lib/engines-1.1/cryptodev.so: cannot open shared object file: No such file or directory
281473327998592:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
281473327998592:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414:
281473327998592:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:334:id=cryptodev
281473327998592:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): libcryptodev.so: cannot open shared object file: No such file or directory
281473327998592:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
281473327998592:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414:
Doing aes-128-cbc for 3s on 16 size blocks: 298903 aes-128-cbc's in 0.12s
Doing aes-128-cbc for 3s on 64 size blocks: 299974 aes-128-cbc's in 0.11s
Doing aes-128-cbc for 3s on 256 size blocks: 280208 aes-128-cbc's in 0.08s
Doing aes-128-cbc for 3s on 1024 size blocks: 210924 aes-128-cbc's in 0.10s
Doing aes-128-cbc for 3s on 8192 size blocks: 93296 aes-128-cbc's in 0.06s
Doing aes-128-cbc for 3s on 16384 size blocks: 55769 aes-128-cbc's in 0.02s
OpenSSL 1.1.1d 10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -I/home/jenkins/ci/lsdk/master/all/build/apps/components_LS_arm64_ubuntu/usr/local/include
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 39853.73k 174530.33k 896665.60k 2159861.76k 12738013.87k 45685964.80k
root@localhost:~#

[PedroRamos]

Thanks for the clarification of how to use the module. Is quite different from the documentation. Just for comparison, here are the results on my FRWY-LS1046A:

user@localhost:~$ openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 59134465 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 33540976 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 12081517 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 3341181 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 439070 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 220000 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d  10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr) 
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600e
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc     315383.81k   715540.82k  1030956.12k  1140456.45k  1198953.81k  1201493.33k
user@localhost:~$ sudo find / -name cryptodev.ko                                                                                                                                                            
/boot/modules/5.4.47/extra/cryptodev.ko
user@localhost:~$ sudo insmod /boot/modules/5.4.47/extra/cryptodev.ko                                                                                                                                       
[sudo] password for user: 
user@localhost:~$ openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 249199 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 249273 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 240816 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 194568 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 78937 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 49745 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d  10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr) 
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600e
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc       1329.06k     5317.82k    20549.63k    66412.54k   215550.63k   271674.03k
user@localhost:~$ openssl speed -evp aes-128-cbc -engine cryptodev
invalid engine "cryptodev"
281473086252672:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(/usr/local/lib/engines-1.1/cryptodev.so): /usr/local/lib/engines-1.1/cy
281473086252672:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
281473086252672:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414:
281473086252672:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:334:id=cryptodev
281473086252672:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): libcryptodev.so: cannot open shared object file: No y
281473086252672:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
281473086252672:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414:
Doing aes-128-cbc for 3s on 16 size blocks: 250532 aes-128-cbc's in 0.07s
Doing aes-128-cbc for 3s on 64 size blocks: 249712 aes-128-cbc's in 0.09s
Doing aes-128-cbc for 3s on 256 size blocks: 242782 aes-128-cbc's in 0.08s
Doing aes-128-cbc for 3s on 1024 size blocks: 195150 aes-128-cbc's in 0.11s
Doing aes-128-cbc for 3s on 8192 size blocks: 79874 aes-128-cbc's in 0.05s
Doing aes-128-cbc for 3s on 16384 size blocks: 50303 aes-128-cbc's in 0.03s
OpenSSL 1.1.1d  10 Sep 2019
built on: Thu Dec 10 23:22:02 2020 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) idea(int) blowfish(ptr) 
compiler: aarch64-linux-gnu-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600e
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      57264.46k   177572.98k   776902.40k  1816669.09k 13086556.16k 27472145.07k