How to test OpenSSL hardware offloading in iMX8MP

[lishaocheng]

I use iMX8MP EVK and imx-yocto-L5.10.52。

first ，test openssl ：

root@imx8mpevk:~# openssl speed sha256
Doing sha256 for 3s on 16 size blocks: 11536688 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 8664550 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 5114298 sha256's in 2.99s
Doing sha256 for 3s on 1024 size blocks: 1932152 sha256's in 3.00s
Doing sha256 for 3s on 8192 size blocks: 284675 sha256's in 3.00s
Doing sha256 for 3s on 16384 size blocks: 144224 sha256's in 3.00s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256           61529.00k   184843.73k   437879.69k   659507.88k   777352.53k   787655.34k

and then, test openssl with cryptodev engine:

root@imx8mpevk:~# modprobe cryptodev
[  151.419636] cryptodev: loading out-of-tree module taints kernel.
[  151.420713] cryptodev: driver 1.12 loaded.
root@imx8mpevk:~# openssl engine
(devcrypto) /dev/crypto engine
(dynamic) Dynamic engine loading support
root@imx8mpevk:~# openssl speed -engine devcrypto sha256
engine "devcrypto" set.
Doing sha256 for 3s on 16 size blocks: 11536289 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 8664651 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 5114791 sha256's in 3.00s
Doing sha256 for 3s on 1024 size blocks: 1917203 sha256's in 2.99s
Doing sha256 for 3s on 8192 size blocks: 284712 sha256's in 3.00s
Doing sha256 for 3s on 16384 size blocks: 144242 sha256's in 3.00s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256           61526.87k   184845.89k   436462.17k   656593.94k   777453.57k   787753.64k

The test results are the same！

Why doesn't cryptodev improve the performance of openssl?

How to test caam's performance boost to openssl?

[lishaocheng]

https://community.nxp.com/t5/i-MX-Processors/Why-is-AES-on-CAAM-slower-than-OpenSSL-w-o-CAAM/m-p/294...

Similar problem, also not solved

[lishaocheng · Accepted solution]

The test results show that the CPU usage drops significantly after using CAAM to accelerate OpenSSL.

1. no caam

# serial terminal
root@imx8mpevk:~# cat /proc/interrupts | grep jr
258:          3          0          0          0     GICv3 138 Level     30902000.jr
root@imx8mpevk:~# openssl speed sha256
Doing sha256 for 3s on 16 size blocks: 11486709 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 8664294 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 5114777 sha256's in 2.99s
Doing sha256 for 3s on 1024 size blocks: 1930810 sha256's in 3.00s
Doing sha256 for 3s on 8192 size blocks: 284695 sha256's in 3.00s
Doing sha256 for 3s on 16384 size blocks: 144240 sha256's in 3.00s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256           61262.45k   184838.27k   437920.71k   659049.81k   777407.15k   787742.72k
root@imx8mpevk:~# cat /proc/interrupts | grep jr
258:          3          0          0          0     GICv3 138 Level     30902000.jr

# ssh terminal
root@imx8mpevk:~# top
top - 11:08:22 up 43 min,  3 users,  load average: 0.08, 0.02, 0.01
Tasks: 211 total,   2 running, 209 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.7 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   5613.3 total,   5102.7 free,    409.3 used,    101.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5092.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1505 root      20   0    6220   4016   3524 R 100.0   0.1   0:06.37 openssl
     32 root      20   0       0      0      0 S   0.3   0.0   0:02.19 ksoftirqd/3
    688 root      20   0  388608   6960   4656 S   0.3   0.1   0:17.13 netopeer2-serve
    874 root      20   0    3152   2272   1796 S   0.3   0.0   0:00.20 dropbear
   1504 root      20   0    5200   2824   2140 R   0.3   0.0   0:00.10 top
      1 root      20   0  159664   8180   5744 S   0.0   0.1   0:03.70 systemd

2. use -engine devcrypto 

# serial terminal
root@imx8mpevk:~# modprobe cryptodev
[ 2772.203074] cryptodev: loading out-of-tree module taints kernel.
[ 2772.204336] cryptodev: driver 1.12 loaded.
root@imx8mpevk:~# cat /proc/interrupts | grep jr
258:          3          0          0          0     GICv3 138 Level     30902000.jr
root@imx8mpevk:~# openssl speed -engine devcrypto sha256
engine "devcrypto" set.
Doing sha256 for 3s on 16 size blocks: 11535971 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 8664262 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 5113400 sha256's in 3.00s
Doing sha256 for 3s on 1024 size blocks: 1916868 sha256's in 2.99s
Doing sha256 for 3s on 8192 size blocks: 284616 sha256's in 3.00s
Doing sha256 for 3s on 16384 size blocks: 144181 sha256's in 3.00s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256           61525.18k   184837.59k   436343.47k   656479.21k   777191.42k   787420.50k
root@imx8mpevk:~# cat /proc/interrupts | grep jr
258:         57          0          0          0     GICv3 138 Level     30902000.jr

# ssh terminal
root@imx8mpevk:~# top
top - 11:13:48 up 48 min,  3 users,  load average: 0.40, 0.16, 0.05
Tasks: 212 total,   2 running, 210 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   5613.3 total,   5101.4 free,    410.4 used,    101.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5091.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1537 root      20   0    6220   4064   3580 R  99.7   0.1   0:09.49 openssl
   1536 root      20   0    5204   2756   2072 R   0.7   0.0   0:00.11 top
    688 root      20   0  388608   6960   4656 S   0.3   0.1   0:19.22 netopeer2-serve
   1499 root      20   0       0      0      0 I   0.3   0.0   0:00.03 kworker/u8:3-events_freezable_power_
      1 root      20   0  159664   8180   5744 S   0.0   0.1   0:03.71 systemd

3. use -evp

# serial terminal
root@imx8mpevk:~# cat /proc/interrupts | grep jr
258:         57          0          0          0     GICv3 138 Level     30902000.jr
root@imx8mpevk:~# openssl speed -engine devcrypto -evp sha256
engine "devcrypto" set.
Doing sha256 for 3s on 16 size blocks: 68271 sha256's in 0.39s
Doing sha256 for 3s on 64 size blocks: 44507 sha256's in 0.14s
Doing sha256 for 3s on 256 size blocks: 43450 sha256's in 0.14s
Doing sha256 for 3s on 1024 size blocks: 39230 sha256's in 0.17s
Doing sha256 for 3s on 8192 size blocks: 22280 sha256's in 0.06s
Doing sha256 for 3s on 16384 size blocks: 14771 sha256's in 0.09s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256            2800.86k    20346.06k    79451.43k   236303.06k  3041962.67k  2688978.49k
root@imx8mpevk:~# cat /proc/interrupts | grep jr
258:     398125          0          0          0     GICv3 138 Level     30902000.jr

# ssh terminal
root@imx8mpevk:~# top
top - 11:18:54 up 53 min,  3 users,  load average: 0.15, 0.09, 0.04
Tasks: 212 total,   2 running, 210 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us, 29.9 sy,  0.0 ni, 44.5 id,  0.0 wa,  6.6 hi, 19.0 si,  0.0 st
%Cpu1  :  0.0 us,  4.3 sy,  0.0 ni, 95.0 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us, 16.3 sy,  0.0 ni, 83.0 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
%Cpu3  :  3.5 us, 31.0 sy,  0.0 ni, 64.1 id,  0.0 wa,  1.4 hi,  0.0 si,  0.0 st
MiB Mem :   5613.3 total,   5101.2 free,    410.6 used,    101.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5091.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1556 root      20   0    6220   4092   3604 R  54.2   0.1   0:04.61 openssl
    517 root     -51   0       0      0      0 S  19.7   0.0   0:01.22 irq/258-3090200
    516 root      20   0       0      0      0 S  16.2   0.0   0:01.33 30902000.jr-eng
   1557 root      20   0    5204   2716   2028 R   1.4   0.0   0:00.05 top
     32 root      20   0       0      0      0 S   0.7   0.0   0:02.74 ksoftirqd/3
    688 root      20   0  388608   6960   4656 S   0.7   0.1   0:21.29 netopeer2-serve
      1 root      20   0  159664   8180   5744 S   0.0   0.1   0:03.72 systemd

[Harvey021]

Hi @lishaocheng 

the options should be both engine and evp, and the multi, known as jr is one, if you do not specify it. 

Best regards

Harvey

[lishaocheng]

I have do it  , The test results are strange：

root@imx8mpevk:~# openssl speed -engine devcrypto -evp sha256
engine "devcrypto" set.
Doing sha256 for 3s on 16 size blocks: 68900 sha256's in 0.23s
Doing sha256 for 3s on 64 size blocks: 44369 sha256's in 0.17s
Doing sha256 for 3s on 256 size blocks: 43580 sha256's in 0.23s
Doing sha256 for 3s on 1024 size blocks: 39242 sha256's in 0.10s
Doing sha256 for 3s on 8192 size blocks: 22048 sha256's in 0.07s
Doing sha256 for 3s on 16384 size blocks: 14753 sha256's in 0.12s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256            4793.04k    16703.62k    48506.43k   401838.08k  2580245.94k  2014276.27k
root@imx8mpevk:~# openssl speed -engine devcrypto -evp sha256 -elapsed
engine "devcrypto" set.
You have chosen to measure elapsed time instead of user CPU time.
Doing sha256 for 3s on 16 size blocks: 68052 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 45175 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 44212 sha256's in 3.00s
Doing sha256 for 3s on 1024 size blocks: 39164 sha256's in 3.00s
Doing sha256 for 3s on 8192 size blocks: 22221 sha256's in 3.00s
Doing sha256 for 3s on 16384 size blocks: 14788 sha256's in 3.00s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256             362.94k      963.73k     3772.76k    13367.98k    60678.14k    80762.20k

[lishaocheng]

If I use  -evp , The test results are strange：

root@imx8mpevk:~# openssl speed -engine devcrypto -evp sha256
engine "devcrypto" set.
Doing sha256 for 3s on 16 size blocks: 68900 sha256's in 0.23s
Doing sha256 for 3s on 64 size blocks: 44369 sha256's in 0.17s
Doing sha256 for 3s on 256 size blocks: 43580 sha256's in 0.23s
Doing sha256 for 3s on 1024 size blocks: 39242 sha256's in 0.10s
Doing sha256 for 3s on 8192 size blocks: 22048 sha256's in 0.07s
Doing sha256 for 3s on 16384 size blocks: 14753 sha256's in 0.12s
OpenSSL 1.1.1k  25 Mar 2021
built on: Thu Mar 25 13:28:38 2021 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: aarch64-poky-linux-gcc  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=                      -fdebug-prefix-map=  -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256            4793.04k    16703.62k    48506.43k   401838.08k  2580245.94k  2014276.27k

Only a few hundred milliseconds per test。