ssl hardware offloading in nginx issue

BaiYanfu · ‎02-11-2022

Hello, I am trying to test ssl hardware offloading in nginx on lx2160ardb with lsdk 21.08, nginx config like followings:

	ssl_protocols TLSv1.1; # Dropping SSLv3, ref: POODLE
	ssl_certificate /etc/nginx/certs/cert.pem;
	ssl_certificate_key /etc/nginx/certs/key.pem;
	ssl_ciphers AES128-SHA:AES256-SHA;
	ssl_prefer_server_ciphers on;

After modprobe dpaa2_caam and modprobe cryptodev, restart nginx, then curl from client, will report error: "sslv3 alert bad record mac", like followings:

baiyanfu@ADS-01:~$ curl -v -k https://172.18.53.70
* Rebuilt URL to: https://172.18.53.70/
* Hostname was NOT found in DNS cache
*   Trying 172.18.53.70...
* Connected to 172.18.53.70 (172.18.53.70) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS alert, Server hello (2):
* error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert bad record mac
* Closing connection 0
curl: (35) error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert bad record mac

And from localhost, curl is ok.

root@localhost:~# curl -k https://127.0.0.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

But I tested with wget for 1000 times, It will need 100s, while when no cryptodev, it only need 20s.

root@localhost:~/ssl/wget# cat wget.sh
for i in `seq 1 1000`; do
	wget --no-check-certificate https://127.0.0.1 >/dev/null 2>&1
done
root@localhost:~/ssl/wget# time ./wget.sh

real	1m43.516s
user	0m9.217s
sys	0m13.221s

So, I have two questions:

1. If I want to curl from another client, how should I do ?

2. Why wget became slower with cryptodev than no cryptodev ?

Thanks!

yipingwang · ‎02-23-2022

Thanks for the openssl measurement.

From the openssl test result last row.

16 3.56 Gbps 28.47 Gbps 10.02 Mbps

...

65536 13.28 Gbps 3.37 Gbps 26.85 Gbps

As mentioned in Feb 17. "~3Gbps per AESA engine. LX2160A has 8 of the AES engine." That matched what the SEC engine can do and customer can observe the optimized performance.

Also, it confirm the "inefficiency" for SEC engine with small frame/packet size operation.

So back to the original nginx question.

1) serial wget for index.html is not a good way to test crypto acceleration. To get good performance, customer needs to use wget for a large file, like 1GB or larger files in parallel. customer should able to see 8x performance gain compare to serial wget command for testing.

2) We do not really optimize user space applications, like NGINX. We only able the base layer for customer to optimize base on their application.

NGINX is not tested or benchmark by the LSDK release.

The enablement is provided in the LSDK user Guide, section ""9.1.1.3 Hardware Offloading with OpenSSL ##### TLS 1.1 and TLS 1.2 Offloading in Nginx Server Nginx does not use any openssl engines by default. If an engine is to be used, including cryptodev, it must be explicitly listed in nginx configuration file. Here is a fragment of nginx configuration file that activates cryptodev and allows hardware offloading of TLS 1.1 and TLS 1.2 record layer protocol:

/etc/nginx/nginx.conf:

ssl_engine cryptodev;

worker_processes 4;

worker_cpu_affinity 0001 0010 0100 1000; #for 4 Core CPU; For 2 Core CPU worker_cpu_affinity 01 10; ...

# HTTPS server

#

server {

listen 443;

server_name localhost;

ssl on;

ssl_certificate server.crt;

NXP Semiconductors

Linux user space

Layerscape

ssl_certificate_key server.key;

ssl_session_timeout 5m;

ssl_protocols TLSv1.1;

ssl_ciphers AES128-SHA:AES256-SHA;

ssl_prefer_server_ciphers on;

location / {

root /var/www/localhost/html;

index index.html index.htm;

}

...

Worker processes and affinity should be set according to the number of CPU cores available on the platform. Refer to nginx documentation for more details.

#####

If customer needs help to optimize a non-supported application like nginx, they may wants to use professional/paid service. Keep in mind optimization is depends on customer use cases. As they already observed.

1) performance is efficient for large frame/file.

2) https is a synchronous protocol. It is not efficient in nature for multitasking/parallel operation. Performance gain can achieve by convert it to asynchronous. BUt this may break nginx/https protocol.

3) customer already discovered the best performance for offload the "aes-128-cbc" is 26.85 Gbps. That set the ceiling of performance for offload the crypto operation to caam.

元の投稿で解決策を見る

yipingwang · ‎02-14-2022

Probably the problem is caused by nginx which you used. I download nginx-1.6.2.tar.gz and build it on the target board, I got normal result. Please refer to my following procedure.

1. Build nginx-1.6.2 on lx2160ardb.

$ wget http://nginx.org/download/nginx-1.6.2.tar.gz

$ tar -xzvf nginx-1.6.2.tar.gz

$ cd nginx-1.6.2

$ ./configure

a. In file objs/Makefile delete “-Werror” in CFLAGS

b. In file src/os/unix/ngx_user.c delete the following line

/* work around the glibc bug */

cd.current_salt[0] = ~salt[0];

$ make

$ make install

2. Set up nginx on lx2160ardb

root@localhost:~# ls-addni dpmac.18
Created interface: eth0 (object:dpni.2, endpoint: dpmac.18)
root@localhost:~# ifconfig eth0 100.1.1.99
root@localhost:~# ping 100.1.1.100
PING 100.1.1.100 (100.1.1.100) 56(84) bytes of data.
64 bytes from 100.1.1.100: icmp_seq=1 ttl=64 time=0.317 ms
64 bytes from 100.1.1.100: icmp_seq=2 ttl=64 time=0.100 ms
64 bytes from 100.1.1.100: icmp_seq=3 ttl=64 time=0.116 ms
root@localhost:~# modprobe dpaa2_caam
[ 116.474026] fsl_mc_dprc dprc.1 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 512
root@localhost:~# modprobe cryptodev
root@localhost:~# lsmod
Module Size Used by
cryptodev 57344 0
dpaa2_caam 114688 0
caamhash_desc 16384 1 dpaa2_caam
caamalg_desc 40960 1 dpaa2_caam
error 24576 3 caamalg_desc,caamhash_desc,dpaa2_caam
libdes 24576 1 dpaa2_caam
xt_CHECKSUM 16384 1
ipt_REJECT 16384 2
nf_reject_ipv4 16384 1 ipt_REJECT
ip6table_mangle 16384 1
ip6table_nat 16384 1
xt_addrtype 16384 2
ip6table_filter 16384 1
ip6_tables 32768 3 ip6table_filter,ip6table_nat,ip6table_mangle
xt_conntrack 16384 2
mali_dp 73728 0
root@localhost:~# /usr/local/nginx/sbin/nginx

3. On the other target board execute curl command.

root@localhost:~/linuxptp# curl -k http://100.1.1.99
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

4. On LX2160ARDB target board running nginx

root@localhost:~# cat wget.sh
for i in `seq 1 1000`; do
wget --no-check-certificate http://127.0.0.1 >/dev/null 2>&1
done
root@localhost:~# time ./wget.sh

real 0m8.864s
user 0m2.757s
sys 0m5.442s

BaiYanfu · ‎02-14-2022

Thanks for your reply.

It seems like you tested with https instead of https.

we need to configure nginx with --with-http_ssl_module, and then configure ssl in nginx.conf like followings:

    # HTTPS server
    #
    server {
        listen       443 ssl;
        server_name  localhost;

        #ssl_certificate      cert.pem;
        #ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        #ssl_protocols TLSv1 TLSv1.1 ; # Dropping SSLv3, ref: POODLE
        ssl_certificate /etc/nginx/certs/cert.pem;
        ssl_certificate_key /etc/nginx/certs/key.pem;
        ssl_ciphers AES128-SHA:AES256-SHA;
        ssl_prefer_server_ciphers on;

        location / {
            root   html;
            index  index.html index.htm;
        }
    }

then the issued should be reproduced.

Thanks

yipingwang · ‎02-15-2022

Would you please send your following files to me to do more verification?

/etc/nginx/certs/cert.pem

/etc/nginx/certs/key.pem

BaiYanfu · ‎02-15-2022

OK, I generate with openssl, reference to v21.08 user guide section 9.1.1.3.

$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes

root@localhost:~/ssl/tls# cat /etc/nginx/certs/cert.pem
-----BEGIN CERTIFICATE-----
MIIDkzCCAnugAwIBAgIUeOgs9L7iQ/7hBd08bcudfW1GzqIwDQYJKoZIhvcNAQEL
BQAwWTELMAkGA1UEBhMCQ04xCzAJBgNVBAgMAkJKMQswCQYDVQQHDAJCSjEOMAwG
A1UECgwFQkFJRFUxETAPBgNVBAsMCFNlY3VyaXR5MQ0wCwYDVQQDDAR0ZXN0MB4X
DTIyMDIxNTA2MDM0NloXDTIzMDIxNTA2MDM0NlowWTELMAkGA1UEBhMCQ04xCzAJ
BgNVBAgMAkJKMQswCQYDVQQHDAJCSjEOMAwGA1UECgwFQkFJRFUxETAPBgNVBAsM
CFNlY3VyaXR5MQ0wCwYDVQQDDAR0ZXN0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAxyt29BwrBJDXXSyV0w9SpAi86706fPk8/7kvxtYUi/T1qG8RLkIT
tiCEWMf3yczoKy2WOX/uvvyY4M8ABgUHM/z1ZaOOYhUATlLKCHIw+PZWcZArHY4j
uzKKRQi9+HhCdhgftNP6IZRmyje9LGlBpDVOxzqV7XvJOC1gZ10Vs6p0yLvsjOIp
MCaW1Cq+Quo2NpRMmeSgCzzMDILMAWTdnrw8uOnb+maCRX9Ljc9UZrtoc/t30LxW
3zMqE3PdSSM5rfCbIIEKV61bD6pwbS5Y6/yWQH4KRo6DxSwxnh5oAUnfBksB7bW/
c+flOCJp6c5hjQxniUxxraO7y4GORaYGjwIDAQABo1MwUTAdBgNVHQ4EFgQUtmnw
+MS6VisVzXRyha36LN56o2gwHwYDVR0jBBgwFoAUtmnw+MS6VisVzXRyha36LN56
o2gwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAxUSH5DZDAqOk
FRwtreNsMF/qAzVzctB2T2JLLySGKEqBJdgnTznA68xvi7Cng2cs8XUhOI7WB1FE
UvWVHRNDNr8reuCHutp5W7x30wry14Gz/j00f9zHGZ7fDvHR1wX2zsdiWLzyrVM6
Eh8MN0bnOi1hUE3oGNLskCmpFailg3H88IAu28jX9WaYc2omWA2U8QhIAbqCssU/
ojqf35F0FoSICxqZuJAkiB07nIqd130rBNSTxEl69zGNj2bFVmvAjrkJdY3ntYvL
GoSivJ5ywDkae/Okiu+uBbmLMQholf2/tzVnMqfEZaVNV863gB+5W05xy9XgHjXW
Rt5Rn7N/vQ==
-----END CERTIFICATE-----

root@localhost:~/ssl/tls# cat /etc/nginx/certs/key.pem
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDHK3b0HCsEkNdd
LJXTD1KkCLzrvTp8+Tz/uS/G1hSL9PWobxEuQhO2IIRYx/fJzOgrLZY5f+6+/Jjg
zwAGBQcz/PVlo45iFQBOUsoIcjD49lZxkCsdjiO7MopFCL34eEJ2GB+00/ohlGbK
N70saUGkNU7HOpXte8k4LWBnXRWzqnTIu+yM4ikwJpbUKr5C6jY2lEyZ5KALPMwM
gswBZN2evDy46dv6ZoJFf0uNz1Rmu2hz+3fQvFbfMyoTc91JIzmt8JsggQpXrVsP
qnBtLljr/JZAfgpGjoPFLDGeHmgBSd8GSwHttb9z5+U4ImnpzmGNDGeJTHGto7vL
gY5FpgaPAgMBAAECggEAIxwtZxEEIFTsMjem3cMIvXPw4dnE0r0ZHLdnjbS1gIqZ
N60EqVPqp6OJ0SDHNKLTlrD9l3tO+goLZS34ghcm0upX1CgF9JJhMG11whFnQqwi
Rvi0MQyajeh16X+GBaQliy3rgPQRVxQTHri+t3KCqNDg50wTm/7DVQzivrXxGm+1
E33fRfRwuAQi8VwIuMdJqGIAaelgljmZrLOQJK2+ocIqmy/DWG0rdSxqlURtA6Zg
JWyggthiwNlWBzAqFv+5xrchMtoVPiLNr5Dw9H2AeTCFn3wGv44g5Nacn9SNTbGN
6BmCZ++0i86W3klHZFldEFgSaPD2dFABntqmwISDwQKBgQD44W15LkAP43bN5OI0
zI3dEhSBA4ARMGjnF18ikQ2ryZL/3yx88MTFMte4DDnuEv0lbYF7WugCQwOjLIOj
CsdJ6/RgYpBKxFLyLNCx9tON8o5LUjWaHIt8s9gnZgfnQ4jyvqN1LuMg9ierGbUi
hYHikFWkpG2faKD0yOusIA8tIQKBgQDM3gA8SW8CC3+zil94LsP/RiTLNnM3UHyL
+8ga+ZJ8DaXwZSHZcDciqcIEWHTuUEzKmNFhtxyCrEVbJizZAeRZzLv+P714jU2E
a9gdZypxe2RY+rMa7b4l9SzrYvBjCwY44+5QSVWnvgIct4phZGo+/8p88EYd3n6J
FFnMLwuNrwKBgQDrfJESyho/bmBpLJqEjs/kT0krUV6J5R49GjYgBNPEqbP3evkj
NsVYOWU3dgoSI9F26ohLaZaBHVuBt4rOaFJstV/DGz+gHubE0nadDQjbFJhi2GRz
b3LkCV8K2FW+En6wAIW1SSoLdB3QnUBwa5acAyml4RcAcAVr1/h2Itd5QQKBgEYA
zd/Lz+zI9RfqfnjYsgx2cXgm4PPTwtsqa9Qb15wgJcWhKRHdsFld3HgTqdENC1Bn
K5TSNyHHr+NXhheY9+hfdxWHvjyFPCm1mCFj8TLZ0Q4+jBj1h/X8cx7zreSKDdQ+
vl4c0PICkUIyzUw+PbfNO+m5SJxIPMCmfZCxuqbtAoGBAK/nGBw9Qm3kJNASR3B9
T98bujh7gQTWRchtf4LNlnjoih+xBA3ufvhZfopbplibHUzDCWrn4EKXrWLZz+IX
jyezSuR5nYREmfqITkh/Q1gQHaKOaB/Mq9P37I/rdEh8rF6tPMrlQbQSMu+0+HYv
/fhiN1PLmisxTLnWsD63b5bo
-----END PRIVATE KEY-----

Thanks

yipingwang · ‎02-15-2022

I still didn't reproduce your problem, please refer to my attached console log.

On lx2160ardb-64, I downloaded, built and installed nginx-1.20.2.

After inserted dpaa2_caam and cryptodev modules, I copied your configuration section to /usr/local/nginx/conf/nginx.conf, then run /usr/local/nginx/sbin/nginx.

I run command "curl -k https://100.1.1.99" on lx2160ardb-65, you could see the result is normal.

BaiYanfu · ‎02-15-2022

Thanks for your help.

I reinstall the image and found the issue. It maybe because I add /usr/local/lib to the first line in file /etc/ld.so.conf, because without this line, openssl will report error.

root@localhost:~/ssl/certs# openssl
openssl: symbol lookup error: openssl: undefined symbol: EVP_mdc2, version OPENSSL_1_1_0

nginx is ok now, but openssl can not work again.

Thanks

yipingwang · ‎02-15-2022

Please refer to

https://blog.csdn.net/qq_37684467/article/details/115681736

BaiYanfu · ‎02-15-2022

Hello, thanks for your reply.

we have two openssl, /usr/bin/openssl and /usr/local/bin/openssl, I think /usr/bin/openssl is official version, and /usr/local/bin/openssl is nxp version, right ?

If we use /usr/local/bin/openssl, then we need put /usr/local/lib before /usr/lib, otherwise /usr/local/bin/openssl will report error, but when openssl is ok, nginx will have the previous issue.

root@localhost:~# cat /etc/ld.so.conf.d/libc.conf
# libc default configuration
/usr/local/lib
root@localhost:~# cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

root@localhost:~# ldconfig
root@localhost:~# /usr/local/bin/openssl engine -t -c
/usr/local/bin/openssl: symbol lookup error: /usr/local/bin/openssl: undefined symbol: EVP_mdc2, version OPENSSL_1_1_0
root@localhost:~# vim /etc/ld.so.conf
root@localhost:~# cat /etc/ld.so.conf
/usr/local/lib
include /etc/ld.so.conf.d/*.conf

root@localhost:~# ldconfig
root@localhost:~# /usr/local/bin/openssl engine -t -c
(devcrypto) /dev/crypto engine
 [DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA256, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB, MD5, SHA1, SHA224, SHA256, SHA384, SHA512]
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]

And if nginx is using /usr/lib/libssl.so, it is not using hardware ssl offloading, right ?

Thanks

BaiYanfu · ‎02-16-2022

Hello, I tested again with caam_jr and cryptodev driver. wget result is not good too.

1. If I didn't add /usr/local/lib to /etc/ld.so.conf file, nginx will not use caam. the interrupt counter for caam jr is not incremented.

root@localhost:~# cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

root@localhost:~# modprobe caam_jr
root@localhost:~# modprobe cryptodev
root@localhost:~# lsmod
Module                  Size  Used by
cryptodev              57344  0
caam_jr               229376  0
fsl_jr_uio             20480  0
caamkeyblob_desc       16384  1 caam_jr
caam                   45056  1 caam_jr
crypto_engine          16384  1 caam_jr
rng_core               24576  1 caam_jr
caamhash_desc          16384  1 caam_jr
caamalg_desc           40960  1 caam_jr
error                  24576  6 caamalg_desc,caamkeyblob_desc,caamhash_desc,caam,caam_jr,fsl_jr_uio
libdes                 24576  1 caam_jr
xt_CHECKSUM            16384  1
ipt_REJECT             16384  2
nf_reject_ipv4         16384  1 ipt_REJECT
ip6table_mangle        16384  1
ip6table_nat           16384  1
ip6table_filter        16384  1
ip6_tables             32768  3 ip6table_filter,ip6table_nat,ip6table_mangle
xt_addrtype            16384  2
xt_conntrack           16384  2
mali_dp                73728  0
root@localhost:~# /usr/local/nginx/sbin/nginx
root@localhost:~/ssl/wget# time ./wget.sh

real	0m22.077s
user	0m7.211s
sys	0m6.630s
root@localhost:~# cat /proc/interrupts |grep jr
397:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 172 Level     8010000.jr
398:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 173 Level     8020000.jr
399:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 174 Level     fsl-jr0

2. Add /usr/local/lib at the first line in file /etc/ld.so.conf, we can see the interrupt counter for caam jr is increatmented.

but wget test will consume more than 100 seconds, but when no caam, it only need 20 seconds.

root@localhost:~/ssl/wget# cat /etc/ld.so.conf
/usr/local/lib
include /etc/ld.so.conf.d/*.conf

root@localhost:~/ssl/wget# ldconfig
root@localhost:~/ssl/wget# /usr/local/bin/openssl engine -t -c
(dynamic) Dynamic engine loading support
     [ unavailable ]
root@localhost:~/ssl/wget# modprobe caam_jr
root@localhost:~/ssl/wget# modprobe cryptodev
root@localhost:~/ssl/wget# /usr/local/bin/openssl engine -t -c
(devcrypto) /dev/crypto engine
 [DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB, MD5, SHA1, SHA224, SHA256, SHA384, SHA512]
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]
root@localhost:~/ssl/wget# cat /proc/interrupts |grep jr
397:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 172 Level     8010000.jr
398:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 173 Level     8020000.jr
399:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 174 Level     fsl-jr0
root@localhost:~/ssl/wget# /usr/local/nginx/sbin/nginx
root@localhost:~/ssl/wget# time ./wget.sh

real	2m26.442s
user	0m8.834s
sys	0m15.243s
root@localhost:~/ssl/wget# cat /proc/interrupts |grep jr
397:     336918          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 172 Level     8010000.jr
398:     428257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 173 Level     8020000.jr
399:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     GICv3 174 Level     fsl-jr0

curl from another client is ok too.

baiyanfu@ADS-01:~$ curl -k https://172.18.53.70/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>
</html>

Thanks

yipingwang · ‎02-16-2022

I also reproduced wget problem with openssl build manually following page 1095 following LSDK 21.08 user guilde.

$ git clone https://source.codeaurora.org/external/qoriq/qoriq-components/openssl

$ cd openssl
$ $ ./Configure enable-devcryptoeng -I./include --prefix=/usr/local --openssldir=lib/ssl linuxaarch64
shared
$ make
$ sudo make install

File: /etc/ld.so.conf
...
/usr/local/lib
...
/usr/lib
...

$ sudo ldconfig

I need to discussed this problem with AE team, will provide more feedback later.

BaiYanfu · ‎02-16-2022

ok, thanks

yipingwang · ‎02-17-2022

There are multiple crypto accelerator in the LX2160A.

CAAM is a offload engine. So it use minimal CPU cycles except the handshake between the ARM core and the CAAM. CAAM works best for large chuck of data and can do ~3Gbps per AESA engine. LX2160A has 8 of the AES engine. i.e. +30Gbps performance.

LX2160A A72 core also has Crypto Extension (CE). Which is a multi vector processor that can speed us AES operation. It does not do RSA and other public/private key operation. Because CE is running at core frequency, and can access data from cache. Therefore it is much faster than CAAM (which run different clock and access data from DDR, not CPU cache), but CE will use up 100% of your CPU utilization, leave little room for you application (e.g. nginx in this case).

In the linux, you can discover all the registered CE by running:

root@localhost:~# cat /proc/crypto | grep ce

driver : essiv-cbc-aes-sha256-ce

driver : cts-cbc-aes-ce

driver : xts-aes-ce

driver : ctr-aes-ce

driver : cbc-aes-ce

driver : ecb-aes-ce

driver : cbcmac-aes-ce

driver : xcbc-aes-ce

driver : cmac-aes-ce

driver : __essiv-cbc-aes-sha256-ce

driver : __cts-cbc-aes-ce

driver : __xts-aes-ce

driver : ctr-aes-ce

driver : __ctr-aes-ce

driver : __cbc-aes-ce

driver : __ecb-aes-ce

driver : ccm-aes-ce

driver : aes-ce

driver : gcm-aes-ce

driver : sha256-ce

driver : sha224-ce

driver : sha1-ce

root@localhost:~#

If customer disabled CE or need module that are not listed above, then they will compare the CAAM with regular CPU crypto operation, which CPU is much slower (but there is no good reason for doing so).

Customer can use "mpstat" to measure the CPU utilization when they run

wget --no-check-certificate https://127.0.0.1 >/dev/null 2>&1 done and compare the difference between CAAM and CE, and decide which crypto accelerator to use.

p.s. Also customer may want to keep in mind the CE is not FIPS certified. If they need FIPS certify, then they will want to use CAAM.

BaiYanfu · ‎02-17-2022

Thanks a lot, which driver should I modprobe and how shlould I configure nginx to use ce driver?

yipingwang · ‎02-20-2022

I check your test script again. Your loop root@localhost:~/ssl/wget# cat wget.sh for i in `seq 1 1000`; do

wget --no-check-certificate https://127.0.0.1 >/dev/null 2>&1 done

You are getting an index.html, which is a very small file, few K Byte? The CAAM performance for small file is always bad because the setup/overhead involves of offload to caam engine. You may want to wget a 100MB file to make a better comparison.

Regards to your nginx question. From the previous output of "root@localhost:~# cat /proc/crypto | grep ce" there is no TLS1.x or AES128-SHA offload.

i.e.

root@localhost:~# cat /proc/crypto | grep ce

driver : essiv-cbc-aes-sha256-ce

driver : cts-cbc-aes-ce

driver : xts-aes-ce

driver : ctr-aes-ce

driver : cbc-aes-ce

driver : ecb-aes-ce

driver : cbcmac-aes-ce

driver : xcbc-aes-ce

driver : cmac-aes-ce

driver : __essiv-cbc-aes-sha256-ce

driver : __cts-cbc-aes-ce

driver : __xts-aes-ce

driver : ctr-aes-ce

driver : __ctr-aes-ce

driver : __cbc-aes-ce

driver : __ecb-aes-ce

driver : ccm-aes-ce

driver : aes-ce

driver : gcm-aes-ce

driver : sha256-ce

driver : sha224-ce

driver : sha1-ce

This is what I try to explain to you CE use can be very limited.

If you run the following from LX2160A,

root@localhost:~# modprobe caamalg_desc

root@localhost:~# modprobe dpaa2_caam

root@localhost:~# cat /proc/crypto | grep caam-qi2

driver : md5-caam-qi2

driver : hmac-md5-caam-qi2

driver : sha512-caam-qi2

driver : hmac-sha512-caam-qi2

driver : sha384-caam-qi2

driver : hmac-sha384-caam-qi2

driver : sha256-caam-qi2

driver : hmac-sha256-caam-qi2

driver : sha224-caam-qi2

driver : hmac-sha224-caam-qi2

driver : sha1-caam-qi2

driver : hmac-sha1-caam-qi2

driver : tls12-hmac-sha256-cbc-aes-caam-qi2

driver : tls11-hmac-sha1-cbc-aes-caam-qi2

driver : seqiv-authenc-hmac-sha512-rfc3686-ctr-aes-caam-qi2

driver : authenc-hmac-sha512-rfc3686-ctr-aes-caam-qi2

driver : rfc7539esp-chacha20-poly1305-caam-qi2

driver : rfc7539-chacha20-poly1305-caam-qi2

driver : seqiv-authenc-hmac-sha384-rfc3686-ctr-aes-caam-qi2

driver : authenc-hmac-sha384-rfc3686-ctr-aes-caam-qi2

driver : seqiv-authenc-hmac-sha256-rfc3686-ctr-aes-caam-qi2

driver : authenc-hmac-sha256-rfc3686-ctr-aes-caam-qi2

driver : seqiv-authenc-hmac-sha224-rfc3686-ctr-aes-caam-qi2

driver : authenc-hmac-sha224-rfc3686-ctr-aes-caam-qi2

driver : seqiv-authenc-hmac-sha1-rfc3686-ctr-aes-caam-qi2

driver : authenc-hmac-sha1-rfc3686-ctr-aes-caam-qi2

driver : seqiv-authenc-hmac-md5-rfc3686-ctr-aes-caam-qi2

driver : authenc-hmac-md5-rfc3686-ctr-aes-caam-qi2

driver : echainiv-authenc-hmac-sha512-cbc-des-caam-qi2

driver : authenc-hmac-sha512-cbc-des-caam-qi2

driver : echainiv-authenc-hmac-sha384-cbc-des-caam-qi2

driver : authenc-hmac-sha384-cbc-des-caam-qi2

driver : echainiv-authenc-hmac-sha256-cbc-des-caam-qi2

driver : authenc-hmac-sha256-cbc-des-caam-qi2

driver : echainiv-authenc-hmac-sha224-cbc-des-caam-qi2

driver : authenc-hmac-sha224-cbc-des-caam-qi2

driver : echainiv-authenc-hmac-sha1-cbc-des-caam-qi2

driver : authenc-hmac-sha1-cbc-des-caam-qi2

driver : echainiv-authenc-hmac-md5-cbc-des-caam-qi2

driver : authenc-hmac-md5-cbc-des-caam-qi2

driver : echainiv-authenc-hmac-sha512-cbc-des3_ede-caam-qi2

driver : authenc-hmac-sha512-cbc-des3_ede-caam-qi2

driver : echainiv-authenc-hmac-sha384-cbc-des3_ede-caam-qi2

driver : authenc-hmac-sha384-cbc-des3_ede-caam-qi2

driver : echainiv-authenc-hmac-sha256-cbc-des3_ede-caam-qi2

driver : authenc-hmac-sha256-cbc-des3_ede-caam-qi2

driver : echainiv-authenc-hmac-sha224-cbc-des3_ede-caam-qi2

driver : authenc-hmac-sha224-cbc-des3_ede-caam-qi2

driver : echainiv-authenc-hmac-sha1-cbc-des3_ede-caam-qi2

driver : authenc-hmac-sha1-cbc-des3_ede-caam-qi2

driver : echainiv-authenc-hmac-md5-cbc-des3_ede-caam-qi2

driver : authenc-hmac-md5-cbc-des3_ede-caam-qi2

driver : echainiv-authenc-hmac-sha512-cbc-aes-caam-qi2

driver : authenc-hmac-sha512-cbc-aes-caam-qi2

driver : echainiv-authenc-hmac-sha384-cbc-aes-caam-qi2

driver : authenc-hmac-sha384-cbc-aes-caam-qi2

driver : echainiv-authenc-hmac-sha256-cbc-aes-caam-qi2

driver : authenc-hmac-sha256-cbc-aes-caam-qi2

driver : echainiv-authenc-hmac-sha224-cbc-aes-caam-qi2

driver : authenc-hmac-sha224-cbc-aes-caam-qi2

driver : echainiv-authenc-hmac-sha1-cbc-aes-caam-qi2

driver : authenc-hmac-sha1-cbc-aes-caam-qi2

driver : echainiv-authenc-hmac-md5-cbc-aes-caam-qi2

driver : authenc-hmac-md5-cbc-aes-caam-qi2

driver : gcm-aes-caam-qi2

driver : rfc4543-gcm-aes-caam-qi2

driver : rfc4106-gcm-aes-caam-qi2

driver : chacha20-caam-qi2

driver : xts-aes-caam-qi2

driver : rfc3686-ctr-aes-caam-qi2

driver : ctr-aes-caam-qi2

driver : cbc-des-caam-qi2

driver : cbc-3des-caam-qi2

driver : cbc-aes-caam-qi2

You can see TLS1.2 and 1.1 are offload to CAAM engine.

driver : tls12-hmac-sha256-cbc-aes-caam-qi2

driver : tls11-hmac-sha1-cbc-aes-caam-qi2

So for NGINX, you can only offload to CAAM, not CE (as far as I know, CE does not support/no driver for TLS or AES128-SHA).

For NGINX setup, please refers to LSDKUG_Rev21.09(or whatever version you are using), section 9.1.1.3 Hardware Offloading with OpenSSL.

...

TLS 1.1 and TLS 1.2 Offloading in Nginx Server ...

BaiYanfu · ‎02-22-2022

Thanks for you reply. I tested again with caam driver and could not get a satisfied result, do we have a test data about nginx with ssl offload using caam ?

yipingwang · ‎02-22-2022

Please refer to the feedback from the application team.

We only run IPsec tunnel benchmark for the LSDK release, which show ~60Gbps for IPsec (AES-HMAC-SHA-2).
For ngnix, it is a linux user space application and a lot of customization and fine tuning in linux to optimize performance. Does customer has a specific performance target needs to achieve? Are they still running the same test, which the issue is sync nature of the nginx protocol.
e.g.
-instead of single wget, is it ok to multiple wget in parallel to mask the sync nature of https?
-What is the average file size, or we can use 1G file per flow?
-TCP window size
etc.

one more question, does the customer get the performance that they need for openssl speed test? If they are OK with openssl speedtest, the issue is really with nginx, not CAAM raw performance.

BaiYanfu · ‎02-23-2022

Thanks for your help very much. the followings is my openssl speed test results.

bytes	no cryptodev no -multi	no cryptodev with -multi 8	with cryptodev no -multi	with cryptodev with -multi 8
16	3.56 Gbps	28.47 Gbps	10.02 Mbps
128	9.97 Gbps	79.80 Gbps	78.82 Mbps
1024	12.60 Gbps		582.22 Mbps
16384	13.22 Gbps	105.86 Gbps	2.47 Gbps	19.13 Gbps
65536	13.28 Gbps		3.37 Gbps	26.85 Gbps

yipingwang · ‎02-23-2022

Thanks for the openssl measurement.

From the openssl test result last row.

16 3.56 Gbps 28.47 Gbps 10.02 Mbps

...

65536 13.28 Gbps 3.37 Gbps 26.85 Gbps

As mentioned in Feb 17. "~3Gbps per AESA engine. LX2160A has 8 of the AES engine." That matched what the SEC engine can do and customer can observe the optimized performance.

Also, it confirm the "inefficiency" for SEC engine with small frame/packet size operation.

So back to the original nginx question.

1) serial wget for index.html is not a good way to test crypto acceleration. To get good performance, customer needs to use wget for a large file, like 1GB or larger files in parallel. customer should able to see 8x performance gain compare to serial wget command for testing.

2) We do not really optimize user space applications, like NGINX. We only able the base layer for customer to optimize base on their application.

NGINX is not tested or benchmark by the LSDK release.

The enablement is provided in the LSDK user Guide, section ""9.1.1.3 Hardware Offloading with OpenSSL ##### TLS 1.1 and TLS 1.2 Offloading in Nginx Server Nginx does not use any openssl engines by default. If an engine is to be used, including cryptodev, it must be explicitly listed in nginx configuration file. Here is a fragment of nginx configuration file that activates cryptodev and allows hardware offloading of TLS 1.1 and TLS 1.2 record layer protocol:

/etc/nginx/nginx.conf:

ssl_engine cryptodev;

worker_processes 4;

worker_cpu_affinity 0001 0010 0100 1000; #for 4 Core CPU; For 2 Core CPU worker_cpu_affinity 01 10; ...

# HTTPS server

#

server {

listen 443;

server_name localhost;

ssl on;

ssl_certificate server.crt;

NXP Semiconductors

Linux user space

Layerscape

ssl_certificate_key server.key;

ssl_session_timeout 5m;

ssl_protocols TLSv1.1;

ssl_ciphers AES128-SHA:AES256-SHA;

ssl_prefer_server_ciphers on;

location / {

root /var/www/localhost/html;

index index.html index.htm;

}

...

Worker processes and affinity should be set according to the number of CPU cores available on the platform. Refer to nginx documentation for more details.

#####

If customer needs help to optimize a non-supported application like nginx, they may wants to use professional/paid service. Keep in mind optimization is depends on customer use cases. As they already observed.

1) performance is efficient for large frame/file.

2) https is a synchronous protocol. It is not efficient in nature for multitasking/parallel operation. Performance gain can achieve by convert it to asynchronous. BUt this may break nginx/https protocol.

3) customer already discovered the best performance for offload the "aes-128-cbc" is 26.85 Gbps. That set the ceiling of performance for offload the crypto operation to caam.