imx93 cbc-aes-tee immediate crash

[electro1]

Hi,

We get an immediate crash when running an aes benchmark on unmodified lf-6.6.36-2.1.0 kernel with CONFIG_TEE_CRYPTO=y:

# cryptsetup benchmark -c aes

# Tests are approximate using memory only (no storage IO).
[ 108.233625] Unable to handle kernel paging request at virtual address ceff8000821f7000
[ 108.241634] Mem abort info:
[ 108.244479] ESR = 0x0000000096000047
[ 108.248242] EC = 0x25: DABT (current EL), IL = 32 bits
[ 108.253639] SET = 0, FnV = 0
[ 108.256742] EA = 0, S1PTW = 0
[ 108.259894] FSC = 0x07: level 3 translation fault
[ 108.264838] Data abort info:
[ 108.267728] ISV = 0, ISS = 0x00000047, ISS2 = 0x00000000
[ 108.273262] CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[ 108.278363] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 108.283733] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000081deb000
[ 108.290516] [ceff8000821f7000] pgd=10000000affff003, p4d=10000000affff003, pud=10000000afffe003, pmd=10000000afff8003, pte=0000000000000000
[ 108.303146] Internal error: Oops: 0000000096000047 [#1] PREEMPT SMP
[ 108.309426] Modules linked in:
[ 108.312494] CPU: 1 PID: 434 Comm: cryptsetup Tainted: G B 6.6.36-gd23d64eea511 #1
[ 108.327035] pstate: a0400009 (NzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 108.334001] pc : __memcpy+0x128/0x260
[ 108.337685] lr : sg_copy_buffer+0x114/0x228
[ 108.341888] sp : ffff8000833c7920
[ 108.345207] x29: ffff8000833c7990 x28: 0000000000000003 x27: 0000000000010000
[ 108.352374] x26: ceff8000821f4000 x25: 0000000000000001 x24: 0000000000010000
[ 108.359533] x23: ebff00001159b000 x22: 0000000000003000 x21: 0000000000001000
[ 108.366700] x20: 0000000000001000 x19: 0000000000004000 x18: 0000000000000000
[ 108.373859] x17: ffff8000810b1434 x16: ffff8000806e42b0 x15: 0000000000000000
[ 108.381026] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 108.388185] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 108.395344] x8 : ffff8000833c7900 x7 : 0000000000000000 x6 : 0000000000000000
[ 108.402503] x5 : ceff8000821f8000 x4 : ebff00001159c000 x3 : ceff8000821f7000
[ 108.409670] x2 : 0000000000001000 x1 : ebff00001159b000 x0 : ceff8000821f7000
[ 108.416838] Call trace:
[ 108.419290] __memcpy+0x128/0x260
[ 108.422618] sg_copy_to_buffer+0x18/0x28
[ 108.426553] aes_crypt+0x170/0x368
[ 108.429976] skcipher_encrypt+0x58/0x80
[ 108.433824] crypto_skcipher_encrypt+0x5c/0x80
[ 108.438287] skcipher_recvmsg+0x548/0x688
[ 108.442318] sock_read_iter+0x1cc/0x240
[ 108.446174] vfs_read+0x464/0x4b0
[ 108.449511] ksys_read+0x190/0x1c0
[ 108.452925] __arm64_sys_read+0x4c/0x68
[ 108.456773] invoke_syscall.constprop.0+0x68/0x140
[ 108.461583] do_el0_svc+0x80/0x128
[ 108.464998] el0_svc+0x50/0x190
[ 108.468153] el0t_64_sync_handler+0x120/0x130
[ 108.472529] el0t_64_sync+0x190/0x198
[ 108.476216] Code: 927cec03 cb0e0021 8b0e0042 a9411c26 (a900340c)
[ 108.482307] ---[ end trace 0000000000000000 ]---
[ 108.486985] note: cryptsetup[434] exited with preempt_count 1

[Harvey021]

Hi @electro1 

Please accept my sincere apologies for the delay. We've still continued working with R&D team for the issue and pushing them for an answer as soon as possible. As the case has lasted a long time. so, I'll create a new ticket to keep following, closing the current one.

Regards

Harvey

[omar_aberkan]

Any update on this? We are really waiting a very long time now.

[Harvey021]

hi @electro1 

We try to reproduce issue on our side, unmodified lf-6.6.36-2.1.0 kernel with CONFIG_TEE_CRYPTO=y, CONFIG_CRYPTO_USER_API_SKCIPHER=y
root@imx93evk:~/cryptsetup# zcat /proc/config.gz | grep CONFIG_TEE_CRYPTO
CONFIG_TEE_CRYPTO=y
root@imx93evk:~/cryptsetup# zcat /proc/config.gz | grep CONFIG_CRYPTO_USER_API_SKCIPHER
CONFIG_CRYPTO_USER_API_SKCIPHER=y
root@imx93evk:~/cryptsetup# export LD_LIBRARY_PATH=/root/cryptsetup/.libs
root@imx93evk:~/cryptsetup# ./cryptsetup benchmark -c aes
# Tests are approximate using memory only (no storage IO).
# Algorithm | Key | Encryption | Decryption
aes-cbc 256b 432.4 MiB/s 509.9 MiB/s

In order to reproduce the issue, I'd like,
1). Based on lf-6.6.36-2.1.0 kernel, can you provide RT patch link?
2). Can you support full log for 'fio' and 'cryptsetup' test?

Regards

Harvey

[electro1]

That's interesting. Maybe it is easier to focus on one error at a time? Let's start with the cryptsetup benchmark -c aes crash with unmodified lf-6.6.36-2.1.0 kernel (no rt patch).

Here is some output that might be interesting to double check that we are running the same algorithms:

root@imx93:~# dmesg | grep tee
[ 0.000000] OF: reserved mem: 0x0000000096000000..0x0000000097dfffff (30720 KiB) nomap non-reusable optee_core@96000000
[ 0.000000] OF: reserved mem: 0x0000000097e00000..0x0000000097ffffff (2048 KiB) nomap non-reusable optee_shm@97e00000
[ 0.913556] optee: probing for conduit method.
[ 0.918023] optee: revision 4.2 (612bc5a6)
[ 0.918409] optee: dynamic shared memory is enabled
[ 0.927690] optee: initialized driver
[ 1.480733] tee_crypt algorithms registered in /proc/crypto

root@imx93:~# cat /proc/crypto | grep -A 3 "cbc(aes)"
name : cbc(aes)
driver : cbc-aes-tee
module : kernel
priority : 3000
--
name : xcbc(aes)
driver : xcbc-aes-ce
module : kernel
priority : 300
--
name : essiv(cbc(aes),sha256)
driver : essiv-cbc-aes-sha256-ce
module : kernel
priority : 301
--
name : cts(cbc(aes))
driver : cts-cbc-aes-ce
module : kernel
priority : 300
--
name : cbc(aes)
driver : cbc-aes-ce
module : kernel
priority : 300

root@imx93:~# zcat /proc/config.gz | grep TEE
# CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
# CONFIG_NETFILTER_XT_TARGET_TEE is not set
# CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
CONFIG_ARM_SCMI_TRANSPORT_OPTEE=y
CONFIG_HW_RANDOM_OPTEE=y
# CONFIG_TCG_FTPM_TEE is not set
# CONFIG_HID_STEELSERIES is not set
# CONFIG_RTC_DRV_OPTEE is not set
CONFIG_IMX_IRQSTEER=y
CONFIG_TEE=y
CONFIG_OPTEE=y
# CONFIG_OPTEE_INSECURE_LOAD_IMAGE is not set
CONFIG_TEE_CRYPTO=y
CONFIG_TRUSTED_KEYS_TEE=y

I'm not sure exactly which full log i could provide? Can you double check that you have the cbc-aes-tee driver in /proc/crypto?

[Harvey021]

1. I can't find highlighted part in my imx93.
Besides Linux kernel, I guess you use custom OP-TEE, can I know what configures are set for OP-TEE?
2. For RT-patch you are using, can you share RT-patch link for kernel 6.6.36?

Regards

Harvey

[electro1]

Hi,

For OP-TEE we the Yocto recipes from NXP 6.6.36 BSP which gives us

PLATFORM_FLAVOR = "mx93evk"

in addition we have added this configuration as specified in chapter 10.5.5 in iMX Linux User Guide:

CFG_IMX_TRUSTED_ARM_CE=y
CFG_WITH_SOFTWARE_RNG=n

and made sure to disable ELE drivers in Linux:

# CONFIG_IMX_SEC_ENCLAVE is not set
# CONFIG_IMX_ELE_TRNG is not set

The RT-patch we use, but note that we get the cryptsetup crash also without it:

https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.6/older/patch-6.6.36-rt35.patch.xz

[Harvey021]

Hi 

According to 'cryptsetup' issue, I did some debug work, finally I found it's caused by incorrect crypt length used for tee_crypt driver. tee_crypt driver default supports max buffer length is 4K, but 'cryptsetup' send 64K packet to tee_crypt driver for encryption.

<cryptsetup path>/lib/crypto_backend/cipher_check.c
#define CIPHER_BLOCK_BYTES 65536
size_t done = 0, block = CIPHER_BLOCK_BYTES;

Because 'cryptsetup' and tee_crypt driver are open source code, not owned by NXP.

You can change 'cryptsetup' or tee_crypt driver to fix this issue.

Here I provide tee_crypt driver change for reference,

<kernel path>/drivers/tee/crypto/tee_skcipher.c
tc_prv_ctx->shm_pool = memremap(tc_prv_ctx->shm_pool_paddr,
- AES_BLOCK_SIZE + 2 * SZ_64K, MEMREMAP_WB);

For RT-patch issue, I can apply patch in above link on BSP lf-6.6.36.

Can you share what error meet? Can you share?

1. kernel configure

2. OP-TEE configure

3. full running log with fio, including error log

Regards

Harvey

[electro1]

Hi,

Good that you could reproduce the cryptsetup issue.

The tee_crypt driver is only available in the linux-imx kernel and it was written by an NXP employee , it is not part of the mainline kernel.

Without being a kernel expert, I feel the fix for the driver is not correct. An application should not be able to crash the driver by using an unsupported parameter. Either it should be rejected, or the driver should handle the different parameters that can be used.

What if another application uses 128KB packets? I feel the driver should divide the work in smaller steps to handle any packet size or something similar.

RT-patch dmcrypt issue

When it comes to the dmcrypt issue we use this additional kernel config:

CONFIG_PREEMPT_RT=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_DM_CRYPT=y

OP-TEE config should be the same as before.

I then use these two scripts.

dm-crypt-init.sh:

#!/bin/sh
# dm-crypt-init.sh

mkdir -p /data/crypt

KEYNAME=dm_trust_plainkey
KEY=$(keyctl add trusted $KEYNAME 'new 32' @s)
keyctl pipe $KEY > /data/$KEYNAME.blob
keyctl list @s

DEV=/dev/loop0
ALGO="capi:cbc-aes-tee-plain"

dd if=/dev/zero of=/data/crypt/encrypted.img bs=1M count=512 && losetup /dev/loop0 /data/crypt/encrypted.img
BLOCKS=$(blockdev --getsz /dev/loop0)

TABLE="0 $BLOCKS crypt $ALGO :32:trusted:$KEYNAME 0 $DEV 0 1 sector_size:512"
dmsetup -v create encrypted --table "$TABLE"

mkfs.ext4 /dev/mapper/encrypted
mkdir -p /data/mnt/encrypted
mount -t ext4 /dev/mapper/encrypted /data/mnt/encrypted

run-fio.sh

#!/bin/sh
# run-fio.sh

fio --filename=/data/mnt/encrypted/fio.bin --size=128MB --rw=rw --bs=8k --ioengine=sync --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=throughput-test-job --eta-newline=1

Here is an example test run:

root@imx93:~# ./dm-crypt-init.sh 
2 keys in keyring:
647505922: ----s-rv 0 0 user: invocation_id
160646651: --alswrv 0 0 trusted: dm_trust_plainkey
512+0 records in
512+0 records out
[ 31.367193] loop0: detected capacity change from 0 to 1048576
Name: encrypted
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 254, 1
Number of targets: 1

mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 131072 4k blocks and 32768 inodes
Filesystem UUID: b18c61f8-6dfb-4407-bb71-b69f7bcd8a19
Superblock backups stored on blocks: 
32768, 98304

Allocating group tables: done 
Writing inode tables: done 
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

[ 33.442700] EXT4-fs (dm-1): mounted filesystem b18c61f8-6dfb-4407-bb71-b69f7bcd8a19 r/w with ordered data mode. Quota mode: none.
root@imx93:~# ./run-fio.sh 
throughput-test-job: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=sync, iodepth=1
fio-3.36-117-gb2403
Starting 1 process
throughput-test-job: Laying out IO file (1 file / 128MiB)
Jobs: 1 (f=1): [M(1)][2.5%][r=14.1MiB/s,w=14.1MiB/s][r=1811,w=1810 IOPS][eta 01m:57s]
Jobs: 1 (f=1): [M(1)][3.3%][r=49.7MiB/s,w=51.2MiB/s][r=6359,w=6557 IOPS][eta 01m:56s]
Jobs: 1 (f=1): [M(1)][4.2%][r=71.8MiB/s,w=70.8MiB/s][r=9195,w=9064 IOPS][eta 01m:55s]
Jobs: 1 (f=1): [M(1)][5.0%][r=36.4MiB/s,w=36.1MiB/s][r=4658,w=4618 IOPS][eta 01m:54s]
Jobs: 1 (f=1): [M(1)][6.7%][r=21.1MiB/s,w=20.8MiB/s][r=2704,w=2658 IOPS][eta 01m:52s] 
Jobs: 1 (f=1): [M(1)][7.5%][r=9262KiB/s,w=9190KiB/s][r=1157,w=1148 IOPS][eta 01m:51s]
Jobs: 1 (f=1): [M(1)][8.3%][r=58.4MiB/s,w=58.5MiB/s][r=7469,w=7487 IOPS][eta 01m:50s]
Jobs: 1 (f=1): [M(1)][9.2%][r=9.87MiB/s,w=9.79MiB/s][r=1263,w=1252 IOPS][eta 01m:49s]
Jobs: 1 (f=1): [M(1)][10.0%][r=6009KiB/s,w=6105KiB/s][r=751,w=763 IOPS][eta 01m:48s]
Jobs: 1 (f=1): [M(1)][10.8%][r=18.2MiB/s,w=18.9MiB/s][r=2328,w=2420 IOPS][eta 01m:47s]
Jobs: 1 (f=1): [M(1)][11.7%][r=2733KiB/s,w=3020KiB/s][r=341,w=377 IOPS][eta 01m:46s]
./run-fio.sh: line 3: 511 Segmentation fault (core dumped) fio --filename=/data/mnt/encrypted/fio.bin --size=128MB --rw=rw --bs=8k --ioengine=sync --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=throughput-test-job --eta-newline=1
root@imx93:~# [ 106.152684] systemd[1]: sshd@1-10.2.2.130:22-10.2.2.128:59470.service: Deactivated successfully.
[ 106.172954] systemd[1]: session-c2.scope: Deactivated successfully.
[ 106.183417] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[ 106.200326] systemd[1]: Journal Audit Socket was skipped because of an unmet condition check (ConditionSecurity=audit).
[ 106.236436] systemd[1]: Starting Journal Service...
[ 106.263938] systemd-journald[534]: Collecting audit messages is disabled.
[ 106.273399] systemd-journald[534]: File /var/log/journal/3a5531c948cf48d58c1052782ba34b41/system.journal corrupted or uncleanly shut down, renaming and replacing.
[ 106.328301] systemd[1]: Started Journal Service.

This is when running from serial console, when running from SSH session I usually loose the SSH session. In this example I lost my separate SSH session, this is what systemd logs about. We have also seen other strange behavior from systemd. It all points to some kernel corruption. Sometimes the crash happens quickly and sometimes it takes longer.

[Harvey021]

Hi, 

Sorry to reply back to you with delay. 

For memory alloc issue in kernel tee driver, I'll sync with driver owner.

For RT-patch dm-crypt issue, I can reproduce the issue on my side.

Seems "Segmentation fault" is caused by 'fio' application, I'm investigating the root cause of this issue, will update when I get more findings.

Regards

Harvey

[electro1]

Any update on this?

[omar_aberkan]

I also get strange behaviour by using this driver. The encryption of the partition goes fine. But after decryption the partition, systemd is unable to start?  Disabling CONFIG_IMX_SEC_ENCLAVE causes CONFIG_SOC_IMX9 to be disabled, which if find very strange, maybe this is the cause why it gives problems?

Commands that i used: 

execute "keyctl add trusted ${TARGET_ROOTFS_LUKS_KEY_NAME} 'load $(cat ${TARGET_LUKS_KEY_DIR}/${TARGET_ROOTFS_LUKS_KEY_NAME})' @u"
execute "dmsetup create ${TARGET_ROOTFS_NAME} --table \"0 $(blockdev --getsz ${TARGET_ROOTFS_PART}) crypt capi:xts-aes-tee-plain :32:trusted:${TARGET_ROOTFS_LUKS_KEY_NAME} 0 ${TARGET_ROOTFS_PART} 0 1 sector_size:512\""

Systemd fails to start:

[UNSUPP] Starting of Root Slice unsupported.
[DEPEND] Dependency failed for Kernel Configuration File System.
[DEPEND] Dependency failed for Root Mount.
[DEPEND] Dependency failed for Bind mount volatile /srv.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for FUSE Control File System.
[DEPEND] Dependency failed for Load/Save OS Random Seed.
[DEPEND] Dependency failed for POSIX Message Queue File System.
[DEPEND] Dependency failed for Network Configuration.
[DEPEND] Dependency failed for Kernel Debug File System.
[DEPEND] Dependency failed for initctl Compatibility Named Pipe.
[DEPEND] Dependency failed for User Database Manager Socket.
[DEPEND] Dependency failed for Journal Socket.
[DEPEND] Dependency failed for Journal Service.
[DEPEND] Dependency failed for Bind mount volatile /var/cache.
[DEPEND] Dependency failed for Record System Boot/Shutdown in UTMP.
[DEPEND] Dependency failed for Record Runlevel Change in UTMP.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for udev Control Socket.
[DEPEND] Dependency failed for Bind mount volatile /var/spool.
[DEPEND] Dependency failed for Kernel Trace File System.

....