"swiotlb buffer is full" when writing large file on encrypted file system using CAAM & 4GB of ram

[Olivier_B]

Hi we are having issues using the CAAM driver on Linux as describe in AN12714 (i.MX Encrypted Storage Using CAAM Secure Keys)

When transferring data to an encrypted partition either in ram or disk, we get the following error

caam_jr 30903000.jr: swiotlb buffer is full (sz: 512 bytes), total 32768 (slots), used 32768 (slots)
caam_jr 30903000.jr: unable to map source

Here is the code that can be used to reproduce the issue. 

mkdir /tmp
mkdir /data
mount -o size=1G -t tmpfs none /tmp/
dd if=/dev/zero of=/tmp/disk bs=1024 count=200000
losetup /dev/loop1 /tmp/disk
caam-keygen create randomkey ecb -s 16
cat /uboot/caam/randomkey | keyctl padd logon logkey: @u
DM_DISABLE_UDEV=1 dmsetup -v create encrypted --table "0 400000 crypt capi:tk(cbc(aes))-plain :36:logon:logkey: 0 /dev/loop1 0 1 sector_size:512"
mkfs.ext4 /dev/mapper/encrypted
mount /dev/mapper/encrypted /data
dd if=/dev/zero of=/data/crap bs=1024 count=100000

We are using Yocto Kirkstone branch and linux-fscl-imx  5.15.87-2.2.0+g8a1a87989b6a. The issue is also present in linux-imx kernel branch. 

The CAAM is working fine on Dunfell yocto version or if we force 3GB of ram instead of 4GB using bootargs=3072M in uboot.

The CAAM driver / DMA / swiotlb bounce buffer seems to be the problem. 

[richc128]

@Olivier_B your patch seemed to work for me. 

[richc128]

I'm seeing similar behavior on the 5.15.32 kirkstone kernel. I get the error when I run the mkfs.ext4 command itself

mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: 0fbd7de1-10f8-455d-80f8-802dfb5a37d6
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): [  138.542267] caam_jr 30902000.jr: swiotlb buf
fer is full (sz: 512 bytes), total 32768 (slots), used 32768 (slots)
[  138.542267] caam_jr 30902000.jr: swiotlb buffer is full (sz: 512 bytes), tota
l 32768 (slots), used 32768 (slots)
[  138.542286] caam_jr 30902000.jr: unable to map destination
[  138.552691] caam_jr 30902000.jr: unable to map destination
[  138.553843] caam_jr 30902000.jr: swiotlb buffer is full (sz: 512 bytes), tota
l 32768 (slots), used 32768 (slots)
[  138.553854] caam_jr 30902000.jr: unable to map destination
[  138.557003] caam_jr 30902000.jr: swiotlb buffer is full (sz: 512 bytes), tota
l 32768 (slots), used 32768 (slots)
[  138.557026] caam_jr 30902000.jr: unable to map destination
[  138.560394] caam_jr 30902000.jr: swiotlb buffer is full (sz: 512 bytes), tota
l 32768 (slots), used 32768 (slots)
[  138.560418] caam_jr 30902000.jr: unable to map destination
[  138.563456] caam_jr 30902000.jr: swiotlb buffer is full (sz: 512 bytes), tota
l 32768 (slots), used 32768 (slots)
[  138.568295] sdhci-esdhc-imx 30b60000.mmc: swiotlb buffer is full (sz: 65024 b
ytes), total 32768 (slots), used 32720 (slots)
[  138.573755] caam_jr 30902000.jr: unable to map destination
[  138.897828] sdhci-esdhc-imx 30b60000.mmc: swiotlb buffer is full (sz: 65024 b
ytes), total 32768 (slots), used 32236 (slots)
[  138.909729] sdhci-esdhc-imx 30b60000.mmc: swiotlb buffer is full (sz: 65024 b
ytes), total 32768 (slots), used 31991 (slots)

[Olivier_B]

Hi @richc128 

You can try the patch I posted a little below, this is the exact same issue I had. 

https://community.nxp.com/t5/i-MX-Processors/quot-swiotlb-buffer-is-full-quot-when-writing-large-fil...

[hector_delgado]

Hi @Olivier_B ,

I hope you're having a great day. Could you try implementing encrypted storage using Rev. 2 of the i.MX Encrypted Storage Using CAAM Secure Keys guide? You can download it using the following link https://www.nxp.com/webapp/Download?colCode=AN12714. 

Let me know the results of this test.

Best regards,
Hector.

[Olivier_B]

Hi @hector_delgado,

Yes we are using the same implementation as the Application Note.Still get the same error The only difference is that the targeted Yocto branch in AN12714 is zeus.

Our project was running fine on the Dunfell release (which is newer than zeus). The issue appeared in Kirkstone LTS release only when we use 4GB of ram. If we pass mem=3072M to the kernel command line, the CAAM works perfectly. This really looks like a driver issue either in caam_jr or the integration with the DMA controller / swiotlb bounce buffer integration. 

Best regards,

Olivier

[hector_delgado]

Hi @Olivier_B ,

We're having some delays trying to reproduce this issue on our side, and we're waiting on a different board to test this successfully with the 4GB of RAM you mentioned. In the meantime could you send us your kernel configuration file? For us, this file is located in linux-imx/arch/arm64/imx_v8_defconfig.

Thank you.

Best regards,
Hector.

[Raphael_B]

Hi @hector_delgado,

I am a coworker of Olivier. Here is the config file that we are using for the linux-fslc image we are testing on. Also, note that we changed one of the command in the original post. We replaced :

cat /uboot/caam/randomkey | keyctl padd logon logkey: @u

with:

cat /data/caam/randomkey | keyctl padd logon logkey: @s

Best regards,

Raphael

[hector_delgado]

Hi @Raphael_B ,

Thank you, we'll update you with any news soon.

Best regads,
Hector.

[pierluigi_p]

Hi @hector_delgado,

here some additional info:

• all the "unable to map" messages are originated from the few lines here
• this behavior is only reproducible using 4GB DDR4 configuration and an encrypted file mounted from a RAM file-system (like /tmp)
• using either 2/4GB LPDDR4 or 2GB DDR4 configuration or an encrypted file mounted from the eMMC, this behavior is not reproducible
• the main difference between the 2GB and 4GB configurations seems to be the mapping of the reserved areas as provided by "cat /proc/iomem": while with the 2GB everything is close to the CMA area (0x98000000), in the 4GB configuration most of the reserved areas (but the CMA) are close to upper RAM limit
• however, this behavior seems not related to the CMA location: we are using the same location for 2 and 4 GB

Is there any relationship between the RAM speed and the CAAM capability to elaborate the data?

Thanks

Best Regards

Pier

[hector_delgado]

Hi @pierluigi_p ,

We have successfully identified the problem; our Software Team has been notified and are actively working on this issue. Since this may take some time, I suggest contacting us again in approximately one month. I apologize for the inconvenience this may cause and thank you for your patience.

Best regards,
Hector.

[Olivier_B · Accepted solution]

Hi @hector_delgado 

After further investigation, we found the issue. The root cause of the problem was the an issue in the crypto / crypto_engine drivers. 

https://marc.info/?l=linux-crypto-vger&m=168200280029011&w=2

I attached the patch. 

Thanks! 

[JonTrulson1z]

FWIW, I ran into this on a Toradex 5.15.77 kernel for IMX8QM. The patch needed to be slightly modified for that kernel, but the fix works well now on that kernel as well. Thanks a lot, made my day

[hector_delgado]

Hi @Olivier_B ,

Thank you for sharing this, I'll review this with our team.

Best regards,
Hector.

[MikkelHaugstrupJensen]

Any chance there is a version of this patch back ported to 5.15?

[Olivier_B]

Yes, it's been back ported to 5.15 on the mainline
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.15.y&id=f716374c4...

The patch should directly work on linux-imx

[hector_delgado]

Hi @Olivier_B ,

Looking through the initial post, I see that you're using linux-fslc-imx 5.15.87-2.2.0+g8a1a87989b6a. As I understand, this is not the NXP-maintained Linux kernel. The post mentioned that this issue is "also present on the linux-imx kernel branch." Could we know which branch/commit in the linux-imx repository did you try exactly? We're trying to replicate this issue on our side and this information would be very helpful. Thank you.

Best regards,
Hector.

[hector_delgado]

Hi @Olivier_B ,

I hope you're doing well. Sorry for the late response, I proceeded with the escalation of this case a few days ago and our team is currently investigating this issue thoroughly, I apologize for not letting you know earlier. Will get back to you with any updates for your issue. Thank you.

Best regards,
Hector.