topic Re: CAAM driver hangs in 4.9/4.14 kernel trees in QorIQ

CAAM driver hangs in 4.9/4.14 kernel trees

jimbrooke-l3 — Thu, 31 May 2018 13:16:54 GMT

Hi,

In the old 4.1.35-based kernel tree that was included with the "old" SDK, the CAAM driver worked well and I was able to do full-disk encryption via DM Crypt / LUKS / cryptsetup. With both the 4.9 and 4.14 kernel trees from LSDK18.03 the driver hangs while doing IO to a dm crypt target.

Has anyone else run into this problem?

sysrq-t shows that the workqueues that are in charge of doing the encryption/decryption are waiting for completion:

[ 595.478281] kworker/u2:0 D 0 6 2 0x00000000
[ 595.478293] Workqueue: kcryptd kcryptd_crypt

[ 595.478296] Call trace:
[ 595.478302] [<ffff000008085ea4>] __switch_to+0xa4/0xc0
[ 595.478308] [<ffff000008c23e88>] __schedule+0x188/0x590
[ 595.478314] [<ffff000008c242c8>] schedule+0x38/0xa0
[ 595.478321] [<ffff000008c2726c>] schedule_timeout+0x17c/0x240
[ 595.478328] [<ffff000008c24ecc>] wait_for_common+0x9c/0x140
[ 595.478334] [<ffff000008c24f84>] wait_for_completion+0x14/0x20
[ 595.478339] [<ffff000008987a9c>] crypt_convert+0x2bc/0x490
[ 595.478344] [<ffff000008987ea4>] kcryptd_crypt+0x234/0x3a0
[ 595.478350] [<ffff0000080d74d8>] process_one_work+0x1d8/0x370
[ 595.478356] [<ffff0000080d76b8>] worker_thread+0x48/0x4c0
[ 595.478361] [<ffff0000080ddcb8>] kthread+0xe8/0xf0
[ 595.478366] [<ffff0000080836c0>] ret_from_fork+0x10/0x50

Leaving the "hang" overnight leaves additional messages on the console (with caam debugging enabled):

[ 1805.852680] caam_jr 1710000.jr: caam_read: start reading at buffer 0, idx 2576
[ 1805.852692] caam_jr 1710000.jr: caam_read: start reading at buffer 0, idx 2640
[ 1805.852700] caam_jr 1710000.jr: caam_read: start reading at buffer 0, idx 2704
[ 1805.852707] caam_jr 1710000.jr: caam_read: start reading at buffer 0, idx 2768

There were approx 39 of these messages every 85-95 minutes. I'm not sure if this indicates that some forward progress is being made. The "idx" continued to increment at each printing.

Re: CAAM driver hangs in 4.9/4.14 kernel trees

_andev_ — Fri, 22 Jun 2018 10:34:37 GMT

Hi Jim,

Any hint about this issue?

I'm having a similar problem, the same call trace from the dmesg. I'm using the imx_4.9.11_1.0.0_ga kernel.

Now I'm testing a kernel with the next options:

# CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API is not set
# CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API is not set
# CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API is not set

and it seems the dm-crypt task is not hanging waiting for data from CAAM... Now the decryption is done by software but it works. It's hard to debug this issue because not always fails, but there is a problem is the CAAM driver...

Re: CAAM driver hangs in 4.9/4.14 kernel trees

_andev_ — Fri, 22 Jun 2018 10:35:41 GMT

Maybe this https://community.nxp.com/thread/433751 is related

Re: CAAM driver hangs in 4.9/4.14 kernel trees

yipingwang — Fri, 29 Jun 2018 08:01:37 GMT

Hello All,

CAAM driver has a defect where it doesn’t keep the requests in a backlog when the job ring queue is full, even if CRYPTO_TFM_REQ_MAY_BACKLOG flag is set, and so it doesn’t use a callback function to signal a -EINPROGRESS when the backlog request is handled.

It just returns -EBUSY is this situation and it doesn’t signal when there is free space in job ring queue nor when there is no request to be handled.

Dm-crypt on the other side, when a -EBUSY value is received, it waits for a -EINPROGRESS value so that he knows he can send requests again.

This defect in CAAM makes that, if there are a lot of requests and CAAM job ring queue becomes full, dm-crypt will wait indefinitely for a signal from CAAM driver because this signal will never be send by any callback functions in CAAM.

The problem is very well explained here:
https://www.redhat.com/archives/dm-devel/2015-May/msg00025.html

A full discussion on this topic can be found here:

https://patchwork.kernel.org/patch/6318181/

Please refer to the attached patch on CAAM driver.

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: CAAM driver hangs in 4.9/4.14 kernel trees

_andev_ — Tue, 03 Jul 2018 13:03:02 GMT

Hi!

I'm trying to apply the patch to the 4.9.11 kernel from imx-4.9.11-1.0.0_ga but there are errors... Maybe is for 4.14?

Regards

Re: CAAM driver hangs in 4.9/4.14 kernel trees

sehun_yu — Fri, 06 Jul 2018 06:26:19 GMT

Nice to meet you.

There is the patch at patchwork site. I've resolved this issue after applying the patch.

https://patchwork.kernel.org/patch/7199241/

This patch more easier to apply than before.

Re: CAAM driver hangs in 4.9/4.14 kernel trees

jimbrooke-l3 — Wed, 18 Jul 2018 18:42:42 GMT

Thanks Sehun Yu. This is the correct approach. I have CAAM working correctly now.

This is very frustrating. A patch that originated from within Freescale more than 2 years ago is absent from their kernel tree, resulting in a driver that hangs.

(As an aside, this is the *exact* same feature that's was broken in the older SEC hardware and the Talitos hardware. I ended up fixing that and sending those patches back upstream at one point)

Thanks again for the pointer.

Re: CAAM driver hangs in 4.9/4.14 kernel trees

_andev_ — Mon, 03 Sep 2018 08:58:22 GMT

Hi,

Still having issues applying the patch to a 4.9.11 kernel.

Applying patch v2-crypto-caam-add-backlogging-support.diff
patching file drivers/crypto/caam/caamalg.c
Hunk #1 succeeded at 2001 (offset 186 lines).
Hunk #2 succeeded at 2011 (offset 186 lines).
Hunk #3 succeeded at 2027 (offset 186 lines).
Hunk #4 succeeded at 2043 (offset 186 lines).
Hunk #5 succeeded at 2061 with fuzz 1 (offset 189 lines).
Hunk #6 succeeded at 2120 (offset 227 lines).
Hunk #7 FAILED at 1911.
Hunk #8 succeeded at 2121 (offset 195 lines).
Hunk #9 succeeded at 2500 (offset 194 lines).
Hunk #10 succeeded at 2552 (offset 194 lines).
Hunk #11 succeeded at 2595 (offset 194 lines).
Hunk #12 succeeded at 2655 (offset 196 lines).
Hunk #13 succeeded at 2806 (offset 187 lines).
Hunk #14 succeeded at 2851 (offset 187 lines).
Hunk #15 succeeded at 3000 (offset 184 lines).
1 out of 15 hunks FAILED -- rejects in file drivers/crypto/caam/caamalg.c
patching file drivers/crypto/caam/caamhash.c
Hunk #1 succeeded at 856 (offset 211 lines).
Hunk #2 succeeded at 876 (offset 211 lines).
Hunk #3 succeeded at 896 (offset 211 lines).
Hunk #4 succeeded at 914 (offset 211 lines).
Hunk #5 succeeded at 934 (offset 211 lines).
Hunk #6 succeeded at 952 (offset 211 lines).
Hunk #7 succeeded at 972 (offset 211 lines).
Hunk #8 succeeded at 990 (offset 211 lines).
Hunk #9 FAILED at 884.
Hunk #10 FAILED at 980.
Hunk #11 FAILED at 1070.
Hunk #12 succeeded at 1422 (offset 273 lines).
Hunk #13 succeeded at 1485 (offset 267 lines).
Hunk #14 FAILED at 1324.
Hunk #15 succeeded at 1712 (offset 286 lines).
Hunk #16 FAILED at 1537.
5 out of 16 hunks FAILED -- rejects in file drivers/crypto/caam/caamhash.c
patching file drivers/crypto/caam/intern.h
patching file drivers/crypto/caam/jr.c
Hunk #1 succeeded at 161 (offset -7 lines).
Hunk #2 FAILED at 202.
Hunk #3 succeeded at 225 (offset -7 lines).
Hunk #4 succeeded at 302 (offset -5 lines).
Hunk #5 succeeded at 412 with fuzz 1 (offset -6 lines).
Hunk #6 FAILED at 428.
2 out of 6 hunks FAILED -- rejects in file drivers/crypto/caam/jr.c

I can try to solve the issues but I would like to know what kernel is supposed this patch is for

Thanks!

Re: CAAM driver hangs in 4.9/4.14 kernel trees

maumontadrien — Tue, 16 Oct 2018 14:26:50 GMT

Hi, I have the same problem to apply the patch with 4.14.47 (LSDK 18.06).

Applying patch crypto-caam-add-backlogging-support.diff
patching file drivers/crypto/caam/caamalg.c
Hunk #1 succeeded at 863 (offset -1058 lines).
Hunk #2 succeeded at 879 (offset -1052 lines).
Hunk #3 succeeded at 897 with fuzz 2 (offset -1050 lines).
misordered hunks! output would be garbled
Hunk #4 FAILED at 855.
Hunk #5 FAILED at 1981.
Hunk #6 succeeded at 962 with fuzz 1 (offset -1034 lines).
Hunk #7 FAILED at 2014.
Hunk #8 FAILED at 2029.
Hunk #9 succeeded at 1359 (offset -1043 lines).
Hunk #10 succeeded at 1411 (offset -1043 lines).
Hunk #11 succeeded at 1454 (offset -1043 lines).
Hunk #12 succeeded at 1510 (offset -1045 lines).
Hunk #13 succeeded at 1682 (offset -1030 lines).
Hunk #14 succeeded at 1728 (offset -1030 lines).
Hunk #15 succeeded at 1910 (offset -998 lines).
4 out of 15 hunks FAILED -- rejects in file drivers/crypto/caam/caamalg.c
patching file drivers/crypto/caam/caamhash.c
Hunk #1 FAILED at 650.
Hunk #2 succeeded at 612 (offset -54 lines).
Hunk #3 FAILED at 686.
Hunk #4 succeeded at 613 (offset -89 lines).
Hunk #5 FAILED at 722.
Hunk #6 succeeded at 614 (offset -124 lines).
Hunk #7 FAILED at 758.
Hunk #8 succeeded at 615 (offset -159 lines).
Hunk #9 FAILED at 880.
Hunk #10 FAILED at 977.
Hunk #11 FAILED at 1069.
Hunk #12 succeeded at 1039 (offset -110 lines).
Hunk #13 succeeded at 1101 (offset -117 lines).
Hunk #14 FAILED at 1326.
Hunk #15 succeeded at 1323 (offset -106 lines).
Hunk #16 FAILED at 1540.
9 out of 16 hunks FAILED -- rejects in file drivers/crypto/caam/caamhash.c
patching file drivers/crypto/caam/intern.h
Hunk #1 succeeded at 12 (offset 1 line).
Hunk #2 succeeded at 40 (offset 1 line).
patching file drivers/crypto/caam/jr.c
Hunk #1 succeeded at 179 (offset 11 lines).
Hunk #2 FAILED at 202.
Hunk #3 succeeded at 243 (offset 11 lines).
Hunk #4 succeeded at 287 (offset 11 lines).
Hunk #5 succeeded at 357 (offset 41 lines).
Hunk #6 succeeded at 465 with fuzz 1 (offset 41 lines).
Hunk #7 FAILED at 434.
2 out of 7 hunks FAILED -- rejects in file drivers/crypto/caam/jr.c
patching file drivers/crypto/caam/jr.h
Hunk #1 succeeded at 18 (offset 3 lines).
Patch crypto-caam-add-backlogging-support.diff does not apply (enforce with -f)

Is there a newest version of this patch ?

Thanks,

Adrien.