CAAM and kernel crypto tests

[EricNelson]

Hi all,

We're seeing some disconcerting error messages on various versions of Linux and various boards when enabling CAAM and

the kernel self-tests.

Testing on SABRE-SD with 3.10.31-alpha

root@imx6qsabresd:~# dmesg
...
alg: hash: Test 1 failed for hmac-sha1-caam
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000010: f1 46 be 00
alg: hash: Test 1 failed for sha1-caam
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000010: 9c d0 d8 9d
alg: hash: Test 1 failed for hmac-sha224-caam
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000010: 47 b4 b1 16 99 12 ba 4f 53 68 4b 22
alg: hash: Test 1 failed for sha224-caam
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000010: 2a ad bc e4 bd a0 b3 f7 e3 6c 9d a7

root@imx6qsabresd:~# cat /proc/cmdline
console=ttymxc0,115200 ip=dhcp root=/dev/mmcblk0p2 rootwait caam

The kernel configuration is essentially the same as the stock v7_defconfig with a few additional debug flags:

~/linux-imx6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- savedefconfig
scripts/kconfig/conf --savedefconfig=defconfig Kconfig

~/linux-imx6$ diff defconfig arch/arm/configs/imx_v7_defconfig
253a254
> CONFIG_USB_PHY=y
329d329
< CONFIG_DEBUG_LOCK_ALLOC=y
334d333
< # CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set

We've seen the same results with kernel 3.0.35_4.1.0 and 3.10.17-1.0.0-ga on Nitrogen6x.

Does anyone have some insight about this?

Thanks in advance,

Eric

[EricNelson · Accepted solution]

Thanks to Steve Cornelius and others at Freescale, we've been able to show that this issue has to do with cache coherency.

In particular, the CAAM drivers expect all inputs and outputs to be aligned with on a cache line boundary (32 bytes on i.MX6).

Since the testmgr code was originally created to test software implementations, as of kernel 3.10.17, it didn't ensure any particular alignment, which caused the failures, and a simple update to address this prevents the errors:

         crypto testmgr: Align result buffer on cache-line · e3e7ad4 · boundarydevices/linux-imx6 · GitHub

As mentioned in the commit log, this requirement for inputs and outputs also exists in userspace code, so applications will need to take care in allocations.

[jonnettleton]

There are actually proper fixes to this problem in Herbert Xu's crypto-2.6 tree.  Specifically LKML: Herbert Xu: Crypto Fixes for 3.13 and LKML: Herbert Xu: Re: Crypto Fixes for 3.13 fix the problems.  They would need to be backported to 3.10.xx

[EricNelson]

Many thanks, Jon

[EricNelson]

Additional notes:

The values shown in the test failures:

alg: hash: Test 1 failed for hmac-sha1-caam  
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
00000010: f1 46 be 00  

Appear to be similar to the expected results. This is the first test vector for hmac-sha1 in crypto/testmgr.c:

static struct hash_testvec hmac_sha1_tv_template[] = {
  {
  .key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b",
  .ksize = 20,
  .plaintext = "Hi There",
  .psize = 8,
  .digest = "\xb6\x17\x31\x86\x55\x05\x72\x64"
   "\xe2\x8b\xc0\xb6\xfb\x37\x8c\x8e\xf1"
   "\x46\xbe",
...

Note that the sequence "f1 46 be" matches the expected value.

Also note that the first 16 bytes of each of the failed tests is showing as zero.

[EricNelson]

Yet more information regarding these tests...

A udelay(20) call at the end of caam_jr_enqueue() allows the tests to succeed, so it appears to be some form of timing bug.

~/linux-imx6$ git diff
diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 80ddddb..c8c4a7e 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -288,6 +288,7 @@ int caam_jr_enqueue(struct device *dev, u32 *desc,

        spin_unlock_bh(&jrp->inplock);

+       udelay(20);
        return 0;
 }
 EXPORT_SYMBOL(caam_jr_enqueue);

[EricNelson]

Yet more detail:

The same udelay(20) in either of these two spots in caam_jr_dequeue() do not result in successful tests:

     linux-imx6/drivers/crypto/caam/jr.c at boundary-imx_3.10.17_1.0.0_ga · boundarydevices/linux-im...

     linux-imx6/drivers/crypto/caam/jr.c at boundary-imx_3.10.17_1.0.0_ga · boundarydevices/linux-imx6 · ...

[karina_valencia]

jamesbone can you continue with the follow up on this case?

[seungwonkim]

Is this problem fixed?? or, solution is just delay???

I have the problem too. but, I need to change delay value up to 100usec for successful test.

It is big problem.

[EricNelson]

Not fixed yet.

[karina_valencia]

jamesbone do you have an update of this case?

[EricNelson]

Further update: In a subsequent boot on 3.10.17-1.0.0-ga, I'm seeing only 8 bytes of zeros:

alg: hash: Test 1 failed for hmac-sha1-caam
00000000: 00 00 00 00 00 00 00 00 e2 8b c0 b6 fb 37 8c 8e
00000010: f1 46 be 00
alg: hash: Test 1 failed for sha1-caam

[EricNelson]

Hi Jaime,

I think you missed that we're testing against 3.0.35_4.1.0, 3.10.17-1.0.0-ga, and 3.10.31_1.0.0-alpha.

Each of them already contains the patches from the other page:

Patch to dma_map_sg_chained:

     http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/commit/?h=imx_3.0.35_4.0.0&id=b30237c790...

3.10.17 sources:

     https://github.com/boundarydevices/linux-imx6/blob/boundary-imx_3.10.17_1.0.0_ga/drivers/crypto/caam...

Patch to driver_algs:

     http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/commit/?h=imx_3.0.35_4.0.0&id=6068d7a77b...

3.10.17 sources:

     https://github.com/boundarydevices/linux-imx6/blob/boundary-imx_3.10.17_1.0.0_ga/drivers/crypto/caam...

I also wasn't clear that the problem we're seeing is only with the hash algorithms.

Regards,

Eric

[jamesbone]

Dear Eric,

On linux 3.0.35 there is a need for some patches in order to get  proper functionality of CAAM

Q&A: Why is CAAM Driver Not Functioning in Linux for iMX6?

Have a great day,
Jaime

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------