Kernel oops during data storage using key blob, kb_ioctl (cryptodev)

[yellapu_anishkh]

Hi NXP,

I am working on imx8dx based target.

Linux:4.14.98

We are using keyblob for storing and getting the keys/certificates, when we try to store the certificate, it internally uses blob generation and encryption functions, which internally calls "kb_ioctl" in the driver(drivers/crypto/caam/key_blob.c)

When we try to store some data, sometimes we are getting below issue. Can you please help us, why is this happening sometimes.

Issue1:

#113 I[ 265.198310] ------------[ cut here ]------------
[ 265.205957] kernel BUG at /usr/src/kernel/mm/slub.c:294!
[ 265.211271] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
NFO > [SequenceEngine:DiagCommEngine] : Session with sessoin I[ 265.216761] Modules linked in: cryptodev(O)
[ 265.226590] CPU: 0 PID: 2546 Comm: java Tainted: G O 4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 265.236244] Hardware name: Freescale i.MX8DX MEK (DT)
[ 265.241293] task: ffff8000180fe800 task.stack: ffff00000ef88000
[ 265.247225] PC is at kfree+0x1e8/0x1f8
[ 265.250975] LR is at kso_cleanup_data+0x20/0x28
[ 265.255505] pc : [<ffff0000081f7ee0>] lr : [<ffff0000085c7d60>] pstate: 60000145
[ 265.262896] sp : ffff00000ef8bcf0
[ 265.266207] x29: ffff00000ef8bcf0 x28: ffff8000182b4900
[ 265.271523] x27: 0000000000000020 x26: ffff80001a1fe700
[ 265.276840] x25: ffff8000182b4880 x24: 0000000000000001
[ 265.282157] x23: ffff0000085c7d60 x22: 00000000000bf6e4
[ 265.287473] x21: ffff800012403c80 x20: ffff80001819cf80
[ 265.292790] x19: ffff7e0000606700 x18: 0000ffff971fd194
[ 265.298107] x17: 0000ffffa5980100 x16: ffff00000813dcc0
[ 265.303424] x15: 000061c164000000 x14: 001406f400000000
[ 265.308740] x13: 0000000000000000 x12: 0000003dbf0c5fba
[ 265.314057] x11: 0000000000000000 x10: 00000000000008e0
[ 265.319374] x9 : ffff00000ef8bad0 x8 : ffff8000180ff140
[ 265.324691] x7 : ffff8000180fe980 x6 : 49856764efcba631
[ 265.330007] x5 : 0000000000000000 x4 : ffff80001819cf80
[ 265.335324] x3 : 000080001743d000 x2 : ffff80001819cf80
[ 265.340641] x1 : ffff000008b2bee0 x0 : ffff7e0000606700
[ 265.345959] Process java (pid: 2546, stack limit = 0xffff00000ef88000)
[ 265.352482] Call trace:
[ 265.354926] Exception stack(0xffff00000ef8bbb0 to 0xffff00000ef8bcf0)
[ 265.361373] bba0: ffff7e0000606700 ffff000008b2bee0
[ 265.369209] bbc0: ffff80001819cf80 000080001743d000 ffff80001819cf80 0000000000000000
[ 265.377046] bbe0: 49856764efcba631 ffff8000180fe980 ffff8000180ff140 ffff00000ef8bad0
[ 265.384881] bc00: 00000000000008e0 0000000000000000 0000003dbf0c5fba 0000000000000000
[ 265.392718] bc20: 001406f400000000 000061c164000000 ffff00000813dcc0 0000ffffa5980100
[ 265.400554] bc40: 0000ffff971fd194 ffff7e0000606700 ffff80001819cf80 ffff800012403c80
[ 265.408390] bc60: 00000000000bf6e4 ffff0000085c7d60 0000000000000001 ffff8000182b4880
[ 265.416226] bc80: ffff80001a1fe700 0000000000000020 ffff8000182b4900 ffff00000ef8bcf0
[ 265.424062] bca0: ffff0000085c7d60 ffff00000ef8bcf0 ffff0000081f7ee0 0000000060000145
[ 265.431898] bcc0: 0000000000000000 0000000000000020 0000ffffffffffff 0000000000000020
[ 265.439733] bce0: ffff00000ef8bcf0 ffff0000081f7ee0
[ 265.444616] [<ffff0000081f7ee0>] kfree+0x1e8/0x1f8
[ 265.449412] [<ffff0000085c7d60>] kso_cleanup_data+0x20/0x28
[ 265.454989] [<ffff0000085c79e4>] sm_release_keystore+0x1c/0x30
[ 265.460827] [<ffff0000085cc85c>] kb_ioctl+0x2d4/0x738
[ 265.465884] [<ffff000008228efc>] do_vfs_ioctl+0xa4/0x8e8
[ 265.471199] [<ffff0000082297bc>] SyS_ioctl+0x7c/0x98
[ 265.476167] Exception stack(0xffff00000ef8bec0 to 0xffff00000ef8c000)
[ 265.482607] bec0: 0000000000000124 00000000c0244b04 0000ffff56b84e80 0000000000000001
[ 265.490443] bee0: 0000000000000125 0000000000000000 0000000000000000 fefeff6a62606b61
[ 265.498279] bf00: 000000000000001d cfa95fbf65be5be4 0101010101010101 0000000000000003
[ 265.506115] bf20: cd7e31f5626cf658 6548eba4563d4c25 000000000000004f 0000000000000000
[ 265.513952] bf40: 0000ffff952ab528 0000ffffa596b910 0000ffffa59f3a70 0000ffff9808af38
[ 265.521788] bf60: 0000ffff56b86050 0000ffffa5844130 0000000000000000 0000000000000005
[ 265.529624] bf80: 0000ffff56b860c8 00000000f91ad238 0000ffff5d9a1ba8 0000000000000000
[ 265.537460] bfa0: 0000ffff5815d800 0000ffff56b84e10 0000ffff9528df30 0000ffff56b84e10
[ 265.545296] bfc0: 0000ffffa596b91c 0000000020000000 0000000000000124 000000000000001d
[ 265.553132] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 265.560969] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 265.566547] Code: 97fffd2c a9425bb5 17ffffd9 d503201f (d4210000)
[ 265.572648] ---[ end trace 835bf4fb1829feed ]---
[ 265.577265] Kernel panic - not syncing: Fatal exception
[ 265.582494] SMP: stopping secondary CPUs
[ 265.586413] Kernel Offset: disabled
[ 265.589905] CPU features: 0x0802008
[ 265.593386] Memory Limit: none
[ 265.596438] Rebooting in 10 seconds..

Issue2: (This is coming from different location in kb_ioctl)

[ 418.582943] ISV = 0, ISS = 0x00000004
[ 418.593123] CM = 0, WnR = 0
[ 418.602490] [007f71028331b6f2] address between user and kernel address ranges
[ 418.615337] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 418.625082] Modules linked in: cryptodev(O)
[ 418.629281] CPU: 0 PID: 2596 Comm: java Tainted: G O 4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 418.638933] Hardware name: Freescale i.MX8DX MEK (DT)
[ 418.643982] task: ffff8000061d8000 task.stack: ffff00000f268000
[ 418.655554] PC is at __memcpy+0x2c/0x180
[ 418.659485] LR is at kb_ioctl+0x288/0x738
[ 418.663494] pc : [<ffff00000883302c>] lr : [<ffff0000085cc810>] pstate: 00000145
[ 418.670894] sp : ffff00000f26bd50
[ 418.674204] x29: ffff00000f26bd50 x28: ffff80001a058800
[ 418.679521] x27: 0000000000000020 x26: ffff80001a20c100
[ 418.684838] x25: ffff80001a058f80 x24: 0000000000000001
[ 418.690154] x23: ffff00000f26bdd8 x22: ffff8000121d4010
[ 418.695471] x21: 0000000000000020 x20: ffff80001a058080
[ 418.700788] x19: 0000ffff6483ccd0 x18: 0000ffffae5cba70
[ 418.706105] x17: 0000ffffae543910 x16: ffff000008229740
[ 418.711421] x15: 0000000000000000 x14: 0000000000000000
[ 418.716738] x13: 0000000000000000 x12: 0000006171e95db8
[ 418.722055] x11: 0000000000000001 x10: 00000000000008e0
[ 418.727371] x9 : 0000000000000000 x8 : ffff80001a058100
[ 418.732688] x7 : 0000000000000000 x6 : ffff80001a058080
[ 418.738005] x5 : 0000000000000040 x4 : 000000000000000e
[ 418.743321] x3 : 0000000000000200 x2 : 0000000000000012
[ 418.748639] x1 : ff7f71028331b6f2 x0 : ffff80001a058080
[ 418.753957] Process java (pid: 2596, stack limit = 0xffff00000f268000)
[ 418.760489] Call trace:
[ 418.762932] Exception stack(0xffff00000f26bc10 to 0xffff00000f26bd50)
[ 418.769379] bc00: ffff80001a058080 ff7f71028331b6f2
[ 418.777216] bc20: 0000000000000012 0000000000000200 000000000000000e 0000000000000040
[ 418.785052] bc40: ffff80001a058080 0000000000000000 ffff80001a058100 0000000000000000
[ 418.792888] bc60: 00000000000008e0 0000000000000001 0000006171e95db8 0000000000000000
[ 418.800724] bc80: 0000000000000000 0000000000000000 ffff000008229740 0000ffffae543910
[ 418.808561] bca0: 0000ffffae5cba70 0000ffff6483ccd0 ffff80001a058080 0000000000000020
[ 418.816397] bcc0: ffff8000121d4010 ffff00000f26bdd8 0000000000000001 ffff80001a058f80
[ 418.824233] bce0: ffff80001a20c100 0000000000000020 ffff80001a058800 ffff00000f26bd50
[ 418.832069] bd00: ffff0000085cc810 ffff00000f26bd50 ffff00000883302c 0000000000000145
[ 418.839905] bd20: ffff00000f26bdd8 0000000000000001 0000ffffffffffff ffff80001a20c100
[ 418.847739] bd40: ffff00000f26bd50 ffff00000883302c
[ 418.852624] [<ffff00000883302c>] __memcpy+0x2c/0x180
[ 418.857596] [<ffff000008228efc>] do_vfs_ioctl+0xa4/0x8e8
[ 418.862908] [<ffff0000082297bc>] SyS_ioctl+0x7c/0x98
[ 418.867876] Exception stack(0xffff00000f26bec0 to 0xffff00000f26c000)
[ 418.874325] bec0: 0000000000000124 00000000c0244b04 0000ffff6483ccd0 0000000000000001
[ 418.882161] bee0: 0000000000000125 0000000000000000 0000000000000000 fefeff6a62606b61
[ 418.889998] bf00: 000000000000001d cfa95fbf65be5be4 0101010101010101 0000000000000003
[ 418.897833] bf20: cd7e31f5626cf658 6548eba4563d4c25 000000000000004f 0000000000000000
[ 418.905669] bf40: 0000ffff8de7b528 0000ffffae543910 0000ffffae5cba70 0000ffffa008af38
[ 418.913506] bf60: 0000ffff6483dea0 0000ffffae41c130 0000000000000000 00000000fb25ce40
[ 418.921342] bf80: 0000ffff6483df18 00000000f91f5ab8 0000ffff6687bba8 0000000000000000
[ 418.929178] bfa0: 0000ffff6c037000 0000ffff6483cc60 0000ffff8de5df30 0000ffff6483cc60
[ 418.937014] bfc0: 0000ffffae54391c 0000000020000000 0000000000000124 000000000000001d
[ 418.944849] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 418.952687] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 418.958265] Code: 36000064 38401423 380014c3 36080064 (78402423)
[ 418.964367] ---[ end trace 26fdd9f03c114792 ]---
[ 418.968982] Kernel panic - not syncing: Fatal exception
[ 418.974212] SMP: stopping secondary CPUs
[ 418.978144] Kernel Offset: disabled
[ 418.981632] CPU features: 0x0802008
[ 418.985121] Memory Limit: none
[ 418.988176] Rebooting in 10 seconds..

[igorpadykov]

Hi Anish

reason may be in software version, as for example "Linux:4.14.98"

support i.MX8QXP only starting from Linux 4.14.98_2.3.0, also please

review   Linux 4.14.98_2.3.1 Patch :      i.MX 8DualX MEK (1GB MEM)

described on documentation link

Embedded Linux for i.MX Applications Processors | NXP 

Also all parts of software like, SECO, SCFW Porting Kit, e.t.c.

should be linked to each other, not recommended to mix them from other

bsp revisions. Had board passed ddr test, as this also may be reason for

"Kernel panic - not syncing" errors. Recommended to use latest RPA tool

from below link, following "Notes" for recommended SCFW version.

i.MX 8/8X Family DDR Tools Release  

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yellapu_anishkh]

Hi igorpadykov,

Thank you for the information. Sorry for the delayed feedback.

Current test setup: We have created 512 DDR configuration using latest tool, using the same file  on 1GB hardware.

Still we are facing oops from USB and key blob drivers. 

Can you please confirm, issue can be from driver side or memory. As in all of the failures PC is at kmalloc, kfree and slab_free.

USB Driver Issue:

[ 913.340663] Unable to handle kernel paging request at virtual address 8071028819747a
[ 913.349627] Mem abort info:
[ 913.358206] Exception class = DABT (current EL), IL = 32 bits
[ 913.369635] SET = 0, FnV = 0
[ 913.378778] EA = 0, S1PTW = 0
[ 913.387176] Data abort info:
[ 913.405086] CM = 0, WnR = 0
[ 913.413670] [008071028819747a] address between user and kernel address ranges
[ 913.426424] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 913.437610] Modules linked in: cryptodev(O)
[ 913.447441] CPU: 1 PID: 2235 Comm: java Tainted: G O 4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 913.462732] Hardware name: Freescale i.MX8DX MEK (DT)
[ 913.473428] task: ffff80001b94e800 task.stack: ffff00000e838000
[ 913.485001] PC is at __kmalloc+0xb8/0x270
[ 913.494643] LR is at __kmalloc+0x28/0x270
[ 913.504295] pc : [<ffff0000081f6c20>] lr : [<ffff0000081f6b90>] pstate: 60000145
[ 913.517333] sp : ffff00000e83b830
[ 913.526290] x29: ffff00000e83b830 x28: ffff800016e80c00
[ 913.537245] x27: ffff800016e80c00 x26: 008071028819747a
[ 913.548199] x25: 0000000000000004 x24: 0000000000000001
[ 913.559154] x23: 0000000000067d89 x22: ffff000008525ea8
[ 913.570109] x21: 00000000014080c0 x20: 008071028819747a
[ 913.581064] x19: ffff800012403c80 x18: 0000000000000001
[ 913.592019] x17: 0000ffff8ea16808 x16: ffff000008211bd8
[ 913.602974] x15: 0000000000000000 x14: 0000000000000000
[ 913.613929] x13: ffff000008f2de30 x12: 0000000000000400
[ 913.624884] x11: ffff00000e83ba60 x10: 0000000000000040
[ 913.635839] x9 : 0000000000000000 x8 : ffff800006198c28
[ 913.646794] x7 : 0000000000000004 x6 : 0000000000000000
[ 913.657749] x5 : 0000000000000001 x4 : ffff80001a92fc00
[ 913.668704] x3 : 0080f1029b269272 x2 : 0000000000000000
[ 913.679659] x1 : ffff80001b94e800 x0 : 0000000000000000
[ 913.690615] Process java (pid: 2235, stack limit = 0xffff00000e838000)
[ 913.702785] Call trace:
[ 913.710876] Exception stack(0xffff00000e83b6f0 to 0xffff00000e83b830)
[ 913.722961] b6e0: 0000000000000000 ffff80001b94e800
[ 913.736435] b700: 0000000000000000 0080f1029b269272 ffff80001a92fc00 0000000000000001
[ 913.749910] b720: 0000000000000000 0000000000000004 ffff800006198c28 0000000000000000
[ 913.852948] Unable to handle kernel paging request at virtual address 7f710280a3d6f2
[ 913.857904] b740: 0000000000000040 ffff00000e83ba60 0000000000000400 ffff000008f2de30

[igorpadykov]

Hi Anish

based on below log, this is not usb but memory issue

>USB Driver Issue:
>Unable to handle kernel paging request at virtual address 8071028819747a
>Mem abort info:

>>We have created 512 DDR configuration using latest tool, using the same file  on 1GB hardware.

for 512 DDR configuration "same file  on 1GB hardware" should not be used.

Software modiciations should be performed using documentation from

i.MX 8/8X Family DDR Tools Release  

Suggest to try first configuration without OPTee, follow sect.5.6.10 OP-TEE enablement

i.MX Yocto Project User’s Guide

Best regards
igor