Linux5.10 panics on LS1028A RDB Board in caam_jr driver (LS1028A sdk patches included)

[jianhan]

Hi, I met with one issue.

When I run linux 5.10 on "LS1028A RDB Board" (LS1028A sdk patches included, I mean I had ported the patches from "git@github.com:nxp-qoriq/linux.git" , which is not included in linux-6.1 yet.) 

I was doing misc tests with several scripts,including restarts some services in linux, instead of booting up.

then below panic happened. This only happened once, and I think may be there is on bug in caam driver?

<5>arm-smmu 5000000.iommu: disabling translation
<3>caam_jr 8010000.jr: job ring error: irqstate: 10440103
<4>------------[ cut here ]------------
<2>kernel BUG at drivers/crypto/caam/jr.c:283!
<0>Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
<4>Modules linked in: ip6table_mangle ip6table_nat iptable_mangle iptable_filter sch_dsmark ip6table_filter xt_comment xt_tcpudp ip6_tables xt_NFLOG iptable_nat ip_tables x_tables nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc [last unloaded: tpm_ftpm_tee]
<4>CPU: 0 PID: 163 Comm: hwrng Tainted: G W O 5.10.154-yocto-standard #1
<4>Hardware name: freescale ls1028a/ls1028a, BIOS 2019.10+fsl+gd1aa7f1b3e 12/21/2021
<4>pstate: 60000085 (nZCv daIf -PAN -UAO -TCO BTYPE=--)
<4>pc : caam_jr_interrupt+0x128/0x130
<4>lr : caam_jr_interrupt+0x128/0x130
<4>sp : ffff80001144be50
<4>x29: ffff80001144be50 x28: ffff800010f61008
<4>x27: ffff800011228000 x26: ffff800010f61008
<4>x25: ffff000027b78800 x24: 0000000000000072
<4>x23: ffff8000113ba0c0 x22: 0000000000000001
<4>x21: ffff800011433000 x20: ffff000027b78e80
<4>x19: 0000000010440103 x18: 0000000000000030
<4>x17: 0000000000000000 x16: 0000000000000000
<4>x15: ffffffffffffffff x14: ffff8000113ebcb8
<4>x13: 0000000000000008 x12: fffffffffffcaa2f
<4>x11: 0000000000000003 x10: ffff8000112a1e50
<4>x9 : ffff8000100a99c0 x8 : ffff800011249e50
<4>x7 : ffff8000112a1e50 x6 : 0000000000000000
<4>x5 : 0000000000000000 x4 : 0000000000000000
<4>x3 : 0000000000000000 x2 : 0000000000000000
<4>x1 : 0000000000000000 x0 : ffff000027422a00
<4>Call trace:
<4> caam_jr_interrupt+0x128/0x130
<4> __handle_irq_event_percpu+0x84/0x2b0
<4> handle_irq_event+0x6c/0xfc
<4> handle_fasteoi_irq+0xc8/0x230
<4> __handle_domain_irq+0xb8/0x130
<4> gic_handle_irq+0x90/0x158
<4> el1_irq+0xcc/0x180
<4> do_softirq+0x5c/0xa4
<4> __local_bh_enable_ip+0xa0/0xac
<4> _raw_spin_unlock_bh+0x40/0x70
<4> caam_jr_enqueue+0x16c/0x330
<4> caam_rng_read_one.constprop.0+0x1b8/0x370
<4> caam_read+0x8c/0xb0
<4> hwrng_fillfn+0xfc/0x1cc
<4> kthread+0x14c/0x160
<4> ret_from_fork+0x10/0x30
<0>Code: 2a1303e2 d00029a1 91048021 940b26cd (d4210000)
<4>---[ end trace 5d619b691e532e3b ]---
<0>Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
<0>Kernel Offset: disabled
<0>CPU features: 0x28040022,21002008
<0>Memory Limit: none
<4>---- STARTING TO DUMP BT-ALL
<4>U

Thanks in advance for any help!

[jianhan]

<5>arm-smmu 5000000.iommu: disabling translation
<3>caam_jr 8010000.jr: job ring error: irqstate: 10440103
<4>------------[ cut here ]------------
<2>kernel BUG at drivers/crypto/caam/jr.c:283!

From first several lines of the log, the first line is printed out from "drivers/iommu", then caam_jr crashes happens.

I think after "iommu" module is removed from driver, any driver may not work, may be there should be some dependency.

[khushbur]

Hi @jianhan 

Can you please try applying below patch.

https://patchwork.kernel.org/patch/7199241/

Thanks

Khushbu

[jianhan]

Hi, @khushbur 
Sorry for my late response.

Thanks for your reply and help, I analyzed the source codes:
caam_jr driver should be removed before iommu drivers removed, since caam_jr driver depends on iommu drivers.

"arm-smmu 5000000.iommu: disabling translation" shows iommu driver removed, then camm_jr irq comes.

after I picked from upstream, it works:)

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c007e720989e3a39923d77...

Anyway, thank you very much!

[kunsong]

hi, @jianhan 

Will the problem not happen again after you add this patch?

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c007e720989e3a39923d77...

I encountered a similar problem, but the problem still exists.

But it is difficult for us to reproduce the problem

caam_jr 8030000.jr: Device is busy
caam_jr 8020000.jr: Device is busy
caam_jr 8010000.jr: Device is busy
arm-smmu 5000000.iommu: disabling translation
reboot: Restarting system

the logs before the previous patch was introduced and Device is busy message was not present.

<5>arm-smmu 5000000.iommu: disabling translation
<3>caam_jr 8010000.jr: job ring error: irqstate: 10440103

 If you know more please give me advice.

Thanks!

BR/SK