i.MX8MQ is having kernel panic

[harshitshah]

Hi Team,

We are using the i.MX8MQ based board, during boot-up we have found that it is giving below kernel crash.

We are using i.MX8MQ processor custom board on L5.4.3_1.0.0 version. (We are not planning to modify the same if this is not a known issue).

===============================================================================

[2021-12-16 10:25:01] [   69.370450] Unable to handle kernel paging request at virtual address ffff8000eabac000
[2021-12-16 10:25:01] [   69.378373] Mem abort info:
[2021-12-16 10:25:01] [   69.381164]   ESR = 0x86000005
[2021-12-16 10:25:01] [   69.384216]   EC = 0x21: IABT (current EL), IL = 32 bits
[2021-12-16 10:25:01] [   69.389525]   SET = 0, FnV = 0
[2021-12-16 10:25:01] [   69.392576]   EA = 0, S1PTW = 0
[2021-12-16 10:25:01] [   69.395716] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000041992000
[2021-12-16 10:25:01] [   69.402414] [ffff8000eabac000] pgd=000000013ffff003, pud=0000000000000000
[2021-12-16 10:25:01] [   69.409203] Internal error: Oops: 86000005 [#1] PREEMPT SMP
[2021-12-16 10:25:01] [   69.414773] Modules linked in:
[2021-12-16 10:25:01] [   69.417830] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.3-lts-lf-5.4.y+gf811858 #1
[2021-12-16 10:25:01] [   69.425569] Hardware name: i.MX8MQ Processor (DT)
[2021-12-16 10:25:01] [   69.431660] pstate: 80000085 (Nzcv daIf -PAN -UAO)
[2021-12-16 10:25:01] [   69.436453] pc : 0xffff8000eabac000
[2021-12-16 10:25:01] [   69.439946] lr : ktime_get+0x40/0x98
[2021-12-16 10:25:01] [   69.443517] sp : ffff80001511bef0
[2021-12-16 10:25:01] [   69.446829] x29: ffff80001511bef0 x28: 0000000000000000 
[2021-12-16 10:25:01] [   69.452138] x27: 0000000000000000 x26: 0000000000000000 
[2021-12-16 10:25:01] [   69.457448] x25: 0000000000000000 x24: 0000000000000000 
[2021-12-16 10:25:01] [   69.462758] x23: ffff0000f80c1c00 x22: 00000000000060ba 
[2021-12-16 10:25:01] [   69.468068] x21: 0000000fed0e376b x20: ffff800014f11b00 
[2021-12-16 10:25:01] [   69.473377] x19: ffff0000ff791d70 x18: ffff0000f3234600 
[2021-12-16 10:25:01] [   69.478687] x17: 0000000000000000 x16: 0000000000000000 
[2021-12-16 10:25:01] [   69.483997] x15: ffff0000f20b3f80 x14: 0000000040000010 
[2021-12-16 10:25:01] [   69.489307] x13: ffff0000ff750000 x12: 0000000000000001 
[2021-12-16 10:25:01] [   69.494616] x11: 0000000000000000 x10: 00000000000009c0 
[2021-12-16 10:25:01] [   69.499926] x9 : ffff80001511be90 x8 : ffff0000f80c2620 
[2021-12-16 10:25:01] [   69.505236] x7 : 0000000000000000 x6 : ffff0000f83a46f0 
[2021-12-16 10:25:01] [   69.510546] x5 : 0000000000000007 x4 : 0000000000000000 
[2021-12-16 10:25:01] [   69.515855] x3 : ffff800010c7d140 x2 : 0000000000000000 
[2021-12-16 10:25:01] [   69.521165] x1 : ffff8000eabac000 x0 : ffff800014e9ff28 
[2021-12-16 10:25:01] [   69.526476] Call trace:
[2021-12-16 10:25:01] [   69.528920]  0xffff8000eabac000
[2021-12-16 10:25:01] [   69.532063]  tick_nohz_idle_enter+0x3c/0x70
[2021-12-16 10:25:01] [   69.536246]  do_idle+0x34/0x280
[2021-12-16 10:25:01] [   69.539386]  cpu_startup_entry+0x24/0x40
[2021-12-16 10:25:01] [   69.543310]  secondary_start_kernel+0x154/0x190
[2021-12-16 10:25:01] [   69.547844] Code: bad PC value
[2021-12-16 10:25:01] [   69.550903] ---[ end trace bc3e5a44512dce5e ]---
[2021-12-16 10:25:01] [   69.555520] Kernel panic - not syncing: Attempted to kill the idle task!
[2021-12-16 10:25:01] [   69.562219] SMP: stopping secondary CPUs
[2021-12-16 10:25:01] [   69.566146] Kernel Offset: disabled
[2021-12-16 10:25:01] [   69.569633] CPU features: 0x0002,2000200c
[2021-12-16 10:25:01] [   69.573639] Memory Limit: none
[2021-12-16 10:25:01] [   69.576695] Rebooting in 10 seconds..
[2021-12-16 10:25:13]

===============================================================================

[2021-12-16 10:26:39] [   79.198401] Unable to handle kernel paging request at virtual address ffff80001582c570
[2021-12-16 10:26:39] [   79.206324] Mem abort info:
[2021-12-16 10:26:39] [   79.209115]   ESR = 0x96000007
[2021-12-16 10:26:39] [   79.212168]   EC = 0x25: DABT (current EL), IL = 32 bits
[2021-12-16 10:26:39] [   79.217476]   SET = 0, FnV = 0
[2021-12-16 10:26:39] [   79.220528]   EA = 0, S1PTW = 0
[2021-12-16 10:26:39] [   79.223665] Data abort info:
[2021-12-16 10:26:39] [   79.226542]   ISV = 0, ISS = 0x00000007
[2021-12-16 10:26:39] [   79.230374]   CM = 0, WnR = 0
[2021-12-16 10:26:39] [   79.233341] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000041992000
[2021-12-16 10:26:39] [   79.240038] [ffff80001582c570] pgd=000000013ffff003, pud=000000013fffe003, pmd=0000000132307003, pte=0000000000000000
[2021-12-16 10:26:39] [   79.250649] Internal error: Oops: 96000007 [#1] PREEMPT SMP
[2021-12-16 10:26:39] [   79.256219] Modules linked in:
[2021-12-16 10:26:39] [   79.259276] CPU: 0 PID: 309 Comm: 30902000.jr-eng Not tainted 5.4.3-lts-lf-5.4.y+gf811858 #1
[2021-12-16 10:26:39] [   79.267709] Hardware name: i.MX8MQ Processor (DT)
[2021-12-16 10:26:39] [   79.273800] pstate: 80000085 (Nzcv daIf -PAN -UAO)
[2021-12-16 10:26:39] [   79.278597] pc : pick_next_task_idle+0x30/0x48
[2021-12-16 10:26:39] [   79.283039] lr : pick_next_task_idle+0x2c/0x48
[2021-12-16 10:26:39] [   79.287479] sp : ffff80001582bd60
[2021-12-16 10:26:39] [   79.290791] x29: ffff80001582bd60 x28: 0000000000000000 
[2021-12-16 10:26:39] [   79.296101] x27: ffff80001582bde0 x26: ffff0000f8372180 
[2021-12-16 10:26:39] [   79.301410] x25: ffff800010fa5ff0 x24: ffff800010f6fc98 
[2021-12-16 10:26:39] [   79.306720] x23: ffff0000f8371c00 x22: ffff800014bdd018 
[2021-12-16 10:26:39] [   79.312030] x21: 0000000000000000 x20: ffff0000f8371c00 
[2021-12-16 10:26:39] [   79.317340] x19: ffff800014dd1e40 x18: 0000000000000358 
[2021-12-16 10:26:39] [   79.322649] x17: 0000000000000000 x16: 0000000000000000 
[2021-12-16 10:26:39] [   79.327959] x15: 0000000000000000 x14: 0000000040000010 
[2021-12-16 10:26:39] [   79.333268] x13: 0000000000000000 x12: 0000000000004722 
[2021-12-16 10:26:39] [   79.338578] x11: 00000000000001d7 x10: 00000000011ce9bc 
[2021-12-16 10:26:39] [   79.343888] x9 : 0000000000004722 x8 : 0000000000000000 
[2021-12-16 10:26:39] [   79.349197] x7 : 0000000000000000 x6 : ffff0000f8371cf0 
[2021-12-16 10:26:39] [   79.354507] x5 : 00000000ffffffff x4 : 0000000000000000 
[2021-12-16 10:26:39] [   79.359816] x3 : ffff800010114bd8 x2 : ffff800010119540 
[2021-12-16 10:26:39] [   79.365126] x1 : 0000000000000000 x0 : ffff800014dd1e40 
[2021-12-16 10:26:39] [   79.370436] Call trace:
[2021-12-16 10:26:39] [   79.372882]  pick_next_task_idle+0x30/0x48
[2021-12-16 10:26:39] [   79.376978]  __schedule+0x3d4/0x560
[2021-12-16 10:26:39] [   79.380464]  schedule+0x40/0xe0
[2021-12-16 10:26:39] [   79.383607]  kthread_worker_fn+0xf8/0x188
[2021-12-16 10:26:39] [   79.387614]  kthread+0xf0/0x120
[2021-12-16 10:26:39] [   79.390756]  ret_from_fork+0x10/0x18
[2021-12-16 10:26:39] [   79.394334] Code: f9404022 f9401c42 d63f0040 aa1303e0 (f9400bf3) 
[2021-12-16 10:26:39] [   79.400426] ---[ end trace 5d29fc571c53c860 ]---
[2021-12-16 10:26:39] [   79.405042] note: 30902000.jr-eng[309] exited with preempt_count 2

What can be the reason for the same? We have followed the hardware guidelines mentioned in the document and performed the DDR calibration. 

Regards.

[jimmychan]

How many boards have this issue? Is the crash randomly? or crash at the same place during boot? Is the DDR stress test passed?

[harshitshah]

Hi @jimmychan ,

There are 3 boards that have this issue out of 15. This crash is happening at the same boot-up, and we are not getting the fs prompt.

Yes, DDR stress test in the overnight case is passing. 

Regards.

[jimmychan]

Could you provide the full boot up log of the good board and the failed board?