i.MX8MQ board is crashing after 12 to 24 hours randomly

[harshitshah]

Hi Team,

We are using the i.MX8MQ based board, during long-run testing we have found that it is giving below crash.

We are using i.MX8MQ processor custom board on L5.4.3_1.0.0 version

[2021-12-19 07:08:54] [125171.644284] SError Interrupt on CPU1, code 0xbf000001 -- SError
[2021-12-19 07:08:54] [125171.644288] CPU: 1 PID: 2577621 Comm: sh Tainted: G         C O      5.4.3-lts-lf-5.4.y+gf811858 #1
[2021-12-19 07:08:54] [125171.644290] Hardware name: i.MX8MQ Processor (DT)
[2021-12-19 07:08:54] [125171.644292] pstate: 80000000 (Nzcv daif -PAN -UAO)
[2021-12-19 07:08:54] [125171.644293] pc : 0000ffff8d3a4ac8
[2021-12-19 07:08:54] [125171.644295] lr : 0000ffff8d3a4088
[2021-12-19 07:08:54] [125171.644297] sp : 0000ffffe4ba2110
[2021-12-19 07:08:54] [125171.644298] x29: 0000ffffe4ba2110 x28: 0000000000000000 
[2021-12-19 07:08:54] [125171.644302] x27: 0000000000000000 x26: 0000000000000000 
[2021-12-19 07:08:54] [125171.644306] x25: 0000000000000000 x24: 0000000000000000 
[2021-12-19 07:08:54] [125171.644309] x23: 0000000000000000 x22: 0000000000000000 
[2021-12-19 07:08:54] [125171.644312] x21: 0000ffffe4ba21a8 x20: 0000000000000000 
[2021-12-19 07:08:54] [125171.644315] x19: 0000000000000000 x18: 0000000000000000 
[2021-12-19 07:08:54] [125171.644320] x17: 0000000000000000 x16: 0000000000000000 
[2021-12-19 07:08:54] [125171.644323] x15: 0000000000000000 x14: 0000000000000000 
[2021-12-19 07:08:54] [125171.644326] x13: 0000000000000000 x12: 0000000000000000 
[2021-12-19 07:08:54] [125171.644328] x11: 0000000000000000 x10: 0000000000000000 
[2021-12-19 07:08:54] [125171.644332] x9 : 0000000000000000 x8 : 0000000000000000 
[2021-12-19 07:08:54] [125171.644335] x7 : 0000000000000000 x6 : 0000000000000000 
[2021-12-19 07:08:54] [125171.644338] x5 : 0000000000000000 x4 : 0000000000000000 
[2021-12-19 07:08:54] [125171.644344] x3 : 0000000000000000 x2 : 0000ffffe4ba2480 
[2021-12-19 07:08:54] [125171.644347] x1 : 0000ffffe4ba23a8 x0 : 0000ffffe4ba2650 
[2021-12-19 07:08:54] [125171.644350] Kernel panic - not syncing: Asynchronous SError Interrupt
[2021-12-19 07:08:54] [125171.644356] CPU: 1 PID: 2577621 Comm: sh Tainted: G         C O      5.4.3-lts-lf-5.4.y+gf811858 #1
[2021-12-19 07:08:54] [125171.644358] Hardware name: i.MX8MQ Processor (DT)
[2021-12-19 07:08:54] [125171.644359] Call trace:
[2021-12-19 07:08:54] [125171.644360]  dump_backtrace+0x0/0x140
[2021-12-19 07:08:54] [125171.644361]  show_stack+0x14/0x20
[2021-12-19 07:08:54] [125171.644363]  dump_stack+0xb4/0xf8
[2021-12-19 07:08:54] [125171.644364]  panic+0x158/0x324
[2021-12-19 07:08:54] [125171.644368]  nmi_panic+0x84/0x88
[2021-12-19 07:08:54] [125171.644369]  arm64_serror_panic+0x74/0x80
[2021-12-19 07:08:54] [125171.644370]  do_serror+0x80/0x138
[2021-12-19 07:08:54] [125171.644372]  el0_error_naked+0x14/0x1c
[2021-12-19 07:08:54] [125171.644408] SMP: stopping secondary CPUs
[2021-12-19 07:08:54] [125171.644409] Kernel Offset: disabled
[2021-12-19 07:08:54] [125171.644410] CPU features: 0x0002,2000200c
[2021-12-19 07:08:54] [125171.644412] Memory Limit: none
[2021-12-19 07:09:06]

 Query:

1) Is it a known issue of the firmware release 5.4.3_1.0.0?  (We might not switch to the current version now due to the product phase unless this is the known issue).

2) If not, then what can be the probable causes for this kind of crash? We have seen these crashes on multiple boards.

Please note that we have already done the DDR calibration and it was working well.

Regards.

[igorpadykov]

Hi Harshi

>1) Is it a known issue of the firmware release 5.4.3_1.0.0? .

no

>2) If not, then what can be the probable causes for this kind of crash?

reason may be power supplies (ripples), may be recommended to check

i.MX 8MDQLQ Hardware Developer’s Guide

or poor soldering (one can try to resolder chip).

Best regards
igor

[harshitshah]

Thank you Igor for the quick reply.

Regarding 2) are you saying to re-solder eMMC, DDR, or i.MX8MQ? 

Regards. 

[igorpadykov]

>are you saying to re-solder eMMC, DDR, or i.MX8MQ? 

i.MX8MQ

Best regards
igor