debug random kernel panic

[nc87]

Hello,

I am trying to debug random kernel crashes and need some help on how to go about debugging these crashes. Any suggestions would be appreciated.

Board: Digi CCIMX6 Connect Core IMX6Q Processor.

Linux Image : Yocto 3.0-r4 running kernel 5.4.84

Here is one of the dump from serial port (also, PC and stack can be pointing to different functions in every dump, so there is no consistency)

Digi Embedded Yocto 3.0-r4 ccimx6sbc ttymxc3

<--- cut here ---
Unhandled fault: page domain fault (0x01b) at 0x00028010
pgd = e30486f7
[00028010] *pgd=00000000
Internal error: : 1b [#1] PREEMPT SMP ARM
Modules linked in: ath6kl_sdio ath6kl_core cryptodev(O)
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 5.4.84-dey+ #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
PC is at try_to_wake_up+0x70/0x3c0
LR is at _raw_spin_lock_irqsave+0x18/0x5c
pc : [<8014ec14>] lr : [<80a04668>] psr: 200d0193
sp : 80f01d98 ip : 00000000 fp : 80f01dcc
r10: 8018d0a8 r9 : 00000000 r8 : 800d0113
r7 : a5af049c r6 : 00000000 r5 : 00000001 r4 : a5af0000
r3 : 00028000 r2 : 0000faeb r1 : 00000000 r0 : 800d0113
Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
Control: 10c5387d Table: 359c804a DAC: 00000051
Process swapper/0 (pid: 0, stack limit = 0x35c03995)
Stack: (0x80f01d98 to 0x80f02000)
1d80: 80f01ee8 ab5fa958
1da0: 000005f5 80f06688 00000000 ffffe000 8018d0a8 00000101 a5af9d64 00000000
1dc0: 00000000 8018d0a8 80f03d00 8018d1b8 ab5fa688 ab5f9500 80f067fc 80f03084
1de0: ab5f9500 ab5f9518 80f01dfc 8018d870 000005f5 80f067fc ab5fa6d8 a5b07d64
1e00: 00000000 7fffffff ffffffff ab5fa600 ab5fa600 00000003 ffffffff 80f06688
1e20: 200d0193 000000a0 00000001 80f03084 80f03080 80f00000 00000101 80f01e48
1e40: 40000001 801022e4 ab5fa730 ab5fa780 38ea48f4 80f03080 80e5f350 80e65d00
1e60: 80c94958 0000000a 80e5f2dc 000989b8 80f03d00 00200002 00000000 80e65d00
1e80: 00000000 80e65cf4 00000001 80f01ee8 00000000 ab00c000 00000000 8012c580
1ea0: 80e65cf4 8017416c 80f06ae4 80f1fffc f4a0010c f4a00100 f4a01100 80f01ee8
1ec0: 000005f5 8040fae8 806ce800 200d0013 ffffffff 80f01f1c 2bfd9f75 80f00000
1ee0: 000005f5 80101a8c 00000000 000005f5 00000001 2a796000 80f0d6fc 00000000
1f00: ab5fb578 2c082578 2bfd9f75 000005f5 000005f5 00000000 fffffff5 80f01f38
1f20: 80153604 806ce800 200d0013 ffffffff 00000051 00000000 80f0b880 00000000
1f40: 80f067fc ab5fb578 80f0d6fc 00000000 80f066e8 80e65570 80f0d6fc 80c94958
1f60: ab5fb578 806ceb5c 00000000 80f00000 80f066a8 801538ac 80f01f78 00000000
1f80: 01000001 80f06688 abfff840 000000ce 80f06680 00000000 80f8c000 00000001
1fa0: 80e45a40 abfff840 00000000 80153c54 80f8c054 80e00db0 ffffffff ffffffff
1fc0: 00000000 80e00584 00000000 80e45a40 f973b43a 80f06688 80e00330 00000051
1fe0: 10c0387d 00001323 18000000 412fc09a 10c5387d 00000000 00000000 00000000
[<8014ec14>] (try_to_wake_up) from [<8018d1b8>] (call_timer_fn.constprop.0+0x24/0x98)
[<8018d1b8>] (call_timer_fn.constprop.0) from [<8018d870>] (run_timer_softirq+0x400/0x418)
[<8018d870>] (run_timer_softirq) from [<801022e4>] (__do_softirq+0x13c/0x2c0)
[<801022e4>] (__do_softirq) from [<8012c580>] (irq_exit+0xc8/0xe0)
[<8012c580>] (irq_exit) from [<8017416c>] (__handle_domain_irq+0x80/0xe8)
[<8017416c>] (__handle_domain_irq) from [<8040fae8>] (gic_handle_irq+0x4c/0x90)
[<8040fae8>] (gic_handle_irq) from [<80101a8c>] (__irq_svc+0x6c/0xa8)
Exception stack(0x80f01ee8 to 0x80f01f30)
1ee0: 00000000 000005f5 00000001 2a796000 80f0d6fc 00000000
1f00: ab5fb578 2c082578 2bfd9f75 000005f5 000005f5 00000000 fffffff5 80f01f38
1f20: 80153604 806ce800 200d0013 ffffffff
[<80101a8c>] (__irq_svc) from [<806ce800>] (cpuidle_enter_state+0x80/0x3a0)
[<806ce800>] (cpuidle_enter_state) from [<806ceb5c>] (cpuidle_enter+0x28/0x38)
[<806ceb5c>] (cpuidle_enter) from [<801538ac>] (do_idle+0x1f8/0x2a4)
[<801538ac>] (do_idle) from [<80153c54>] (cpu_startup_entry+0x18/0x1c)
[<80153c54>] (cpu_startup_entry) from [<80e00db0>] (start_kernel+0x448/0x47c)
Code: e0135005 e1a08000 0a000066 e5943004 (e5933010)
---[ end trace 2bb892aa9465511a ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 889 Comm: kworker/1:2 Tainted: G D O 5.4.84-dey+ #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
[<801110f8>] (unwind_backtrace) from [<8010c404>] (show_stack+0x10/0x14)
[<8010c404>] (show_stack) from [<809e7ae0>] (dump_stack+0x84/0x98)
[<809e7ae0>] (dump_stack) from [<8010fb4c>] (handle_IPI+0x18c/0x1b8)
[<8010fb4c>] (handle_IPI) from [<8040fb28>] (gic_handle_irq+0x8c/0x90)
[<8040fb28>] (gic_handle_irq) from [<80101a8c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xa60ddce0 to 0xa60ddd28)
dce0: 00000000 00000004 ab600010 00000003 ab60ba80 80f0677c 80f067fc 00000004
dd00: 00000001 a60ddde8 ab60ba84 ab60ba88 00000000 a60ddd30 801a6740 801a676c
dd20: 000d0013 ffffffff
[<80101a8c>] (__irq_svc) from [<801a676c>] (smp_call_function_many+0x2d4/0x364)
[<801a676c>] (smp_call_function_many) from [<801a6830>] (smp_call_function+0x34/0x68)
[<801a6830>] (smp_call_function) from [<801a688c>] (on_each_cpu+0x28/0x70)
[<801a688c>] (on_each_cpu) from [<801107c8>] (twd_rate_change+0x28/0x30)
[<801107c8>] (twd_rate_change) from [<80147830>] (notifier_call_chain+0x48/0x84)
[<80147830>] (notifier_call_chain) from [<80147abc>] (__srcu_notifier_call_chain+0x44/0x9c)
[<80147abc>] (__srcu_notifier_call_chain) from [<80147b2c>] (srcu_notifier_call_chain+0x18/0x20)
[<80147b2c>] (srcu_notifier_call_chain) from [<80489890>] (__clk_notify+0x80/0xb4)
[<80489890>] (__clk_notify) from [<8048bb7c>] (__clk_recalc_rates+0x80/0x84)
[<8048bb7c>] (__clk_recalc_rates) from [<8048bb50>] (__clk_recalc_rates+0x54/0x84)
CPU2: stopping
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D O 5.4.84-dey+ #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[<801110f8>] (unwind_backtrace) from [<8010c404>] (show_stack+0x10/0x14)
[<8010c404>] (show_stack) from [<809e7ae0>] (dump_stack+0x84/0x98)
[<809e7ae0>] (dump_stack) from [<8010fb4c>] (handle_IPI+0x18c/0x1b8)
[<8010fb4c>] (handle_IPI) from [<8040fb28>] (gic_handle_irq+0x8c/0x90)
[<8040fb28>] (gic_handle_irq) from [<80101a8c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xab099f28 to 0xab099f70)
9f20: 00000002 000005f5 00000001 2a7b4000 80f0d6fc 00000000
9f40: ab619578 3fdfe623 3fda56f6 000005f5 000005f5 00000000 fffffff5 ab099f78
9f60: 80153604 806ce800 20070013 ffffffff
[<80101a8c>] (__irq_svc) from [<806ce800>] (cpuidle_enter_state+0x80/0x3a0)
[<806ce800>] (cpuidle_enter_state) from [<806ceb5c>] (cpuidle_enter+0x28/0x38)
[<806ceb5c>] (cpuidle_enter) from [<801538ac>] (do_idle+0x1f8/0x2a4)
[<801538ac>] (do_idle) from [<80153c54>] (cpu_startup_entry+0x18/0x1c)
[<80153c54>] (cpu_startup_entry) from [<1010250c>] (0x1010250c)
CPU3: stopping
CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D O 5.4.84-dey+ #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[<801110f8>] (unwind_backtrace) from [<8010c404>] (show_stack+0x10/0x14)
[<8010c404>] (show_stack) from [<809e7ae0>] (dump_stack+0x84/0x98)
[<809e7ae0>] (dump_stack) from [<8010fb4c>] (handle_IPI+0x18c/0x1b8)
[<8010fb4c>] (handle_IPI) from [<8040fb28>] (gic_handle_irq+0x8c/0x90)
[<8040fb28>] (gic_handle_irq) from [<80101a8c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xab09bf28 to 0xab09bf70)
bf20: 00000003 000005f5 00000001 2a7c3000 80f0d6fc 00000000
bf40: ab628578 3fdfe623 3fdd984b 000005f5 000005f5 00000000 fffffff5 ab09bf78
bf60: 80153604 806ce800 20070013 ffffffff
[<80101a8c>] (__irq_svc) from [<806ce800>] (cpuidle_enter_state+0x80/0x3a0)
[<806ce800>] (cpuidle_enter_state) from [<806ceb5c>] (cpuidle_enter+0x28/0x38)
[<806ceb5c>] (cpuidle_enter) from [<801538ac>] (do_idle+0x1f8/0x2a4)
[<801538ac>] (do_idle) from [<80153c54>] (cpu_startup_entry+0x18/0x1c)
[<80153c54>] (cpu_startup_entry) from [<1010250c>] (0x1010250c)
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Thank You

[Bio_TICFSL]

Hello ric87,

The error mean that you need build correct the kernel since is in panic it can't be debugged.

Regards