How to analyze and debug a Linux kernel panic which occurs while running an application? How to pinpoint the reason for memory corruption ?

[peteramond]

Hi All,

I'm getting kernel panics while running linux application on my custom hardware design(iMAX6Q processor based). But another custom hardware which is similar to this can run this linux application without any problem. I did the memory calibration and stress app test for this board. But there was not any issue related to memory. So how can I debug this to clarify whether this is memory related hardware or software problem ? Or how to address which memory is corrupting ? 

Presently I'm using >>addr2line -e vmlinux_with_debug_info 80a00000<< to decode kernel panic message. But it only shows the kernel source when the panic occur(Where program counter stopped) . As an example from the following kernel log when I decode [80a00000] it only give the kernel source. So do you know how to identify the which memory is going to corrupt ?

pc : [<80a00000>] lr : [<809449f8>] psr: 000f0193

Regards,

Peter.

Here is the panic message (LOG)

[ 212.954190] Unable to handle kernel paging request at virtual address 80a00000
[ 212.961455] pgd = cf1d4000
[ 212.964174] [80a00000] *pgd=10a1941e(bad)
[ 212.968228] Internal error: Oops: 8000000d [#1] PREEMPT SMP ARM
[ 212.974157] Modules linked in: mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc ipu_fg_overlay_sdc tw6869 adv7610_video v4l2_int_device videobuf2_dma_contig videobuf2_memops galcore(O)
[ 212.992619] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 4.1.15-2.0.0-ga+yocto+gff4e28b #1
[ 213.001842] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 213.008374] task: ce11bc00 ti: ce14e000 task.ti: ce14e000
[ 213.013789] PC is at __param_str_initcall_debug+0x0/0x10
[ 213.019110] LR is at _raw_spin_unlock_irqrestore+0x38/0x64
[ 213.024602] pc : [<80a00000>] lr : [<809449f8>] psr: 000f0193
[ 213.024602] sp : ce14fd28 ip : ce14fd28 fp : ce14fd34
[ 213.036082] r10: 00000001 r9 : 50191000 r8 : 600f0193
[ 213.041318] r7 : 80e02e00 r6 : ce444a3c r5 : d0f2c880 r4 : ce444600
[ 213.047862] r3 : ce14e000 r2 : 00010001 r1 : 600f0193 r0 : 00000001
[ 213.054409] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 213.061823] Control: 10c5387d Table: 5f1d404a DAC: 00000015
[ 213.067586] Process swapper/3 (pid: 0, stack limit = 0xce14e210)
[ 213.073609] Stack: (0xce14fd28 to 0xce150000)
[ 213.077992] fd20: ce14fd6c ce14fd38 8015e448 809449cc ce14fd64 ce14fd48
[ 213.086194] fd40: 804a7118 ce444600 00000031 d0f293f8 d0f293c0 ce14fdf8 8018f844 d0f294d8
[ 213.094395] fd60: ce14fd84 ce14fd70 8015e54c 8015e244 cf229f18 00000031 ce14fd94 ce14fd88
[ 213.102596] fd80: 8018f874 8015e524 ce14fdbc ce14fd98 80190284 8018f850 8f02f5ac 00000031
[ 213.110797] fda0: d0f293f8 00000000 d0f293c0 00000001 ce14fe34 ce14fdc0 80190894 80190210
[ 213.118998] fdc0: 8f02f5ac 00000031 80e02508 d0f294d0 d0f29498 d0f29460 00000003 d0f293f8
[ 213.127198] fde0: ffffffff 7fffffff 8f02f5ac 00000031 8f02f5ac 00000031 8f02f5ac 00000031
[ 213.135399] fe00: 80196800 dc8ba30f ce14fe44 00000001 d0f2f4c0 ce0353c0 00000010 00000001
[ 213.143599] fe20: ce006d80 80e47000 ce14fe4c ce14fe38 80110ce4 80190768 00000010 d0f2f4c0
[ 213.151801] fe40: ce14fe7c ce14fe50 8018051c 80110cb0 80180470 00000010 00000000 00000010
[ 213.160001] fe60: 00000000 00000001 ce020000 ce14fee8 ce14fe94 ce14fe80 8017bb84 8018047c
[ 213.168202] fe80: 80d99c34 00000000 ce14fec4 ce14fe98 8017beb0 8017bb54 ce14fee8 f4a0010c
[ 213.176402] fea0: 80e02f7c ce14fee8 f4a00100 d0f2bed0 80e08ac4 80e02508 ce14fee4 ce14fec8
[ 213.184603] fec0: 80101560 8017be30 806fb0b0 600f0013 ffffffff ce14ff1c ce14ff8c ce14fee8
[ 213.192804] fee0: 8010d240 80101538 8f02c57f 00000031 8f02c57f 00000000 8f02c197 00000031
[ 213.201005] ff00: 80e89160 00000000 d0f2bed0 80e08ac4 80e02508 ce14ff8c 00000017 ce14ff30
[ 213.209205] ff20: a6aaaaab 806fb0b0 600f0013 ffffffff 8f02c57f 00000031 8f02c197 00000031
[ 213.217406] ff40: 00000000 00000000 000fbf40 00000000 8f02c57f 00000031 000003ff dc8ba30f
[ 213.225607] ff60: 80d97320 ce14e000 80e025e8 80e89160 d0f2bed0 80e08a74 80a02970 ce14ffa0
[ 213.233808] ff80: ce14ff9c ce14ff90 806fb390 806faffc ce14ffdc ce14ffa0 80171a74 806fb378
[ 213.242009] ffa0: 80d9b880 80d9b880 00000000 80e02e00 80e025f0 80e888ea 00000001 80c137fc
[ 213.250210] ffc0: 80d9aec8 80d97300 10c0387d 80e8e35c ce14fff4 ce14ffe0 8010fae0 801717d8
[ 213.258410] ffe0: 5e13006a 00000015 00000000 ce14fff8 1010162c 8010f994 00000000 00000000
[ 213.266599] Backtrace:
[ 213.269107] [<809449c0>] (_raw_spin_unlock_irqrestore) from [<8015e448>] (try_to_wake_up+0x210/0x2e0)
[ 213.278356] [<8015e238>] (try_to_wake_up) from [<8015e54c>] (wake_up_process+0x34/0x4c)
[ 213.286374] r10:d0f294d8 r9:8018f844 r8:ce14fdf8 r7:d0f293c0 r6:d0f293f8 r5:00000031
[ 213.294333] r4:ce444600
[ 213.296926] [<8015e518>] (wake_up_process) from [<8018f874>] (hrtimer_wakeup+0x30/0x38)
[ 213.304943] r5:00000031 r4:cf229f18
[ 213.308593] [<8018f844>] (hrtimer_wakeup) from [<80190284>] (__run_hrtimer+0x80/0x284)
[ 213.316542] [<80190204>] (__run_hrtimer) from [<80190894>] (hrtimer_interrupt+0x138/0x344)
[ 213.324820] r9:00000001 r8:d0f293c0 r7:00000000 r6:d0f293f8 r5:00000031 r4:8f02f5ac
[ 213.332712] [<8019075c>] (hrtimer_interrupt) from [<80110ce4>] (twd_handler+0x40/0x50)
[ 213.340643] r10:80e47000 r9:ce006d80 r8:00000001 r7:00000010 r6:ce0353c0 r5:d0f2f4c0
[ 213.348601] r4:00000001
[ 213.351187] [<80110ca4>] (twd_handler) from [<8018051c>] (handle_percpu_devid_irq+0xac/0x1d0)
[ 213.359726] r5:d0f2f4c0 r4:00000010
[ 213.363384] [<80180470>] (handle_percpu_devid_irq) from [<8017bb84>] (generic_handle_irq+0x3c/0x4c)
[ 213.372444] r10:ce14fee8 r9:ce020000 r8:00000001 r7:00000000 r6:00000010 r5:00000000
[ 213.380400] r4:00000010 r3:80180470
[ 213.384052] [<8017bb48>] (generic_handle_irq) from [<8017beb0>] (__handle_domain_irq+0x8c/0xfc)
[ 213.392764] r5:00000000 r4:80d99c34
[ 213.396412] [<8017be24>] (__handle_domain_irq) from [<80101560>] (gic_handle_irq+0x34/0x6c)
[ 213.404777] r10:80e02508 r9:80e08ac4 r8:d0f2bed0 r7:f4a00100 r6:ce14fee8 r5:80e02f7c
[ 213.412736] r4:f4a0010c r3:ce14fee8
[ 213.416379] [<8010152c>] (gic_handle_irq) from [<8010d240>] (__irq_svc+0x40/0x74)
[ 213.423880] Exception stack(0xce14fee8 to 0xce14ff30)
[ 213.428955] fee0: 8f02c57f 00000031 8f02c57f 00000000 8f02c197 00000031
[ 213.437157] ff00: 80e89160 00000000 d0f2bed0 80e08ac4 80e02508 ce14ff8c 00000017 ce14ff30
[ 213.445352] ff20: a6aaaaab 806fb0b0 600f0013 ffffffff
[ 213.450417] r7:ce14ff1c r6:ffffffff r5:600f0013 r4:806fb0b0
[ 213.456208] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] (cpuidle_enter+0x24/0x28)
[ 213.464487] r10:ce14ffa0 r9:80a02970 r8:80e08a74 r7:d0f2bed0 r6:80e89160 r5:80e025e8
[ 213.472444] r4:ce14e000
[ 213.475028] [<806fb36c>] (cpuidle_enter) from [<80171a74>] (cpu_startup_entry+0x2a8/0x43c)
[ 213.483323] [<801717cc>] (cpu_startup_entry) from [<8010fae0>] (secondary_start_kernel+0x158/0x164)
[ 213.492382] r7:80e8e35c
[ 213.494966] [<8010f988>] (secondary_start_kernel) from [<1010162c>] (0x1010162c)
[ 213.502376] r5:00000015 r4:5e13006a
[ 213.506017] Code: 00000000 00000000 00000000 00000000 (74696e69)
[ 213.512134] ---[ end trace 4fd1b31399dcd550 ]---
[ 213.516772] Kernel panic - not syncing: Fatal exception in interrupt
[ 213.523150] CPU1: stopping
[ 213.525897] CPU: 1 PID: 37 Comm: cfinteractive Tainted: G D O 4.1.15-2.0.0-ga+yocto+gff4e28b #1
[ 213.535569] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 213.542111] Backtrace:
[ 213.544626] [<8010c358>] (dump_backtrace) from [<8010c5d4>] (show_stack+0x20/0x24)
[ 213.552213] r7:80e88b48 r6:80e45d94 r5:00000000 r4:80e45d94
[ 213.557996] [<8010c5b4>] (show_stack) from [<8093ee58>] (dump_stack+0x7c/0xbc)
[ 213.565249] [<8093eddc>] (dump_stack) from [<80110038>] (handle_IPI+0x2e4/0x2f8)
[ 213.572660] r7:80e88b48 r6:00000005 r5:80d99c34 r4:80d99c34
[ 213.578440] [<8010fd54>] (handle_IPI) from [<80101594>] (gic_handle_irq+0x68/0x6c)
[ 213.586025] r10:00000004 r9:ce323d90 r8:00000001 r7:f4a00100 r6:ce323c40 r5:80e02f7c
[ 213.593989] r4:f4a0010c r3:ce323c40
[ 213.597635] [<8010152c>] (gic_handle_irq) from [<8010d240>] (__irq_svc+0x40/0x74)
[ 213.605135] Exception stack(0xce323c40 to 0xce323c88)
[ 213.610213] 3c40: 00000003 d0f2f89c 00000003 00000003 80e02650 d0f13180 80e02e00 d0f13184
[ 213.618416] 3c60: 00000001 ce323d90 00000004 ce323cbc 00000003 ce323c88 801a738c 801a73bc
[ 213.626611] 3c80: 200f0013 ffffffff
[ 213.630114] r7:ce323c74 r6:ffffffff r5:200f0013 r4:801a73bc
[ 213.635897] [<801a714c>] (smp_call_function_many) from [<801a7458>] (smp_call_function+0x48/0x88)
[ 213.644784] r10:ffffffff r9:ce323d88 r8:00000000 r7:00000000 r6:00000001 r5:ce323d90
[ 213.652749] r4:80110990
[ 213.655331] [<801a7410>] (smp_call_function) from [<801a74d0>] (on_each_cpu+0x38/0x90)
[ 213.663262] r7:00000000 r6:00000001 r5:ce323d90 r4:80110990
[ 213.669041] [<801a7498>] (on_each_cpu) from [<80110d2c>] (twd_rate_change+0x38/0x40)
[ 213.676799] r7:00000000 r6:00000002 r5:ce323d88 r4:ffffffff
[ 213.682590] [<80110cf4>] (twd_rate_change) from [<801523c4>] (notifier_call_chain+0x54/0x94)
[ 213.691061] [<80152370>] (notifier_call_chain) from [<80152894>] (__srcu_notifier_call_chain+0x54/0x70)
[ 213.700468] r9:ce323d88 r8:00000002 r7:00000000 r6:00000000 r5:ce0a5184 r4:ce0a519c
[ 213.708356] [<80152840>] (__srcu_notifier_call_chain) from [<801528d8>] (srcu_notifier_call_chain+0x28/0x30)
[ 213.718198] r10:00000001 r9:80e8b384 r8:00000002 r7:80e02508 r6:ce026000 r5:80e74b44
[ 213.726161] r4:ce0a5180
[ 213.728745] [<801528b0>] (srcu_notifier_call_chain) from [<8076b7ec>] (__clk_notify+0xa0/0xa8)
[ 213.737383] [<8076b74c>] (__clk_notify) from [<8076b89c>] (__clk_recalc_rates+0xa8/0xac)
[ 213.745488] r8:ce003f80 r7:00000001 r6:1daee080 r5:00000002 r4:ce026000
[ 213.752328] [<8076b7f4>] (__clk_recalc_rates) from [<8076b870>] (__clk_recalc_rates+0x7c/0xac)
[ 213.760954] r7:00000001 r6:3b5dc100 r5:00000002 r4:ce026000
[ 213.766732] [<8076b7f4>] (__clk_recalc_rates) from [<8076b870>] (__clk_recalc_rates+0x7c/0xac)
[ 213.775358] r7:00000001 r6:3b5dc100 r5:00000002 r4:ce026a80
[ 213.781142] [<8076b7f4>] (__clk_recalc_rates) from [<8076e228>] (clk_core_set_parent+0x1bc/0x2dc)
[ 213.790028] r7:00000001 r6:00000000 r5:ce026380 r4:ce026400
[ 213.795806] [<8076e06c>] (clk_core_set_parent) from [<8076e374>] (clk_set_parent+0x2c/0x30)
[ 213.804171] r9:00000000 r8:2f34f600 r7:80e02508 r6:000f32a0 r5:000c15c0 r4:80eed96c
[ 213.812067] [<8076e348>] (clk_set_parent) from [<806f98f0>] (imx6q_set_target+0x490/0x544)
[ 213.820371] [<806f9460>] (imx6q_set_target) from [<806ef704>] (__cpufreq_driver_target+0x184/0x2b0)
[ 213.829431] r10:00000000 r9:00000000 r8:80eed8a4 r7:00000000 r6:80e02508 r5:ce6b8700
[ 213.837393] r4:00000001
[ 213.839981] [<806ef580>] (__cpufreq_driver_target) from [<806f8414>] (cpufreq_interactive_speedchange_task+0x264/0x354)
[ 213.850777] r10:80d9ae10 r9:80e02650 r8:80e02e00 r7:00000000 r6:000f32a0 r5:80d9ae10
[ 213.858740] r4:d0f1ee10
[ 213.861328] [<806f81b0>] (cpufreq_interactive_speedchange_task) from [<801514e4>] (kthread+0xfc/0x114)
[ 213.870649] r10:00000000 r9:00000000 r8:00000000 r7:806f81b0 r6:00000000 r5:ce304480
[ 213.878613] r4:00000000
[ 213.881200] [<801513e8>] (kthread) from [<80108028>] (ret_from_fork+0x14/0x2c)
[ 213.888436] r7:00000000 r6:00000000 r5:801513e8 r4:ce304480
[ 213.894206] CPU2: stopping
[ 213.896948] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D O 4.1.15-2.0.0-ga+yocto+gff4e28b #1
[ 213.906186] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 213.912728] Backtrace:
[ 213.915236] [<8010c358>] (dump_backtrace) from [<8010c5d4>] (show_stack+0x20/0x24)
[ 213.922823] r7:80e88b48 r6:80e45d94 r5:00000000 r4:80e45d94
[ 213.928604] [<8010c5b4>] (show_stack) from [<8093ee58>] (dump_stack+0x7c/0xbc)
[ 213.935857] [<8093eddc>] (dump_stack) from [<80110038>] (handle_IPI+0x2e4/0x2f8)
[ 213.943267] r7:80e88b48 r6:00000005 r5:80d99c34 r4:80d99c34
[ 213.949049] [<8010fd54>] (handle_IPI) from [<80101594>] (gic_handle_irq+0x68/0x6c)
[ 213.956633] r10:80e02508 r9:80e08b14 r8:d0f1eed0 r7:f4a00100 r6:ce14dee8 r5:80e02f7c
[ 213.964596] r4:f4a0010c r3:ce14dee8
[ 213.968243] [<8010152c>] (gic_handle_irq) from [<8010d240>] (__irq_svc+0x40/0x74)
[ 213.975744] Exception stack(0xce14dee8 to 0xce14df30)
[ 213.980821] dee0: 00000000 d0f224c0 dc8ba30f dc8ba30f b0d4adf9 00000031
[ 213.989024] df00: 80e89160 00000004 d0f1eed0 80e08b14 80e02508 ce14df8c ce14ded0 ce14df30
[ 213.997220] df20: 80944994 806fb0b0 600b0013 ffffffff
[ 214.002286] r7:ce14df1c r6:ffffffff r5:600b0013 r4:806fb0b0
[ 214.008074] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] (cpuidle_enter+0x24/0x28)
[ 214.016353] r10:ce14dfa0 r9:80a02970 r8:80e08a74 r7:d0f1eed0 r6:80e89160 r5:80e025e8
[ 214.024317] r4:ce14c000
[ 214.026904] [<806fb36c>] (cpuidle_enter) from [<80171a74>] (cpu_startup_entry+0x2a8/0x43c)
[ 214.035199] [<801717cc>] (cpu_startup_entry) from [<8010fae0>] (secondary_start_kernel+0x158/0x164)
[ 214.044259] r7:80e8e35c
[ 214.046843] [<8010f988>] (secondary_start_kernel) from [<1010162c>] (0x1010162c)
[ 214.054253] r5:00000015 r4:5e13006a
[ 214.057892] CPU0: stopping
[ 214.060634] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D O 4.1.15-2.0.0-ga+yocto+gff4e28b #1
[ 214.069872] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 214.076414] Backtrace:
[ 214.078922] [<8010c358>] (dump_backtrace) from [<8010c5d4>] (show_stack+0x20/0x24)
[ 214.086510] r7:80e88b48 r6:80e45d94 r5:00000000 r4:80e45d94
[ 214.092290] [<8010c5b4>] (show_stack) from [<8093ee58>] (dump_stack+0x7c/0xbc)
[ 214.099543] [<8093eddc>] (dump_stack) from [<80110038>] (handle_IPI+0x2e4/0x2f8)
[ 214.106953] r7:80e88b48 r6:00000005 r5:80d99c34 r4:80d99c34
[ 214.112732] [<8010fd54>] (handle_IPI) from [<80101594>] (gic_handle_irq+0x68/0x6c)
[ 214.120317] r10:80e02508 r9:80e08b14 r8:d0f04ed0 r7:f4a00100 r6:80e01e98 r5:80e02f7c
[ 214.128282] r4:f4a0010c r3:80e01e98
[ 214.131930] [<8010152c>] (gic_handle_irq) from [<8010d240>] (__irq_svc+0x40/0x74)
[ 214.139430] Exception stack(0x80e01e98 to 0x80e01ee0)
[ 214.144502] 1e80: 00000000 d0f084c0
[ 214.152704] 1ea0: dc8ba30f dc8ba30f b0d3b15f 00000031 80e89160 00000004 d0f04ed0 80e08b14
[ 214.160907] 1ec0: 80e02508 80e01f3c 80e01e80 80e01ee0 80944994 806fb0b0 60010013 ffffffff
[ 214.169098] r7:80e01ecc r6:ffffffff r5:60010013 r4:806fb0b0
[ 214.174885] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] (cpuidle_enter+0x24/0x28)
[ 214.183164] r10:80e01f50 r9:80a02970 r8:80e08a74 r7:d0f04ed0 r6:80e89160 r5:80e025e8
[ 214.191126] r4:80e00000
[ 214.193711] [<806fb36c>] (cpuidle_enter) from [<80171a74>] (cpu_startup_entry+0x2a8/0x43c)
[ 214.202016] [<801717cc>] (cpu_startup_entry) from [<8093cb20>] (rest_init+0x98/0x9c)
[ 214.209774] r7:80e02500
[ 214.212376] [<8093ca88>] (rest_init) from [<80d00d9c>] (start_kernel+0x404/0x424)
[ 214.219873] r5:80e8e000 r4:80e8e04c
[ 214.223524] [<80d00998>] (start_kernel) from [<1000807c>] (0x1000807c)
[ 214.230075] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

[igorpadykov]

thread is duplicated and answered on

https://community.nxp.com/message/1021142?commentID=1021142#comment-1021142