Reason for kernel panic ? Unable to handle kernel paging request at virtual address 7e263df8 / Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: 80193638

[peteramond]

Dear All,

This is regarding imx6q, 2GB Alliance Memory (4Gb x 4, 4 RAM chips) custom hardware design which is most similar to nitrogen6_max. I have manufactured 50 units of this design and I could get work only 10 boards out of  50 without any kernel panics.

I did memory calibration properly and did stress test about 4 hours and it passed without any conflict.

Other boards get kernel panics randomly. Here I have attached kernel log. Is this a memory problem ? When I do the program counter decoding using addr2line it shows >>Problem related to scheduling.

Any idea of this ?

Regards.

Kernel Log

Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: 80193638

CPU2: stopping
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 4.1.15-1.0.0_ga+yocto+g76c24a0 #8
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<8010b528>] (dump_backtrace) from [<8010b748>] (show_stack+0x18/0x1c)
r7:00000000 r6:80d1c2e0 r5:80d1c2e0 r4:00000000
[<8010b730>] (show_stack) from [<80862be8>] (dump_stack+0x8c/0xcc)
[<80862b5c>] (dump_stack) from [<8010e72c>] (handle_IPI+0x180/0x194)
r7:00000000 r6:00000005 r5:00000002 r4:80c60c04
[<8010e5ac>] (handle_IPI) from [<801014a4>] (gic_handle_irq+0x60/0x64)
r9:0000002e r8:f6b72f95 r7:f4a00100 r6:ce0a7f08 r5:80d02888 r4:f4a0010c
[<80101444>] (gic_handle_irq) from [<8010c340>] (__irq_svc+0x40/0x74)
Exception stack(0xce0a7f08 to 0xce0a7f50)
7f00: 00000000 d0f23440 dc8ba30f dc8ba30f 80d02508 d0f1fed0
7f20: f6ab55bb 0000002e f6b72f95 0000002e 00000001 ce0a7f94 ce0a7ef0 ce0a7f50
7f40: 808683a4 8063c8cc 60010013 ffffffff
r7:ce0a7f3c r6:ffffffff r5:60010013 r4:8063c8cc
[<8063c7dc>] (cpuidle_enter_state) from [<8063ca68>] (cpuidle_enter+0x1c/0x20)
r10:00000001 r9:ce0a7fa8 r8:80d07ad0 r7:d0f1fed0 r6:809028c0 r5:80d025e8
r4:ce0a6000
[<8063ca4c>] (cpuidle_enter) from [<80168280>] (cpu_startup_entry+0x208/0x330)
[<80168078>] (cpu_startup_entry) from [<8010e348>] (secondary_start_kernel+0x150/0x15c)
r7:80d51308
[<8010e1f8>] (secondary_start_kernel) from [<1010154c>] (0x1010154c)
r5:00000015 r4:5e08806a
CPU1: stopping
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 4.1.15-1.0.0_ga+yocto+g76c24a0 #8
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<8010b528>] (dump_backtrace) from [<8010b748>] (show_stack+0x18/0x1c)
r7:00000000 r6:80d1c2e0 r5:80d1c2e0 r4:00000000
[<8010b730>] (show_stack) from [<80862be8>] (dump_stack+0x8c/0xcc)
[<80862b5c>] (dump_stack) from [<8010e72c>] (handle_IPI+0x180/0x194)
r7:00000000 r6:00000005 r5:00000001 r4:80c60c04
[<8010e5ac>] (handle_IPI) from [<801014a4>] (gic_handle_irq+0x60/0x64)
r9:0000002e r8:f6b730e3 r7:f4a00100 r6:ce0a5f08 r5:80d02888 r4:f4a0010c
[<80101444>] (gic_handle_irq) from [<8010c340>] (__irq_svc+0x40/0x74)
Exception stack(0xce0a5f08 to 0xce0a5f50)
5f00: 00000000 d0f17440 dc8ba30f dc8ba30f 80d02508 d0f13ed0
5f20: f631f76b 0000002e f6b730e3 0000002e 00000001 ce0a5f94 ce0a5ef0 ce0a5f50
5f40: 808683a4 8063c8cc 60030013 ffffffff
r7:ce0a5f3c r6:ffffffff r5:60030013 r4:8063c8cc
[<8063c7dc>] (cpuidle_enter_state) from [<8063ca68>] (cpuidle_enter+0x1c/0x20)
r10:00000001 r9:ce0a5fa8 r8:80d07ad0 r7:d0f13ed0 r6:809028c0 r5:80d025e8
r4:ce0a4000
[<8063ca4c>] (cpuidle_enter) from [<80168280>] (cpu_startup_entry+0x208/0x330)
[<80168078>] (cpu_startup_entry) from [<8010e348>] (secondary_start_kernel+0x150/0x15c)
r7:80d51308
[<8010e1f8>] (secondary_start_kernel) from [<1010154c>] (0x1010154c)
r5:00000015 r4:5e08806a
CPU3: stopping
CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 4.1.15-1.0.0_ga+yocto+g76c24a0 #8
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<8010b528>] (dump_backtrace) from [<8010b748>] (show_stack+0x18/0x1c)
r7:00000000 r6:80d1c2e0 r5:80d1c2e0 r4:00000000
[<8010b730>] (show_stack) from [<80862be8>] (dump_stack+0x8c/0xcc)
[<80862b5c>] (dump_stack) from [<8010e72c>] (handle_IPI+0x180/0x194)
r7:00000000 r6:00000005 r5:00000003 r4:80c60c04
[<8010e5ac>] (handle_IPI) from [<801014a4>] (gic_handle_irq+0x60/0x64)
r9:0000002e r8:f6b730e3 r7:f4a00100 r6:ce0a9f08 r5:80d02888 r4:f4a0010c
[<80101444>] (gic_handle_irq) from [<8010c340>] (__irq_svc+0x40/0x74)
Exception stack(0xce0a9f08 to 0xce0a9f50)
9f00: 00000000 80d97c28 dc8ba30f dc8ba30f 80d02508 d0f2bed0
9f20: f6445fa8 0000002e f6b730e3 0000002e 00000001 ce0a9f94 ce0a9ef0 ce0a9f50
9f40: 808683a4 8063c8cc 600e0013 ffffffff
r7:ce0a9f3c r6:ffffffff r5:600e0013 r4:8063c8cc
[<8063c7dc>] (cpuidle_enter_state) from [<8063ca68>] (cpuidle_enter+0x1c/0x20)
r10:00000001 r9:ce0a9fa8 r8:80d07ad0 r7:d0f2bed0 r6:809028c0 r5:80d025e8
r4:ce0a8000
[<8063ca4c>] (cpuidle_enter) from [<80168280>] (cpu_startup_entry+0x208/0x330)
[<80168078>] (cpu_startup_entry) from [<8010e348>] (secondary_start_kernel+0x150/0x15c)
r7:80d51308
[<8010e1f8>] (secondary_start_kernel) from [<1010154c>] (0x1010154c)
r5:00000015 r4:5e08806a
---[ end Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: 80193638

------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at /home/tengri/Desktop/YOCTO_wayland/core-image-minimal/tmp/work-shared/nitrogen6x/kernel-source/kernel/time/tick-broadcast.c:621 tick_handle_oneshot_broadcast+0x1f4/0x218()
Modules linked in: brcmfmac brcmutil cfg80211 usb_f_ecm g_ether usb_f_rndis libcomposite u_ether tw6869 videobuf2_dma_contig videobuf2_memops mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc ipu_fg_overlay_sdc adv7610_video v4l2_int_device galcore(O)
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.1.15-1.0.0_ga+yocto+g76c24a0 #8
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<8010b528>] (dump_backtrace) from [<8010b748>] (show_stack+0x18/0x1c)
r7:8019164c r6:80d1c2e0 r5:80d1c2e0 r4:00000000
[<8010b730>] (show_stack) from [<80862be8>] (dump_stack+0x8c/0xcc)
[<80862b5c>] (dump_stack) from [<80130148>] (warn_slowpath_common+0x88/0xb8)
r7:8019164c r6:0000026d r5:00000009 r4:00000000
[<801300c0>] (warn_slowpath_common) from [<80130264>] (warn_slowpath_null+0x24/0x2c)
r8:ffffffff r7:80d02650 r6:00000004 r5:80c60648 r4:80d4e9ee
[<80130240>] (warn_slowpath_null) from [<8019164c>] (tick_handle_oneshot_broadcast+0x1f4/0x218)
[<80191458>] (tick_handle_oneshot_broadcast) from [<8011b084>] (mxc_timer_interrupt+0x34/0x3c)
r10:80d4e9ab r9:ce003cc0 r8:00000011 r7:00000000 r6:00000000 r5:ce003d20
r4:00000001
[<8011b050>] (mxc_timer_interrupt) from [<80171a1c>] (handle_irq_event_percpu+0x80/0x13c)
r5:ce003d20 r4:80d07900
[<8017199c>] (handle_irq_event_percpu) from [<80171b1c>] (handle_irq_event+0x44/0x64)
r10:80d01d38 r9:ce008000 r8:00000001 r7:80d01e48 r6:80d07900 r5:ce003d20
r4:ce003cc0
[<80171ad8>] (handle_irq_event) from [<80174cf0>] (handle_fasteoi_irq+0xe8/0x198)
r7:80d01e48 r6:80d09170 r5:ce003d20 r4:ce003cc0
[<80174c08>] (handle_fasteoi_irq) from [<80171014>] (generic_handle_irq+0x34/0x44)
r7:80d01e48 r6:00000011 r5:00000000 r4:00000011
[<80170fe0>] (generic_handle_irq) from [<8017130c>] (__handle_domain_irq+0x84/0xf4)
r5:00000000 r4:80c60c04
[<80171288>] (__handle_domain_irq) from [<80101470>] (gic_handle_irq+0x2c/0x64)
r10:80d01e48 r9:ce008000 r8:00000001 r7:f4a00100 r6:80d01d38 r5:80d02888
r4:f4a0010c r3:80d01d38
[<80101444>] (gic_handle_irq) from [<8010c340>] (__irq_svc+0x40/0x74)
Exception stack(0x80d01d38 to 0x80d01d80)
1d20: 00000000 80a9c1c0
1d40: 80d539c0 00000000 00000082 00000000 00000010 00000000 00000001 ce008000
1d60: 80d01e48 80d01ddc 80d01d70 80d01d80 80419e58 8013375c 200e0113 ffffffff
r7:80d01d6c r6:ffffffff r5:200e0113 r4:8013375c
[<801336a4>] (__do_softirq) from [<80133c04>] (irq_exit+0xc8/0x104)
r10:80d01e48 r9:ce008000 r8:00000001 r7:00000000 r6:00000010 r5:00000000
r4:80c60c04
[<80133b3c>] (irq_exit) from [<80171310>] (__handle_domain_irq+0x88/0xf4)
r5:00000000 r4:80c60c04
[<80171288>] (__handle_domain_irq) from [<80101470>] (gic_handle_irq+0x2c/0x64)
r10:d0f0b440 r9:ffffd99a r8:80d02509 r7:f4a00100 r6:80d01e48 r5:80d02888
r4:f4a0010c r3:80d01e48
[<80101444>] (gic_handle_irq) from [<8010c340>] (__irq_svc+0x40/0x74)
Exception stack(0x80d01e48 to 0x80d01e90)
1e40: 0000005d 00000000 dc8ba30f dc8ba30f 00000000 80a953b8
1e60: 00000000 80d53528 80d02509 ffffd99a d0f0b440 80d01eb4 80d01de0 80d01e90
1e80: 80170054 80861ab8 600e0013 ffffffff
r7:80d01e7c r6:ffffffff r5:600e0013 r4:80861ab8
[<80861928>] (panic) from [<8012ff4c>] (print_tainted+0x0/0xb8)
r3:d6dc8ba3 r2:dc8ba30f r1:80193638 r0:80a953b8
r7:0000002f
[<8012ff30>] (__stack_chk_fail) from [<80193638>] (__tick_nohz_idle_enter+0x4c4/0x4cc)
[<80193174>] (__tick_nohz_idle_enter) from [<8019393c>] (tick_nohz_idle_enter+0x48/0x80)
r10:00000001 r9:80d01f58 r8:00000001 r7:d0f07ed0 r6:809028c0 r5:80d025e8
r4:80c60650
[<801938f4>] (tick_nohz_idle_enter) from [<801680f4>] (cpu_startup_entry+0x7c/0x330)
r5:80d025e8 r4:80d00000
[<80168078>] (cpu_startup_entry) from [<80860b48>] (rest_init+0x90/0x94)
r7:80d02500
[<80860ab8>] (rest_init) from [<80c00d90>] (start_kernel+0x400/0x420)
r5:80d51000 r4:80d5104c
[<80c00990>] (start_kernel) from [<1000807c>] (0x1000807c)
---[ end trace 96a3428ae6d0ad12 ]---

[igorpadykov]

Hi Peter

from log problem happens from "adv7610", so one can try to remove

it from image. Try to rebuild image anew from scratch. May be helpful to check without

gpu drivers. Form hardware point of view one can check power supplies ripples

(should be <5%) with oscilloscope.

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[peteramond]

igorpadykov‌

Hi..Thank you for your reply.

Modules linked in: adv7610_video mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc tw6869

Those are the modules which we have connected to custom board and we are using several drivers. I think it just shows them and non of above are errors.

Can you see the Kernel log again and I have edited it and that is the new kernel log.

Regards,

Peter.