Kernel Panic: Device Reboot in endless loop.

surendradhobale — Wed, 30 Aug 2017 08:01:41 GMT

Hello,

We have custom board based on the T1042 QorIQ processor. Booting of the uboot, kernel and FileSystem working fine in the normal case. In one of the software download scenario, we receive the kernel panic.

We also enable and trigger the watchdog through the script.

When issue is observed, received kernel panic logs,

###################################################

Machine check in kernel mode.
Caused by (from MCSR=a000): Load Error Report
Guarded Load Error Report
Oops: Machine check, sig: 7 [#1]
SMP NR_CPUS=4 CoreNet Generic
Modules linked in: Mvbc02xDrv(O) ath10k_pci
CPU: 2 PID: 2875 Comm: java Tainted: G O 4.1.35-rt41 #1
task: d4552c90 ti: da5cc000 task.ti: 8e73e000
NIP: 60000fc8 LR: 0feb5d84 CTR: 0feb5ec0
REGS: da5cdf10 TRAP: 0204 Tainted: G O (4.1.35-rt41)
MSR: 00021002 <CE,ME> CR: 28044842 XER: 20000000
DEAR: e23ad004 ESR: 00000000
GPR00: 0f9c8c64 112fbcb0 11306930 01bffff4 51a1f774 0022eab0 017a2ab0 00000000
GPR08: 78ba924c 01bffff4 00000000 e23ad000 e23ad000 00040005 0eaa3800 00000004
GPR16: 112fdda0 25208708 14e5d128 112feb74 0760b818 0000000a 00000000 00d18008
GPR24: 112fbd28 00d18008 112fdd70 00d18008 00d18008 50b37784 0ffb5220 017a2ab0
NIP [60000fc8] DataTLBError+0x88/0xc0
LR [0feb5d84] 0xfeb5d84
Call Trace:
Instruction dump:
518d5f7a 554c6cfa 7d6c582e 556c0029 41800018 41820034 656b8000 716a003f
7d4c5a78 4800000c 514cbd38 39400000 <816c0004> 7dad5879 7dab6050 7dab682e
---[ end trace 538075413eab0b30 ]---

Machine check in kernel mode.
Caused by (from MCSR=a000): Load Error Report
Guarded Load Error Report
Oops: Machine check, sig: 7 [#2]
SMP NR_CPUS=4 CoreNet Generic
Modules linked in: Mvbc02xDrv(O) ath10k_pci
CPU: 2 PID: 3691 Comm: default.script Tainted: G      D    O    4.1.35-rt41 #1
task: d455f950 ti: da5cc000 task.ti: 62aba000
NIP: 60000fc8 LR: 6015d46c CTR: 00000002
REGS: da5cdf10 TRAP: 0204   Tainted: G      D    O     (4.1.35-rt41)
MSR: 00021002 <CE,ME> CR: 28048884 XER: 00000000
DEAR: e23af004 ESR: 00000000
GPR00: 70706365 62abbd60 d455f950 5fd6ecd7 60ae8e24 00000009 5fd6ecd3 bafdf9fd
GPR08: cbdc8379 876e38d7 00000000 e23af000 e23af000 00040005 d4719300 57d4a000
GPR16: d4ec4300 60c70000 5fd6ecd7 00000001 100b8953 10000000 100b86c4 d49dc4c0
GPR24: 57d64a00 00000000 60ae8e24 00000009 00000003 5fd6ecd7 62aba000 60ae8e24
NIP [60000fc8] DataTLBError+0x88/0xc0
LR [6015d46c] load_elf_binary+0x96c/0x1030
Call Trace:
[62abbd60] [6015d444] load_elf_binary+0x944/0x1030 (unreliable)
[62abbde0] [601109fc] search_binary_handler+0xac/0x230
[62abbe10] [6015b508] load_script+0x258/0x280
[62abbeb0] [601109fc] search_binary_handler+0xac/0x230
[62abbee0] [60111dec] do_execveat_common+0x4dc/0x5e0
[62abbf30] [60111f1c] do_execve+0x2c/0x40
[62abbf40] [6000eb00] ret_from_syscall+0x0/0x3c
--- interrupt: c01 at 0xff18a34
    LR = 0xff1922c
Instruction dump:
518d5f7a 554c6cfa 7d6c582e 556c0029 41800018 41820034 656b8000 716a003f
7d4c5a78 4800000c 514cbd38 39400000 <816c0004> 7dad5879 7dab6050 7dab682e
---[ end trace 538075413eab0b31 ]---

iptables denied: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:24:9b:14:f3:3f:08:00 SRC=192.168.0.125 DST=192.168.0.255 LEN=68 TOS=0x00 PREC=0x00 TTL=128 ID=862 PROTO=UDP SPT=49915 DPT=1947 LEN=48
Machine check in kernel mode.
Caused by (from MCSR=a000): Load Error Report
Guarded Load Error Report
Oops: Machine check, sig: 7 [#3]
SMP NR_CPUS=4 CoreNet Generic
Modules linked in: Mvbc02xDrv(O) ath10k_pci
CPU: 1 PID: 3693 Comm: pppd Tainted: G D O 4.1.35-rt41 #1
task: d458e050 ti: da5d2000 task.ti: 62b16000
NIP: 60000fc8 LR: 60050d98 CTR: 60056d40
REGS: da5d3f10 TRAP: 0204 Tainted: G D O (4.1.35-rt41)
MSR: 00021002 <CE,ME> CR: 2de528e2 XER: 20000000
DEAR: e250a004 ESR: 00000000
GPR00: 60050d54 62b17f30 d458e050 00000e6d 00000000 60c29fa8 00000000 00000000
GPR08: 00000000 00000000 00000000 e250a000 e250a000 00040005 00000000 00000000
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
GPR24: 00000000 00000000 00000000 0000072c 5fb65e90 0000072c 0ff33220 57903068
NIP [60000fc8] DataTLBError+0x88/0xc0
LR [60050d98] schedule_tail+0x58/0xc0
Call Trace:
[62b17f30] [60050d54] schedule_tail+0x14/0xc0 (unreliable)
[62b17f40] [6000ebd4] ret_from_fork+0x50/0x58
Instruction dump:
518d5f7a 554c6cfa 7d6c582e 556c0029 41800018 41820034 656b8000 716a003f
7d4c5a78 4800000c 514cbd38 39400000 <816c0004> 7dad5879 7dab6050 7dab682e
---[ end trace 538075413eab0b32 ]---

Machine check in kernel mode.
Caused by (from MCSR=a000): Load Error Report
Guarded Load Error Report
Oops: Machine check, sig: 7 [#4]
SMP NR_CPUS=4 CoreNet Generic
Modules linked in: Mvbc02xDrv(O) ath10k_pci
CPU: 3 PID: 3694 Comm: default.script Tainted: G      D    O    4.1.35-rt41 #1
task: d44ec010 ti: da5c6000 task.ti: 62600000
NIP: 60000fc8 LR: 6015b624 CTR: 600d2b90
REGS: da5c7f10 TRAP: 0204   Tainted: G      D    O     (4.1.35-rt41)
MSR: 00021002 <CE,ME> CR: 28028888 XER: 20000000
DEAR: e23ab004 ESR: 00000000
GPR00: 6015dac0 62601d50 d44ec010 00000000 00000f90 00000000 57cb106c 00000006
GPR08: 5fffffff 0834ef8f 00000000 e23ab000 e23ab000 00040005 d4ee0a80 fffffff2
GPR16: d47be100 00000006 57cb11ac 00000001 100b8953 10000000 100b86c4 57c70000
GPR24: d4eca580 00000000 d4eca4c0 d4ec10a0 d4eca280 d420c200 62600000 fffffff2
NIP [60000fc8] DataTLBError+0x88/0xc0
LR [6015b624] padzero+0x44/0x70
Call Trace:
[62601d50] [d420c200] 0xd420c200 (unreliable)
[62601d60] [6015dac0] load_elf_binary+0xfc0/0x1030
[62601de0] [601109fc] search_binary_handler+0xac/0x230
[62601e10] [6015b508] load_script+0x258/0x280
[62601eb0] [601109fc] search_binary_handler+0xac/0x230
[62601ee0] [60111dec] do_execveat_common+0x4dc/0x5e0
[62601f30] [60111f1c] do_execve+0x2c/0x40
[62601f40] [6000eb00] ret_from_syscall+0x0/0x3c
--- interrupt: c01 at 0xff18a34
    LR = 0xff1922c
Instruction dump:
518d5f7a 554c6cfa 7d6c582e 556c0029 41800018 41820034 656b8000 716a003f
7d4c5a78 4800000c 514cbd38 39400000 <816c0004> 7dad5879 7dab6050 7dab682e
---[ end trace 538075413eab0b33 ]---

iptables denied: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:24:9b:14:f3:3f:08:00 SRC=192.168.0.125 DST=192.168.0.255 LEN=68 TOS=0x00 PREC=0x00 TTL=128 ID=868 PROTO=UDP SPT=49915 DPT=1947 LEN=48
INFO: rcu_sched self-detected stall on CPU
0: (1 GPs behind) idle=c9f/140000000000001/0 softirq=0/0 fqs=5192
(t=5250 jiffies g=25078 c=25077 q=24)
Task dump for CPU 0:
pppd            R running      0 1836      1 0x00000004
Call Trace:
[d4221b10] [60078d08] rcu_dump_cpu_stacks+0xa8/0x100 (unreliable)
[d4221b30] [6007cc1c] rcu_check_callbacks+0x50c/0x790
[d4221b90] [600809dc] update_process_times+0x3c/0x70
[d4221ba0] [60095428] tick_sched_handle.isra.16+0x38/0x50
[d4221bb0] [600954a8] tick_sched_timer+0x68/0xe0
[d4221be0] [60081814] __run_hrtimer.isra.34+0x54/0xf0
[d4221c00] [60082108] hrtimer_interrupt+0x118/0x330
[d4221c70] [6000958c] __timer_interrupt+0xac/0x1b0
[d4221c90] [600098c0] timer_interrupt+0xb0/0xe0
[d4221cb0] [6000f1b0] ret_from_except+0x0/0x18
--- interrupt: 901 at smp_call_function_many+0x2b0/0x310
    LR = smp_call_function_many+0x280/0x310
[d4221d70] [6009a78c] smp_call_function_many+0x25c/0x310 (unreliable)
[d4221db0] [60016504] __flush_tlb_page+0xb4/0xf0
[d4221dd0] [600f4a04] ptep_clear_flush+0x24/0x40
[d4221de0] [600e306c] wp_page_copy.isra.88+0x1dc/0x490
[d4221e20] [600e4cc0] do_wp_page+0x190/0x740
[d4221e80] [600e6c80] handle_mm_fault+0xc20/0x1030
[d4221f00] [60014974] do_page_fault+0x294/0x5d0
[d4221f40] [6000efac] handle_page_fault+0xc/0x80
--- interrupt: 301 at 0xfe5e824
    LR = 0xfe5e678
Task dump for CPU 1:
pppd            R running      0 3697   1836 0x00000004
Call Trace:
[d4a1de80] [00021002] 0x21002 (unreliable)
iptables denied: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:24:9b:14:f3:3f:08:00 SRC=192.168.0.125 DST=192.168.0.255 LEN=68 TOS=0x00 PREC=0x00 TTL=128 ID=870 PROTO=UDP SPT=49915 DPT=1947 LEN=48
watchdog watchdog0: Timer expired. System will reboot soon!
iptables denied: IN=eth0 OUT= MAC=01:00:5e:00:00:01:90:f6:52:b5:e7:fe:08:00 SRC=192.168.0.254 DST=224.0.0.1 LEN=32 TOS=0x0E PREC=0xE0 TTL=1 ID=0 PROTO=2

##########################################

After this feeding of the watchdog is stopped and continualsy board is rebooting.

Can anyone help to understand, cause of the kernel panic?

Re: Kernel Panic: Device Reboot in endless loop.

Pavel — Thu, 30 Nov 2017 03:30:06 GMT

Your message dump informs about Machine check.

NXP CodeWarrior for the QorIQ supports u-boot and kernel debugging. It is possibly for debugging your software.

Look at the Sections 7.5 and 7.6 of the " Targeting_PA_Processors.pdf" file. This file is available using the following path:

\Freescale\CW_PA_v10.5.1\PA\Help\PDF

This CodeWarrior also supports attach possibility. Use this possibility debugging of your problem.

Have a great day,
Pavel Chubakov

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

topic Re: Kernel Panic: Device Reboot in endless loop. in T-Series

Kernel Panic: Device Reboot in endless loop.

Re: Kernel Panic: Device Reboot in endless loop.