System with kernel 3.14.52 hang up

[jhhong]

Hi.

The problem log is:

INFO: rcu_preempt detected stalls on CPUs/tasks: { 1 2} (detected by 3, t=2102 jiffies, g=1012372, c=1012371, q=284)
Backtrace for cpu 3 (current):
CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.14.52-141236-g279192c-dirty #22
[<c0014e9c>] (unwind_backtrace) from [<c0011a30>] (show_stack+0x10/0x14)
[<c0011a30>] (show_stack) from [<c08ae130>] (dump_stack+0x64/0xb4)
[<c08ae130>] (dump_stack) from [<c0013c08>] (smp_send_all_cpu_backtrace+0x5c/0xc8)
[<c0013c08>] (smp_send_all_cpu_backtrace) from [<c007afec>] (rcu_check_callbacks+0x700/0x860)
[<c007afec>] (rcu_check_callbacks) from [<c003d528>] (update_process_times+0x3c/0x5c)
[<c003d528>] (update_process_times) from [<c0084d80>] (tick_sched_timer+0x70/0xdc)
[<c0084d80>] (tick_sched_timer) from [<c00501a4>] (__run_hrtimer.isra.15+0x58/0x114)
[<c00501a4>] (__run_hrtimer.isra.15) from [<c0050b28>] (hrtimer_interrupt+0x10c/0x2b4)
[<c0050b28>] (hrtimer_interrupt) from [<c00145f4>] (twd_handler+0x2c/0x40)
[<c00145f4>] (twd_handler) from [<c007430c>] (handle_percpu_devid_irq+0x68/0x84)
[<c007430c>] (handle_percpu_devid_irq) from [<c0070c28>] (generic_handle_irq+0x20/0x30)
[<c0070c28>] (generic_handle_irq) from [<c000ecc0>] (handle_IRQ+0x38/0x90)
[<c000ecc0>] (handle_IRQ) from [<c0008578>] (gic_handle_irq+0x28/0x60)
[<c0008578>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xd409df60 to 0xd409dfa8)
df60: d409dfa8 00000000 0b6eff63 000013ed ee7171f0 00000000 0b5bacc6 000013ed
df80: c0d53208 412fc09a 00000001 00000000 00000000 d409dfa8 c007c3e8 c05e17f8
dfa0: 600f0013 ffffffff
[<c0012500>] (__irq_svc) from [<c05e17f8>] (cpuidle_enter_state+0x50/0xec)
[<c05e17f8>] (cpuidle_enter_state) from [<c05e1940>] (cpuidle_idle_call+0xac/0x164)
[<c05e1940>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00709a4>] (cpu_startup_entry+0x9c/0x138)
[<c00709a4>] (cpu_startup_entry) from [<10008644>] (0x10008644)

sending IPI to all other CPUs:
IPI backtrace for cpu 0

CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.52-141236-g279192c-dirty #22
task: c0d509a8 ti: c0d40000 task.ti: c0d40000
PC is at cpuidle_enter_state+0x50/0xec
LR is at ktime_get+0x58/0x118
pc : [<c05e17f8>] lr : [<c007c3e8>] psr: 600f0013
sp : c0d41f70 ip : 00000000 fp : 00000000
r10: 00000001 r9 : 412fc09a r8 : c0d53208
r7 : 000013ed r6 : 15a37c43 r5 : 00000000 r4 : ee6ff1f0
r3 : 000013ed r2 : 15d514e0 r1 : 00000000 r0 : c0d41f70
Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 24b0004a DAC: 00000015

PC: 0xc05e1778:

LR: 0xc007c368:

SP: 0xc0d41ef0:

R0: 0xc0d41ef0:

R4: 0xee6ff170:

R8: 0xc0d53188:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.52-141236-g279192c-dirty #22
[<c0014e9c>] (unwind_backtrace) from [<c0011a30>] (show_stack+0x10/0x14)
[<c0011a30>] (show_stack) from [<c08ae130>] (dump_stack+0x64/0xb4)
[<c08ae130>] (dump_stack) from [<c0013d64>] (handle_IPI+0xf0/0x1ec)
[<c0013d64>] (handle_IPI) from [<c00085a8>] (gic_handle_irq+0x58/0x60)
[<c00085a8>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xc0d41f28 to 0xc0d41f70)
1f20: c0d41f70 00000000 15d514e0 000013ed ee6ff1f0 00000000
1f40: 15a37c43 000013ed c0d53208 412fc09a 00000001 00000000 00000000 c0d41f70
1f60: c007c3e8 c05e17f8 600f0013 ffffffff
[<c0012500>] (__irq_svc) from [<c05e17f8>] (cpuidle_enter_state+0x50/0xec)
[<c05e17f8>] (cpuidle_enter_state) from [<c05e1940>] (cpuidle_idle_call+0xac/0x164)
[<c05e1940>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00709a4>] (cpu_startup_entry+0x9c/0x138)
[<c00709a4>] (cpu_startup_entry) from [<c0c13a94>] (start_kernel+0x300/0x35c)
snvs-secvio 20cc000.caam-snvs: Unhandled Security Violation Interrupt 2 = Watchdog
INFO: rcu_preempt detected stalls on CPUs/tasks: { 1 2} (detected by 3, t=8407 jiffies, g=1012372, c=1012371, q=1216)
Backtrace for cpu 3 (current):

When I change boot args maxcpus=0,the system will not hang up.

The linux kernel version is  3.14.52.

Our platform is Android5.1.1.

How to fix the problem.

[igorpadykov]

Hi jh

this may be caused by ARM or gpu errata, please check that
uboot/include/configs/mx6_common.h for CONFIG_ARM_ERRATA
and for gpu please try latest kernel L4.1.15 where gpu errata
were fixed as described in Table 9 attached Release Notes.

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[jhhong]

The system is hang-up -------When playing video or browse the web, the image stays in a frame, it can't  touch, the serial port can't input.

As you said, can we move  GPU viv5.0.11p8 into our kernel 3.14.52 that use in Android5.1.1 or Debian ?

Otherwise ,The following log also causes the system to hang-up.

[jhhong]

Goodix-TS 1-005d: I2C transfer error: -5
Goodix-TS 1-005d: I2C write end_cmd error
usb 1-1: USB disconnect, device number 2
mmc3: Card stuck in programming state! mmcblk3 mmc_blk_err_check
end_request: I/O error, dev mmcblk3, sector 15074248
EXT4-fs warning (device mmcblk3p4): ext4_end_bio:317: I/O error -5 writing to inode 49434 (offset 0 size 0 starting block 1884282)
Buffer I/O error on device mmcblk3p4, logical block 1575033
mmc3: Timeout waiting for hardware interrupt.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at drivers/mmc/host/sdhci.c:994 sdhci_send_command+0xb18/0xca4()
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.52-141236-g279192c-dirty #23
[<c0015208>] (unwind_backtrace) from [<c0011a34>] (show_stack+0x10/0x14)
[<c0011a34>] (show_stack) from [<c08b8588>] (dump_stack+0x64/0xb4)
[<c08b8588>] (dump_stack) from [<c0033908>] (warn_slowpath_common+0x64/0x84)
[<c0033908>] (warn_slowpath_common) from [<c0033944>] (warn_slowpath_null+0x1c/0x24)
[<c0033944>] (warn_slowpath_null) from [<c06012ac>] (sdhci_send_command+0xb18/0xca4)
[<c06012ac>] (sdhci_send_command) from [<c060203c>] (sdhci_timeout_timer+0x70/0xb4)
[<c060203c>] (sdhci_timeout_timer) from [<c003ce50>] (call_timer_fn.isra.23+0x24/0x84)
[<c003ce50>] (call_timer_fn.isra.23) from [<c003d084>] (run_timer_softirq+0x1d4/0x20c)
[<c003d084>] (run_timer_softirq) from [<c0037404>] (__do_softirq+0x104/0x218)
[<c0037404>] (__do_softirq) from [<c00377bc>] (irq_exit+0xa8/0xec)
[<c00377bc>] (irq_exit) from [<c000ecc4>] (handle_IRQ+0x3c/0x90)
[<c000ecc4>] (handle_IRQ) from [<c0008578>] (gic_handle_irq+0x28/0x60)
[<c0008578>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xc0cedf28 to 0xc0cedf70)
df20: c0cedf70 00000000 d54c2545 00000665 ee70f270 00000000
df40: d4b5c3e0 00000665 c0cff3c8 412fc09a 00000001 00000000 00000000 c0cedf70
df60: c0081ee8 c05ebc0c 60070013 ffffffff
[<c0012500>] (__irq_svc) from [<c05ebc0c>] (cpuidle_enter_state+0x50/0xec)
[<c05ebc0c>] (cpuidle_enter_state) from [<c05ebd54>] (cpuidle_idle_call+0xac/0x164)
[<c05ebd54>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00764a4>] (cpu_startup_entry+0x9c/0x138)
[<c00764a4>] (cpu_startup_entry) from [<c0c22a98>] (start_kernel+0x304/0x360)
---[ end trace 2408246319c7366a ]---
mmcblk3: error -110 sending stop command, original cmd response 0x0, card status 0x400e00
mmc3: Card stuck in programming state! mmcblk3 mmc_blk_err_check
end_request: I/O error, dev mmcblk3, sector 8521224
end_request: I/O error, dev mmcblk3, sector 8521232
Aborting journal on device mmcblk3p4-8.
mmc3: Timeout waiting for hardware interrupt.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at drivers/mmc/host/sdhci.c:994 sdhci_send_command+0xb18/0xca4()
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.14.52-141236-g279192c-dirty #23
[<c0015208>] (unwind_backtrace) from [<c0011a34>] (show_stack+0x10/0x14)
[<c0011a34>] (show_stack) from [<c08b8588>] (dump_stack+0x64/0xb4)
[<c08b8588>] (dump_stack) from [<c0033908>] (warn_slowpath_common+0x64/0x84)
[<c0033908>] (warn_slowpath_common) from [<c0033944>] (warn_slowpath_null+0x1c/0x24)
[<c0033944>] (warn_slowpath_null) from [<c06012ac>] (sdhci_send_command+0xb18/0xca4)
[<c06012ac>] (sdhci_send_command) from [<c060203c>] (sdhci_timeout_timer+0x70/0xb4)
[<c060203c>] (sdhci_timeout_timer) from [<c003ce50>] (call_timer_fn.isra.23+0x24/0x84)
[<c003ce50>] (call_timer_fn.isra.23) from [<c003d084>] (run_timer_softirq+0x1d4/0x20c)
[<c003d084>] (run_timer_softirq) from [<c0037404>] (__do_softirq+0x104/0x218)
[<c0037404>] (__do_softirq) from [<c00377bc>] (irq_exit+0xa8/0xec)
[<c00377bc>] (irq_exit) from [<c000ecc4>] (handle_IRQ+0x3c/0x90)
[<c000ecc4>] (handle_IRQ) from [<c0008578>] (gic_handle_irq+0x28/0x60)
[<c0008578>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xc0cedf28 to 0xc0cedf70)
df20: c0cedf70 00000000 e80d3a41 000006f3 ee70f270 00000000
df40: e777066e 000006f3 c0cff3c8 412fc09a 00000001 00000000 00000000 c0cedf70
df60: c0081ee8 c05ebc0c 600f0013 ffffffff
[<c0012500>] (__irq_svc) from [<c05ebc0c>] (cpuidle_enter_state+0x50/0xec)
[<c05ebc0c>] (cpuidle_enter_state) from [<c05ebd54>] (cpuidle_idle_call+0xac/0x164)
[<c05ebd54>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00764a4>] (cpu_startup_entry+0x9c/0x138)
[<c00764a4>] (cpu_startup_entry) from [<c0c22a98>] (start_kernel+0x304/0x360)
---[ end trace 2408246319c7366b ]---
mmcblk3: error -110 sending stop command, original cmd response 0x0, card status 0x400e00
mmc3: Card stuck in programming state! mmcblk3 mmc_blk_err_check
end_request: I/O error, dev mmcblk3, sector 4047088
Buffer I/O error on device mmcblk3p4, logical block 196638
lost page write due to I/O error on mmcblk3p4
mmc3: Timeout waiting for hardware interrupt.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at drivers/mmc/host/sdhci.c:994 sdhci_send_command+0xb18/0xca4()
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.14.52-141236-g279192c-dirty #23
[<c0015208>] (unwind_backtrace) from [<c0011a34>] (show_stack+0x10/0x14)
[<c0011a34>] (show_stack) from [<c08b8588>] (dump_stack+0x64/0xb4)
[<c08b8588>] (dump_stack) from [<c0033908>] (warn_slowpath_common+0x64/0x84)
[<c0033908>] (warn_slowpath_common) from [<c0033944>] (warn_slowpath_null+0x1c/0x24)
[<c0033944>] (warn_slowpath_null) from [<c06012ac>] (sdhci_send_command+0xb18/0xca4)
[<c06012ac>] (sdhci_send_command) from [<c060203c>] (sdhci_timeout_timer+0x70/0xb4)
[<c060203c>] (sdhci_timeout_timer) from [<c003ce50>] (call_timer_fn.isra.23+0x24/0x84)
[<c003ce50>] (call_timer_fn.isra.23) from [<c003d084>] (run_timer_softirq+0x1d4/0x20c)
[<c003d084>] (run_timer_softirq) from [<c0037404>] (__do_softirq+0x104/0x218)
[<c0037404>] (__do_softirq) from [<c00377bc>] (irq_exit+0xa8/0xec)
[<c00377bc>] (irq_exit) from [<c000ecc4>] (handle_IRQ+0x3c/0x90)
[<c000ecc4>] (handle_IRQ) from [<c0008578>] (gic_handle_irq+0x28/0x60)
[<c0008578>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xc0cedf28 to 0xc0cedf70)
df20: c0cedf70 00000000 face5089 00000781 ee70f270 00000000
df40: fa378d7c 00000781 c0cff3c8 412fc09a 00000001 00000000 00000000 c0cedf70
df60: c0081ee8 c05ebc0c 600f0013 ffffffff
[<c0012500>] (__irq_svc) from [<c05ebc0c>] (cpuidle_enter_state+0x50/0xec)
[<c05ebc0c>] (cpuidle_enter_state) from [<c05ebd54>] (cpuidle_idle_call+0xac/0x164)
[<c05ebd54>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00764a4>] (cpu_startup_entry+0x9c/0x138)
[<c00764a4>] (cpu_startup_entry) from [<c0c22a98>] (start_kernel+0x304/0x360)
---[ end trace 2408246319c7366c ]---
mmcblk3: error -110 sending stop command, original cmd response 0x0, card status 0x400e00
mmc3: Card stuck in programming state! mmcblk3 mmc_blk_err_check
end_request: I/O error, dev mmcblk3, sector 13505184
end_request: I/O error, dev mmcblk3, sector 13505192
end_request: I/O error, dev mmcblk3, sector 13505200
end_request: I/O error, dev mmcblk3, sector 13505208
end_request: I/O error, dev mmcblk3, sector 13505216
end_request: I/O error, dev mmcblk3, sector 13505224
end_request: I/O error, dev mmcblk3, sector 13505232
end_request: I/O error, dev mmcblk3, sector 13505240
EXT4-fs warning (device mmcblk3p4): ext4_end_bio:317: I/O error -5 writing to inode 391708 (offset 0 size 32768 starting block 1688156)
Buffer I/O error on device mmcblk3p4, logical block 1378900
Buffer I/O error on device mmcblk3p4, logical block 1378901
Buffer I/O error on device mmcblk3p4, logical block 1378902
Buffer I/O error on device mmcblk3p4, logical block 1378903
Buffer I/O error on device mmcblk3p4, logical block 1378904
Buffer I/O error on device mmcblk3p4, logical block 1378905
Buffer I/O error on device mmcblk3p4, logical block 1378906
Buffer I/O error on device mmcblk3p4, logical block 1378907
mmc3: Timeout waiting for hardware interrupt.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at drivers/mmc/host/sdhci.c:994 sdhci_send_command+0xb18/0xca4()
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.14.52-141236-g279192c-dirty #23
[<c0015208>] (unwind_backtrace) from [<c0011a34>] (show_stack+0x10/0x14)
[<c0011a34>] (show_stack) from [<c08b8588>] (dump_stack+0x64/0xb4)
[<c08b8588>] (dump_stack) from [<c0033908>] (warn_slowpath_common+0x64/0x84)
[<c0033908>] (warn_slowpath_common) from [<c0033944>] (warn_slowpath_null+0x1c/0x24)
[<c0033944>] (warn_slowpath_null) from [<c06012ac>] (sdhci_send_command+0xb18/0xca4)
[<c06012ac>] (sdhci_send_command) from [<c060203c>] (sdhci_timeout_timer+0x70/0xb4)
[<c060203c>] (sdhci_timeout_timer) from [<c003ce50>] (call_timer_fn.isra.23+0x24/0x84)
[<c003ce50>] (call_timer_fn.isra.23) from [<c003d084>] (run_timer_softirq+0x1d4/0x20c)
[<c003d084>] (run_timer_softirq) from [<c0037404>] (__do_softirq+0x104/0x218)
[<c0037404>] (__do_softirq) from [<c00377bc>] (irq_exit+0xa8/0xec)
fec 2188000.ethernet eth0: MDIO read timeout
[<c00377bc>] (irq_exit) from [<c000ecc4>] (handle_IRQ+0x3c/0x90)
[<c000ecc4>] (handle_IRQ) from [<c0008578>] (gic_handle_irq+0x28/0x60)
[<c0008578>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xc0cedf28 to 0xc0cedf70)
df20: c0cedf70 00000000 138547d2 00000810 ee70f270 00000000
df40: 126c394d 00000810 c0cff3c8 412fc09a 00000001 00000000 00000000 c0cedf70
df60: c0081ee8 c05ebc0c 600f0013 ffffffff
[<c0012500>] (__irq_svc) from [<c05ebc0c>] (cpuidle_enter_state+0x50/0xec)
[<c05ebc0c>] (cpuidle_enter_state) from [<c05ebd54>] (cpuidle_idle_call+0xac/0x164)
[<c05ebd54>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00764a4>] (cpu_startup_entry+0x9c/0x138)
[<c00764a4>] (cpu_startup_entry) from [<c0c22a98>] (start_kernel+0x304/0x360)
---[ end trace 2408246319c7366d ]---
mmcblk3: error -110 sending stop command, original cmd response 0x0, card status 0x400e00
mmc3: Card stuck in programming state! mmcblk3 mmc_blk_err_check
end_request: I/O error, dev mmcblk3, sector 15074248
EXT4-fs warning (device mmcblk3p4): ext4_end_bio:317: I/O error -5 writing to inode 49434 (offset 0 size 0 starting block 1884282)
Buffer I/O error on device mmcblk3p4, logical block 1575033
mmc3: Timeout waiting for hardware interrupt.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at drivers/mmc/host/sdhci.c:994 sdhci_send_command+0xb18/0xca4()
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.14.52-141236-g279192c-dirty #23
[<c0015208>] (unwind_backtrace) from [<c0011a34>] (show_stack+0x10/0x14)
[<c0011a34>] (show_stack) from [<c08b8588>] (dump_stack+0x64/0xb4)
[<c08b8588>] (dump_stack) from [<c0033908>] (warn_slowpath_common+0x64/0x84)
[<c0033908>] (warn_slowpath_common) from [<c0033944>] (warn_slowpath_null+0x1c/0x24)
[<c0033944>] (warn_slowpath_null) from [<c06012ac>] (sdhci_send_command+0xb18/0xca4)
[<c06012ac>] (sdhci_send_command) from [<c060203c>] (sdhci_timeout_timer+0x70/0xb4)
[<c060203c>] (sdhci_timeout_timer) from [<c003ce50>] (call_timer_fn.isra.23+0x24/0x84)
[<c003ce50>] (call_timer_fn.isra.23) from [<c003d084>] (run_timer_softirq+0x1d4/0x20c)
[<c003d084>] (run_timer_softirq) from [<c0037404>] (__do_softirq+0x104/0x218)
[<c0037404>] (__do_softirq) from [<c00377bc>] (irq_exit+0xa8/0xec)
[<c00377bc>] (irq_exit) from [<c000ecc4>] (handle_IRQ+0x3c/0x90)
[<c000ecc4>] (handle_IRQ) from [<c0008578>] (gic_handle_irq+0x28/0x60)
[<c0008578>] (gic_handle_irq) from [<c0012500>] (__irq_svc+0x40/0x70)
Exception stack(0xc0cedf28 to 0xc0cedf70)
df20: c0cedf70 00000000 27778b1b 0000089e ee70f270 00000000
df40: 26786be8 0000089e c0cff3c8 412fc09a 00000001 00000000 00000000 c0cedf70
df60: c0081ee8 c05ebc0c 60070013 ffffffff
[<c0012500>] (__irq_svc) from [<c05ebc0c>] (cpuidle_enter_state+0x50/0xec)
[<c05ebc0c>] (cpuidle_enter_state) from [<c05ebd54>] (cpuidle_idle_call+0xac/0x164)
[<c05ebd54>] (cpuidle_idle_call) from [<c000f10c>] (arch_cpu_idle+0x8/0x44)
[<c000f10c>] (arch_cpu_idle) from [<c00764a4>] (cpu_startup_entry+0x9c/0x138)
[<c00764a4>] (cpu_startup_entry) from [<c0c22a98>] (start_kernel+0x304/0x360)
---[ end trace 2408246319c7366e ]---
mmcblk3: error -110 sending stop command, original cmd response 0x0, card status 0x400e00

[igorpadykov]

there is no other solution as to implement all latest gpu/arm errata fixes.