RCU stall on kernel 4.4.240

[ivang]

Hi,

I am getting occasionally kernels hanging (sometimes hours after boot) with the following output occurring and that just goes on until i reboot the machine:

INFO: rcu_sched self-detected stall on CPU
1-...: (52300 ticks this GP) idle=439/140000000000001/0 softirq=806264/806264 fqs=52291
INFO: rcu_sched detected stalls on CPUs/tasks:
1-...: (52300 ticks this GP) idle=439/140000000000001/0 softirq=806264/806264 fqs=52291
(detected by 0, t=52542 jiffies, g=463552, c=463551, q=391502)
Task dump for CPU 1:
WQ#1(System) R running 4640 28173 28140 0x00000002
Backtrace:
[<beb5bd7c>] (0xbeb5bd7c) from [<beb5bdac>] (0xbeb5bdac)
Backtrace aborted due to bad frame pointer <8007759c>
(t=52545 jiffies g=463552 c=463551 q=391595)
Task dump for CPU 1:
WQ#1(System) R running 4640 28173 28140 0x00000002
Backtrace:
[<80014a38>] (dump_backtrace) from [<80014c80>] (show_stack+0x20/0x24)
r7:807b26c0 r6:60010193 r5:807b26c0 r4:bcfbae00
[<80014c60>] (show_stack) from [<80057204>] (sched_show_task+0xbc/0x118)
[<80057148>] (sched_show_task) from [<80059354>] (dump_cpu_task+0x3c/0x4c)
r5:807b26c0 r4:00000001
[<80059318>] (dump_cpu_task) from [<80082344>] (rcu_dump_cpu_stacks+0x94/0xd8)
r5:807b26c0 r4:00000001
[<800822b0>] (rcu_dump_cpu_stacks) from [<80086698>] (rcu_check_callbacks+0x548/0x860)
r9:bf7d3100 r8:807b26c0 r7:807a460c r6:3f033000 r5:807a4584 r4:807a0100
[<80086150>] (rcu_check_callbacks) from [<8008aca4>] (update_process_times+0x48/0xa8)
r10:8009d204 r9:bf7cfc98 r8:805989c4 r7:00002fd9 r6:bcfbae00 r5:ffffe000
r4:00000000
[<8008ac5c>] (update_process_times) from [<8009d200>] (tick_sched_handle+0x58/0x5c)
r7:00002fd9 r6:23800e91 r5:beb5bde8 r4:bf7cfdb8
[<8009d1a8>] (tick_sched_handle) from [<8009d258>] (tick_sched_timer+0x54/0x9c)
[<8009d204>] (tick_sched_timer) from [<8008b9fc>] (__hrtimer_run_queues+0x160/0x324)
r7:bf7cfc00 r6:beb5a000 r5:00000000 r4:bf7cfdb8
[<8008b89c>] (__hrtimer_run_queues) from [<8008c2e8>] (hrtimer_interrupt+0xc8/0x22c)
r10:bf7cfc78 r9:bf7cfc98 r8:bf7cfc14 r7:00000000 r6:8079cc00 r5:00000003
r4:bf7cfc00
[<8008c220>] (hrtimer_interrupt) from [<80018728>] (twd_handler+0x3c/0x50)
r10:807eea3c r9:f4000100 r8:bf022d80 r7:00000010 r6:bf7d5700 r5:bf08a400
r4:00000001
[<800186ec>] (twd_handler) from [<8007c1e8>] (handle_percpu_devid_irq+0x88/0x1a0)
r5:bf08a400 r4:8079e38c
[<8007c160>] (handle_percpu_devid_irq) from [<8007759c>] (generic_handle_irq+0x30/0x44)
r10:807a4cc0 r9:f4000100 r8:bf034000 r7:00000001 r6:00000000 r5:00000010
r4:8079e38c r3:8007c160
[<8007756c>] (generic_handle_irq) from [<800778a0>] (__handle_domain_irq+0x60/0xc8)
[<80077840>] (__handle_domain_irq) from [<800094c0>] (gic_handle_irq+0x60/0xc4)
r9:f4000100 r8:807eea38 r7:beb5bde8 r6:000003ff r5:000003eb r4:f400010c
[<80009460>] (gic_handle_irq) from [<80593534>] (__irq_svc+0x54/0x70)
Exception stack(0xbeb5bde8 to 0xbeb5be30)
bde0: bf1a1e44 00000000 0000b2cc 0000a7b4 00000000 00000000
be00: ffffffff 01bc9294 bf1a1e40 bf1a1e40 00000001 beb5be44 80592d34 beb5be38
be20: 8009ef44 80592d60 80010013 ffffffff
r10:00000001 r9:bf1a1e40 r8:bf1a1e40 r7:beb5be1c r6:ffffffff r5:80010013
r4:80592d60
[<80592d20>] (_raw_spin_lock) from [<8009ef44>] (futex_wake+0xa4/0x194)
[<8009eea0>] (futex_wake) from [<800a0ec4>] (do_futex+0x128/0xd80)
r10:000000f0 r9:01bc9294 r8:00000001 r7:01bc9294 r6:00000081 r5:00000000
r4:00000000
[<800a0d9c>] (do_futex) from [<800a1ba4>] (SyS_futex+0x88/0x188)
r10:000000f0 r9:beb5a000 r8:80010844 r7:01bc9294 r6:00000081 r5:00000000
r4:00000001
[<800a1b1c>] (SyS_futex) from [<80010660>] (ret_fast_syscall+0x0/0x48)
r10:000000f0 r9:beb5a000 r8:80010844 r7:000000f0 r6:7ee2e842 r5:01bc92b0
r4:00000000

There doesn't appear to be any pattern, when it occurs and I cannot reproduce it. Any ideas on how to debug this issue?

Thanks,
Ivan

[jimmychan]

Please try the newer release of BSP.

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

[ivang]

HI,

  but there is some similar bug? if yes can you share some details for identify the issue root cause?

Regards,

Ivan