rcu_preempt self-detected stall on imx6q with PREEMPT_RT kernel

[lc85446]

I am trying achieve better real time performance on imx6q board with 3.14.52 kernel.   I use the PREEMPT_RT patch-3.14.53-rt54,  and also patches from meta-fsl-arm/recipes-kernel/linux/linux-fslc-imx-rt at master · Freescale/meta-fsl-arm · GitHub .

But the kernel hanged because of "rcu_preempt self-detected stall on CPU { 0}",   the log is as below

************************************************************************************************************************

3,19378061,405230787577,-;INFO: rcu_preempt self-detected stall on CPU { 0} (t=2100 jiffies g=17846450 c=17846449 q=25140)
4,19378062,405230787586,-;CPU: 0 PID: 127 Comm: irq/147-20e0000 Tainted: P O 3.14.52-rt54 #1
4,19378063,405230787617,-;[<c010fe10>] (unwind_backtrace) from [<c010b390>] (show_stack+0x20/0x24)
4,19378064,405230787634,-;[<c010b390>] (show_stack) from [<c098827c>] (dump_stack+0x84/0xd0)
4,19378065,405230787647,-;[<c098827c>] (dump_stack) from [<c0192c10>] (rcu_check_callbacks+0x58c/0x9f4)
4,19378066,405230787665,-;[<c0192c10>] (rcu_check_callbacks) from [<c013d788>] (update_process_times+0x58/0x74)
4,19378067,405230787679,-;[<c013d788>] (update_process_times) from [<c019fa4c>] (tick_sched_handle.isra.15+0x54/0x58)
4,19378068,405230787691,-;[<c019fa4c>] (tick_sched_handle.isra.15) from [<c019faac>] (tick_sched_timer+0x5c/0x88)
4,19378069,405230787700,-;[<c019faac>] (tick_sched_timer) from [<c0156588>] (__run_hrtimer+0xb0/0x2d0)
4,19378070,405230787711,-;[<c0156588>] (__run_hrtimer) from [<c0157bec>] (hrtimer_interrupt+0x19c/0x420)
4,19378071,405230787722,-;[<c0157bec>] (hrtimer_interrupt) from [<c010f5d4>] (twd_handler+0x40/0x50)

4,19378072,405230787734,-;[<c010f5d4>] (twd_handler) from [<c0189ad4>] (handle_percpu_devid_irq+0x90/0x1bc)
4,19378073,405230787744,-;[<c0189ad4>] (handle_percpu_devid_irq) from [<c01851d0>] (generic_handle_irq+0x3c/0x4c)
4,19378074,405230787755,-;[<c01851d0>] (generic_handle_irq) from [<c0107908>] (handle_IRQ+0x50/0xa0)
4,19378075,405230787764,-;[<c0107908>] (handle_IRQ) from [<c0100460>] (gic_handle_irq+0x3c/0x6c)
4,19378076,405230787772,-;[<c0100460>] (gic_handle_irq) from [<c010c000>] (__irq_svc+0x40/0x84)
4,19378077,405230787776,-;Exception stack(0xd844be60 to 0xd844bea8)
4,19378078,405230787784,-;be60: 00000000 d83e5140 00000001 00000000 d8211b00 d844a028 d83e5160 00000001
4,19378079,405230787791,-;be80: d8011000 d83e5140 c0186b1c d844bec4 d844be88 d844bea8 c0160be8 c0162154
4,19378080,405230787794,-;bea0: 60000053 ffffffff
4,19378081,405230787804,-;[<c010c000>] (__irq_svc) from [<c0162154>] (migrate_disable+0x64/0xf4)
4,19378082,405230787814,-;[<c0162154>] (migrate_disable) from [<c01331c8>] (__local_bh_disable+0x3c/0x40)
4,19378083,405230787823,-;[<c01331c8>] (__local_bh_disable) from [<c01331e4>] (local_bh_disable+0x18/0x1c)

4,19378084,405230787832,-;[<c01331e4>] (local_bh_disable) from [<c0186b3c>] (irq_forced_thread_fn+0x20/0x74)
4,19378085,405230787843,-;[<c0186b3c>] (irq_forced_thread_fn) from [<c0186eb4>] (irq_thread+0x174/0x1e4)
4,19378086,405230787854,-;[<c0186eb4>] (irq_thread) from [<c0153160>] (kthread+0xdc/0xf4)
4,19378087,405230787864,-;[<c0153160>] (kthread) from [<c0106f40>] (ret_from_fork+0x14/0x20)

************************************************************************************************************************

I try to fix this problem by replacing the "spinlock_t irq_lock;" of struct mxc_hdmi  with the "raw_spinlock_t irq_lock;", but it seems no effect.

Could you have any suggestion to fix this rcu_preempt stall problem?

Thanks,

Li Cheng

[igorpadykov]

Hi Li

usually such kind of errors (rcu_preempt stall) are caused by gpu driver,

since 3.14.52 kernel there were many improvements for gpu driver and

this is described in linux release notes documents on link

i.MX Software | NXP 

Note rt patches are not supported by nxp, also this maybe posted on kernel

mail list

Majordomo Lists at VGER.KERNEL.ORG 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[lc85446]

Hi, igor

 Thanks for the quick response,  I will read the documents.

 But I still think that the rcu_preempt stall here relates to hdmi_video,   because the rcu_preempt  stall  was detected on CPU { 0},  and the thread running on CPU 0 is "PID: 127 Comm: irq/147-20e0000" , which is the irq thread for mxc_hdmi.

I have already changed the spin_lock in static irqreturn_t mxc_hdmi_hotplug(int irq, void *data)  to raw_spin_lock, but it seems little effects.  Maybe there are some other things related to mxc_hdmi  which I haven't noticed .

Thanks,

Li