meta-fsl-arm/linux-rt rcu_preempt stack errors (IMX.6D)

[phillogden]

Hi All,

I am attempting to get the rt kernel working on my project, it appears to be throwing stack errors quite frequently. I think the following log is several errors, but is the most common error I get.

The hardware design is based on the SabreSD reference design, I am using a dual core chip, and have linked in custom drivers for hardware, these are denoted by the `lx_` in the module name.

The stack errors occur while running a QT program with a full GUI and several hardware interaction operations + threads.

The Kernel is built using Yocto and the meta-fsl-arm recipes.

Basically, looking for some help debugging this. I don't have a lot of Linux development under my belt. Have I missed an RT config file/setting during compile time? Am I doing things wrong in my QT application?

Stack Traces:

BUG: scheduling while atomic: swapper/1/0/0x00000002

Modules linked in: g_ffs libcomposite configfs lx_max11646(O) lx_vcnl(O) lx_sht21(O) wl18xx(O) wlcore(O) mac80211(O) cfg80211(O) wlcore_sdio(O) compat(O)

Preemption disabled at:[<  (null)>]   (null)

CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W  O 3.10.17-rt12-1.0.0_ga+g232293e #1

[<8001476c>] (unwind_backtrace+0x0/0x134) from [<80012160>] (show_stack+0x10/0x14)

[<80012160>] (show_stack+0x10/0x14) from [<8056522c>] (__schedule_bug+0x9c/0xb0)

[<8056522c>] (__schedule_bug+0x9c/0xb0) from [<8056971c>] (__schedule+0x434/0x4a8)

[<8056971c>] (__schedule+0x434/0x4a8) from [<805697c4>] (schedule+0x34/0xa4)

[<805697c4>] (schedule+0x34/0xa4) from [<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c)

[<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c) from [<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c)

[<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c) from [<8003497c>] (del_timer+0x28/0x58)

[<8003497c>] (del_timer+0x28/0x58) from [<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180)

[<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180) from [<8004b8fc>] (notifier_call_chain+0x44/0x84)

[<8004b8fc>] (notifier_call_chain+0x44/0x84) from [<8004bb90>] (atomic_notifier_call_chain+0x34/0x48)

[<8004bb90>] (atomic_notifier_call_chain+0x34/0x48) from [<8000f8c0>] (arch_cpu_idle+0xc/0x54)

[<8000f8c0>] (arch_cpu_idle+0xc/0x54) from [<8005e770>] (cpu_startup_entry+0xfc/0x160)

[<8005e770>] (cpu_startup_entry+0xfc/0x160) from [<105616a4>] (0x105616a4)

bad: scheduling from the idle thread!

CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W  O 3.10.17-rt12-1.0.0_ga+g232293e #1

[<8001476c>] (unwind_backtrace+0x0/0x134) from [<80012160>] (show_stack+0x10/0x14)

[<80012160>] (show_stack+0x10/0x14) from [<80055524>] (dequeue_task_idle+0x1c/0x28)

[<80055524>] (dequeue_task_idle+0x1c/0x28) from [<8004f4ec>] (dequeue_task+0x2c/0x4c)

[<8004f4ec>] (dequeue_task+0x2c/0x4c) from [<805693d0>] (__schedule+0xe8/0x4a8)

[<805693d0>] (__schedule+0xe8/0x4a8) from [<805697c4>] (schedule+0x34/0xa4)

[<805697c4>] (schedule+0x34/0xa4) from [<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c)

[<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c) from [<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c)

[<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c) from [<8003497c>] (del_timer+0x28/0x58)

[<8003497c>] (del_timer+0x28/0x58) from [<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180)

[<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180) from [<8004b8fc>] (notifier_call_chain+0x44/0x84)

[<8004b8fc>] (notifier_call_chain+0x44/0x84) from [<8004bb90>] (atomic_notifier_call_chain+0x34/0x48)

[<8004bb90>] (atomic_notifier_call_chain+0x34/0x48) from [<8000f8c0>] (arch_cpu_idle+0xc/0x54)

[<8000f8c0>] (arch_cpu_idle+0xc/0x54) from [<8005e770>] (cpu_startup_entry+0xfc/0x160)

[<8005e770>] (cpu_startup_entry+0xfc/0x160) from [<105616a4>] (0x105616a4)

Unable to handle kernel NULL pointer dereference at virtual address 00000000

pgd = 80004000

[00000000] *pgd=00000000

Internal error: Oops: 80000007 [#1] PREEMPT SMP ARM

Modules linked in: g_ffs libcomposite configfs lx_max11646(O) lx_vcnl(O) lx_sht21(O) wl18xx(O) wlcore(O) mac80211(O) cfg80211(O) wlcore_sdio(O) compat(O)

CPU: 1 PID: 11 Comm: rcu_preempt Tainted: G        W  O 3.10.17-rt12-1.0.0_ga+g232293e #1

task: a8072d00 ti: a8094000 task.ti: a8094000

PC is at 0x0

LR is at enqueue_task+0x2c/0x4c

pc : [<00000000>]    lr : [<8004f4a0>]    psr: 80070093

sp : a8095e20  ip : 00000420  fp : a8095e34

r10: 80b62ea0  r9 : 80b571c0  r8 : 20070013

r7 : a8073e04  r6 : 008eb000  r5 : 814421c0  r4 : 814421c0

r3 : 00000000  r2 : 00000005  r1 : a8073a80  r0 : 814421c0

Flags: Nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel

Control: 10c53c7d  Table: 3874004a  DAC: 00000015

Process rcu_preempt (pid: 11, stack limit = 0xa8094238)

Stack: (0xa8095e20 to 0xa8096000)

5e20: 00000005 a8073a80 a8095e6c a8073a80 a8095e6c 80052d04 a8073084 20070013

5e40: a8095f38 8056ad50 a8095f38 80bf04c0 00000001 80046944 80bf04c0 00000000

5e60: a8094000 80b62ea0 a8095f38 8056a150 80bf04c0 800349f0 00000000 00000000

5e80: 00000001 a8095ecc a8094010 80034a5c a8094030 a8072d00 80bf04c0 80b5a380

5ea0: 80bf04c0 00083306 80bf04c0 a8095ecc 80bf04c0 80b5a380 80bf04c0 00083306

5ec0: 80bf04c0 80568820 00000000 00000000 00200200 00083306 80bf04c0 80034310

5ee0: a8072d00 ffffffff 00000000 00000002 80b6e250 a8094000 00000064 00000002

5f00: 80b6e140 80082ef8 d2b5ffef 81438bec 80b5a380 80b6df40 80b6e16c 80b630f0

5f20: 80b6e250 80b6e29c 80b9f47a a8094000 a8094000 00000000 a8095f38 a8095f38

5f40: 80082c24 a805bee0 00000000 80b6e140 80082c24 00000000 00000000 00000000

5f60: 00000000 80046124 7befcfd6 00000000 f1ff3ef6 80b6e140 00000000 00000000

5f80: a8095f80 a8095f80 00000000 00000000 a8095f90 a8095f90 a8095fac a805bee0

5fa0: 80046080 00000000 00000000 8000ed18 00000000 00000000 00000000 00000000

5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 feb7bf3b 7ffe1bff

[<8004f4a0>] (enqueue_task+0x2c/0x4c) from [<80052d04>] (try_to_wake_up+0x1b8/0x338)

[<80052d04>] (try_to_wake_up+0x1b8/0x338) from [<8056a150>] (rt_spin_lock_slowunlock+0x24/0xd4)

[<8056a150>] (rt_spin_lock_slowunlock+0x24/0xd4) from [<800349f0>] (try_to_del_timer_sync+0x44/0x5c)

[<800349f0>] (try_to_del_timer_sync+0x44/0x5c) from [<80034a5c>] (del_timer_sync+0x54/0xe8)

[<80034a5c>] (del_timer_sync+0x54/0xe8) from [<80568820>] (schedule_timeout+0x150/0x1cc)

[<80568820>] (schedule_timeout+0x150/0x1cc) from [<80082ef8>] (rcu_gp_kthread+0x2d4/0x5d8)

[<80082ef8>] (rcu_gp_kthread+0x2d4/0x5d8) from [<80046124>] (kthread+0xa4/0xa8)

[<80046124>] (kthread+0xa4/0xa8) from [<8000ed18>] (ret_from_fork+0x14/0x3c)

Code: bad PC value

---[ end trace 0000000000000007 ]---

Kernel panic - not syncing: Fatal exception

CPU0: stopping

CPU: 0 PID: 393 Comm: irq/164-wl18xx Tainted: G      D W  O 3.10.17-rt12-1.0.0_ga+g232293e #1

[<8001476c>] (unwind_backtrace+0x0/0x134) from [<80012160>] (show_stack+0x10/0x14)

[<80012160>] (show_stack+0x10/0x14) from [<80013768>] (handle_IPI+0x108/0x160)

[<80013768>] (handle_IPI+0x108/0x160) from [<80008584>] (gic_handle_irq+0x58/0x5c)

[<80008584>] (gic_handle_irq+0x58/0x5c) from [<8000e840>] (__irq_svc+0x40/0x84)

Exception stack(0xa8eb9db8 to 0xa8eb9e00)

9da0:                                                       00000000 00000000

9dc0: 00004fde 00004fdd 80bf04c0 80bf04c0 a8e5db90 00000000 a8eb9e8c a8b28f00

9de0: a8d80480 80bf04c0 80b5a380 a8eb9e00 800504f4 8056ad24 80070113 ffffffff

[<8000e840>] (__irq_svc+0x40/0x84) from [<8056ad24>] (_raw_spin_lock+0x38/0x4c)

[<8056ad24>] (_raw_spin_lock+0x38/0x4c) from [<8056a308>] (rt_spin_lock_slowlock+0x34/0x29c)

[<8056a308>] (rt_spin_lock_slowlock+0x34/0x29c) from [<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c)

[<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c) from [<80034690>] (mod_timer+0x60/0x1c4)

[<80034690>] (mod_timer+0x60/0x1c4) from [<8003f474>] (queue_delayed_work_on+0xfc/0x104)

[<8003f474>] (queue_delayed_work_on+0xfc/0x104) from [<7f0ba6b0>] (wlcore_irq_locked+0xe4/0x348 [wlcore])

[<7f0ba6b0>] (wlcore_irq_locked+0xe4/0x348 [wlcore]) from [<7f0bbee8>] (wlcore_irq+0xcc/0x18c [wlcore])

[<7f0bbee8>] (wlcore_irq+0xcc/0x18c [wlcore]) from [<8007af24>] (irq_thread_fn+0x1c/0x40)

[<8007af24>] (irq_thread_fn+0x1c/0x40) from [<8007b264>] (irq_thread+0x150/0x1b0)

[<8007b264>] (irq_thread+0x150/0x1b0) from [<80046124>] (kthread+0xa4/0xa8)

[<80046124>] (kthread+0xa4/0xa8) from [<8000ed18>] (ret_from_fork+0x14/0x3c)

[phillogden]

Thanks Igor, I'll give the 3.10.17 patch a go.

I am obviously hesitant to move to the 3.14.28 kernel late in the product development that we are in, is there any obvious benefits/downsides to move to 3.14.28?

[igorpadykov]

Hi Phill

it has much more gpu fixes as described in

IMX6L31428101RN  i.MX 6 Yocto Project L3.14.28_1.0.1 Patch Release Notes -Release Notes

Best regards

igor

[igorpadykov]

Hi Phill

this may be caused by arm/gpu bugs fixed on 3.10.17 patches, please refer to attached doc

L3.10.17_1.0.2_iMX6QDLS_BUNDLE  : i.MX 6Quad, i.MX 6Dual, i.MX 6DualLite,

i.MX 6Solo Linux Binary Demo Files and Linux BSP Documentation.

Size (K): 717480 Format: gz Rev #: L3.10.17_1.0.2 Modified: 10/29/2014

In general, recommended to use latest L3.14.28_1.0.0_iMX6QDLS_BUNDLE

Best regards

igor

-----------------------------------------------------------------------------------------------------------------------

Note: If this post answers your question, please click the Correct Answer button. Thank you!

-----------------------------------------------------------------------------------------------------------------------