Hi All,
I am attempting to get the rt kernel working on my project, it appears to be throwing stack errors quite frequently. I think the following log is several errors, but is the most common error I get.
The hardware design is based on the SabreSD reference design, I am using a dual core chip, and have linked in custom drivers for hardware, these are denoted by the `lx_` in the module name.
The stack errors occur while running a QT program with a full GUI and several hardware interaction operations + threads.
The Kernel is built using Yocto and the meta-fsl-arm recipes.
Basically, looking for some help debugging this. I don't have a lot of Linux development under my belt. Have I missed an RT config file/setting during compile time? Am I doing things wrong in my QT application?
Stack Traces:
BUG: scheduling while atomic: swapper/1/0/0x00000002
Modules linked in: g_ffs libcomposite configfs lx_max11646(O) lx_vcnl(O) lx_sht21(O) wl18xx(O) wlcore(O) mac80211(O) cfg80211(O) wlcore_sdio(O) compat(O)
Preemption disabled at:[< (null)>] (null)
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W O 3.10.17-rt12-1.0.0_ga+g232293e #1
[<8001476c>] (unwind_backtrace+0x0/0x134) from [<80012160>] (show_stack+0x10/0x14)
[<80012160>] (show_stack+0x10/0x14) from [<8056522c>] (__schedule_bug+0x9c/0xb0)
[<8056522c>] (__schedule_bug+0x9c/0xb0) from [<8056971c>] (__schedule+0x434/0x4a8)
[<8056971c>] (__schedule+0x434/0x4a8) from [<805697c4>] (schedule+0x34/0xa4)
[<805697c4>] (schedule+0x34/0xa4) from [<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c)
[<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c) from [<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c)
[<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c) from [<8003497c>] (del_timer+0x28/0x58)
[<8003497c>] (del_timer+0x28/0x58) from [<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180)
[<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180) from [<8004b8fc>] (notifier_call_chain+0x44/0x84)
[<8004b8fc>] (notifier_call_chain+0x44/0x84) from [<8004bb90>] (atomic_notifier_call_chain+0x34/0x48)
[<8004bb90>] (atomic_notifier_call_chain+0x34/0x48) from [<8000f8c0>] (arch_cpu_idle+0xc/0x54)
[<8000f8c0>] (arch_cpu_idle+0xc/0x54) from [<8005e770>] (cpu_startup_entry+0xfc/0x160)
[<8005e770>] (cpu_startup_entry+0xfc/0x160) from [<105616a4>] (0x105616a4)
bad: scheduling from the idle thread!
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W O 3.10.17-rt12-1.0.0_ga+g232293e #1
[<8001476c>] (unwind_backtrace+0x0/0x134) from [<80012160>] (show_stack+0x10/0x14)
[<80012160>] (show_stack+0x10/0x14) from [<80055524>] (dequeue_task_idle+0x1c/0x28)
[<80055524>] (dequeue_task_idle+0x1c/0x28) from [<8004f4ec>] (dequeue_task+0x2c/0x4c)
[<8004f4ec>] (dequeue_task+0x2c/0x4c) from [<805693d0>] (__schedule+0xe8/0x4a8)
[<805693d0>] (__schedule+0xe8/0x4a8) from [<805697c4>] (schedule+0x34/0xa4)
[<805697c4>] (schedule+0x34/0xa4) from [<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c)
[<8056a394>] (rt_spin_lock_slowlock+0xc0/0x29c) from [<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c)
[<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c) from [<8003497c>] (del_timer+0x28/0x58)
[<8003497c>] (del_timer+0x28/0x58) from [<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180)
[<803d4170>] (cpufreq_interactive_idle_notifier+0x168/0x180) from [<8004b8fc>] (notifier_call_chain+0x44/0x84)
[<8004b8fc>] (notifier_call_chain+0x44/0x84) from [<8004bb90>] (atomic_notifier_call_chain+0x34/0x48)
[<8004bb90>] (atomic_notifier_call_chain+0x34/0x48) from [<8000f8c0>] (arch_cpu_idle+0xc/0x54)
[<8000f8c0>] (arch_cpu_idle+0xc/0x54) from [<8005e770>] (cpu_startup_entry+0xfc/0x160)
[<8005e770>] (cpu_startup_entry+0xfc/0x160) from [<105616a4>] (0x105616a4)
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = 80004000
[00000000] *pgd=00000000
Internal error: Oops: 80000007 [#1] PREEMPT SMP ARM
Modules linked in: g_ffs libcomposite configfs lx_max11646(O) lx_vcnl(O) lx_sht21(O) wl18xx(O) wlcore(O) mac80211(O) cfg80211(O) wlcore_sdio(O) compat(O)
CPU: 1 PID: 11 Comm: rcu_preempt Tainted: G W O 3.10.17-rt12-1.0.0_ga+g232293e #1
task: a8072d00 ti: a8094000 task.ti: a8094000
PC is at 0x0
LR is at enqueue_task+0x2c/0x4c
pc : [<00000000>] lr : [<8004f4a0>] psr: 80070093
sp : a8095e20 ip : 00000420 fp : a8095e34
r10: 80b62ea0 r9 : 80b571c0 r8 : 20070013
r7 : a8073e04 r6 : 008eb000 r5 : 814421c0 r4 : 814421c0
r3 : 00000000 r2 : 00000005 r1 : a8073a80 r0 : 814421c0
Flags: Nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 3874004a DAC: 00000015
Process rcu_preempt (pid: 11, stack limit = 0xa8094238)
Stack: (0xa8095e20 to 0xa8096000)
5e20: 00000005 a8073a80 a8095e6c a8073a80 a8095e6c 80052d04 a8073084 20070013
5e40: a8095f38 8056ad50 a8095f38 80bf04c0 00000001 80046944 80bf04c0 00000000
5e60: a8094000 80b62ea0 a8095f38 8056a150 80bf04c0 800349f0 00000000 00000000
5e80: 00000001 a8095ecc a8094010 80034a5c a8094030 a8072d00 80bf04c0 80b5a380
5ea0: 80bf04c0 00083306 80bf04c0 a8095ecc 80bf04c0 80b5a380 80bf04c0 00083306
5ec0: 80bf04c0 80568820 00000000 00000000 00200200 00083306 80bf04c0 80034310
5ee0: a8072d00 ffffffff 00000000 00000002 80b6e250 a8094000 00000064 00000002
5f00: 80b6e140 80082ef8 d2b5ffef 81438bec 80b5a380 80b6df40 80b6e16c 80b630f0
5f20: 80b6e250 80b6e29c 80b9f47a a8094000 a8094000 00000000 a8095f38 a8095f38
5f40: 80082c24 a805bee0 00000000 80b6e140 80082c24 00000000 00000000 00000000
5f60: 00000000 80046124 7befcfd6 00000000 f1ff3ef6 80b6e140 00000000 00000000
5f80: a8095f80 a8095f80 00000000 00000000 a8095f90 a8095f90 a8095fac a805bee0
5fa0: 80046080 00000000 00000000 8000ed18 00000000 00000000 00000000 00000000
5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 feb7bf3b 7ffe1bff
[<8004f4a0>] (enqueue_task+0x2c/0x4c) from [<80052d04>] (try_to_wake_up+0x1b8/0x338)
[<80052d04>] (try_to_wake_up+0x1b8/0x338) from [<8056a150>] (rt_spin_lock_slowunlock+0x24/0xd4)
[<8056a150>] (rt_spin_lock_slowunlock+0x24/0xd4) from [<800349f0>] (try_to_del_timer_sync+0x44/0x5c)
[<800349f0>] (try_to_del_timer_sync+0x44/0x5c) from [<80034a5c>] (del_timer_sync+0x54/0xe8)
[<80034a5c>] (del_timer_sync+0x54/0xe8) from [<80568820>] (schedule_timeout+0x150/0x1cc)
[<80568820>] (schedule_timeout+0x150/0x1cc) from [<80082ef8>] (rcu_gp_kthread+0x2d4/0x5d8)
[<80082ef8>] (rcu_gp_kthread+0x2d4/0x5d8) from [<80046124>] (kthread+0xa4/0xa8)
[<80046124>] (kthread+0xa4/0xa8) from [<8000ed18>] (ret_from_fork+0x14/0x3c)
Code: bad PC value
---[ end trace 0000000000000007 ]---
Kernel panic - not syncing: Fatal exception
CPU0: stopping
CPU: 0 PID: 393 Comm: irq/164-wl18xx Tainted: G D W O 3.10.17-rt12-1.0.0_ga+g232293e #1
[<8001476c>] (unwind_backtrace+0x0/0x134) from [<80012160>] (show_stack+0x10/0x14)
[<80012160>] (show_stack+0x10/0x14) from [<80013768>] (handle_IPI+0x108/0x160)
[<80013768>] (handle_IPI+0x108/0x160) from [<80008584>] (gic_handle_irq+0x58/0x5c)
[<80008584>] (gic_handle_irq+0x58/0x5c) from [<8000e840>] (__irq_svc+0x40/0x84)
Exception stack(0xa8eb9db8 to 0xa8eb9e00)
9da0: 00000000 00000000
9dc0: 00004fde 00004fdd 80bf04c0 80bf04c0 a8e5db90 00000000 a8eb9e8c a8b28f00
9de0: a8d80480 80bf04c0 80b5a380 a8eb9e00 800504f4 8056ad24 80070113 ffffffff
[<8000e840>] (__irq_svc+0x40/0x84) from [<8056ad24>] (_raw_spin_lock+0x38/0x4c)
[<8056ad24>] (_raw_spin_lock+0x38/0x4c) from [<8056a308>] (rt_spin_lock_slowlock+0x34/0x29c)
[<8056a308>] (rt_spin_lock_slowlock+0x34/0x29c) from [<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c)
[<800344c8>] (lock_timer_base.isra.29+0x2c/0x4c) from [<80034690>] (mod_timer+0x60/0x1c4)
[<80034690>] (mod_timer+0x60/0x1c4) from [<8003f474>] (queue_delayed_work_on+0xfc/0x104)
[<8003f474>] (queue_delayed_work_on+0xfc/0x104) from [<7f0ba6b0>] (wlcore_irq_locked+0xe4/0x348 [wlcore])
[<7f0ba6b0>] (wlcore_irq_locked+0xe4/0x348 [wlcore]) from [<7f0bbee8>] (wlcore_irq+0xcc/0x18c [wlcore])
[<7f0bbee8>] (wlcore_irq+0xcc/0x18c [wlcore]) from [<8007af24>] (irq_thread_fn+0x1c/0x40)
[<8007af24>] (irq_thread_fn+0x1c/0x40) from [<8007b264>] (irq_thread+0x150/0x1b0)
[<8007b264>] (irq_thread+0x150/0x1b0) from [<80046124>] (kthread+0xa4/0xa8)
[<80046124>] (kthread+0xa4/0xa8) from [<8000ed18>] (ret_from_fork+0x14/0x3c)
Thanks Igor, I'll give the 3.10.17 patch a go.
I am obviously hesitant to move to the 3.14.28 kernel late in the product development that we are in, is there any obvious benefits/downsides to move to 3.14.28?
Hi Phill
it has much more gpu fixes as described in
IMX6L31428101RN i.MX 6 Yocto Project L3.14.28_1.0.1 Patch Release Notes -Release Notes
Best regards
igor
Hi Phill
this may be caused by arm/gpu bugs fixed on 3.10.17 patches, please refer to attached doc
L3.10.17_1.0.2_iMX6QDLS_BUNDLE : i.MX 6Quad, i.MX 6Dual, i.MX 6DualLite,
i.MX 6Solo Linux Binary Demo Files and Linux BSP Documentation.
Size (K): 717480 Format: gz Rev #: L3.10.17_1.0.2 Modified: 10/29/2014
In general, recommended to use latest L3.14.28_1.0.0_iMX6QDLS_BUNDLE
Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------