SMP problem on IMX6Q platform

[kaihuang]

Hi

      Sorry for my poor English.I'm porting Atheros 9271 USB wifi driver to a custom imx6q-based board.the wifi driver is compat-wireless-3.4.5 and provided by Qualcomm FAE.My Linux kernel is 3.0.35.when i enable SMP fature in the kernel config,i found that 9271 wifi is unstable.when the wifi connect to the router,and ping the router, sometime the kernel crashed and sometime the wifi is unreachable.if i use " ifconfig -a" to check network status,the command is blocked,and can't be killed by "killall -9 ifconfig " . when i disable SMP fature in the kernel config, the wifi driver works fine .

when the kernel crashed,the log is :

   64 bytes from 192.168.1.1: seq=1627 ttl=64 time=1.634 ms
64 bytes from 192.168.1.1: seq=1628 ttl=64 time=1.619 ms
64 bytes from 192.168.1.1: seq=1629 ttl=64 time=6.590 ms
64 bytes from 192.168.1.1: seq=1630 ttl=64 time=1.561 ms
64 bytes from 192.168.1.1: seq=1631 ttl=64 time=8.824 ms
64 bytes from 192.168.1.1: seq=1632 ttl=64 time=4.062 ms
64 bytes from 192.168.1.1: seq=1633 ttl=64 time=7.273 ms
64 bytes from 192.168.1.1: seq=1634 ttl=64 time=5.504 ms
64 bytes from 192.168.1.1: seq=1635 ttl=64 time=3.498 ms
64 bytes from 192.168.1.1: seq=1636 ttl=64 time=5.352 ms
64 bytes from 192.168.1.1: seq=1637 ttl=64 time=1.708 ms
64 bytes from 192.168.1.1: seq=1638 ttl=64 time=1.569 ms
64 bytes from 192.168.1.1: seq=1639 ttl=64 time=11.560 ms
64 bytes from 192.168.1.1: seq=1640 ttl=64 time=5.028 ms
64 bytes from 192.168.1.1: seq=1641 ttl=64 time=24.942 ms
64 bytes from 192.168.1.1: seq=1642 ttl=64 time=1.640 ms
64 bytes from 192.168.1.1: seq=1643 ttl=64 time=273.256 ms
64 bytes from 192.168.1.1: seq=1644 ttl=64 time=103.760 ms

64 bytes from 192.168.1.1: seq=1645 ttl=64 time=1.633 ms
64 bytes from 192.168.1.1: seq=1646 ttl=64 time=38.835 ms
64 bytes from 192.168.1.1: seq=1647 ttl=64 time=75.095 ms
Unable to handle kernel paging request at virtual address 01010259
pgd = 80004000
[01010259] *pgd=00000000
Internal error: Oops: 5 [#1] PREEMPT SMP
Modules linked in: uvcvideo ath9k_htc mac80211 ath9k_common ath9k_hw ath cfg80211 ov5640_camera_mipi camera_sensor_clock
CPU: 0    Not tainted  (3.0.35+ #218)
PC is at ar5008_hw_process_ini+0x3a0/0x7e8 [ath9k_hw]
LR is at ath9k_regwrite_flush+0x50/0xcc [ath9k_htc]

pc : [<7f050ba8>]    lr : [<7f0db51c>]    psr: 600f0013
sp : 94457e00  ip : 00000001  fp : 00005b84
r10: 00000145  r9 : 01010201  r8 : 000022d4
r7 : 000022d0  r6 : 00000004  r5 : 000022cc  r4 : 94690000
r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 94661a84
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 10c53c7d  Table: 2471804a  DAC: 00000015
Process kworker/u:0 (pid: 1575, stack limit = 0x944562f0)
Stack: (0x94457e00 to 0x94458000)
7e00: 94457e14 00000004 00000064 00000004 7f0e45e0 00000002 946900d0 00000002
7e20: 48700000 000002ba 94690000 00000000 00000000 fffffffb fffffffb 946902a8
7e40: 01010201 946920bc 94690000 7f049918 9466102c 94661800 946f0540 94690000
7e60: 00000000 fffffffb 9469001c 9466180c 942450a0 942450a0 00000040 94244320
7e80: 942456b8 94690000 00000001 00000000 94690000 7f0d909c 94457ecb 00000001
7ea0: 000000c8 00000000 00000001 00000000 9469001c 946902a8 9469001c 7f0e45e0
7ec0: 942453cc 94244320 01000000 80067530 8099401c 942e6440 9448e680 00000000
7ee0: 942e65d4 8008ff40 94456000 809abc00 94457f94 00000000 943ba420 942449c8
7f00: 94244990 7f0e45e0 00000001 7f0a0bd8 94244a50 7f09cfc0 7f0e4628 94244320
7f20: 943ba420 7f0a0dc4 48400000 00807f4e 00021efb 94244998 00000001 00008000

7f40: 00004048 94244a50 943abd00 809e8560 9461bc00 00000000 00000001 7f0a0bd8
7f60: 94456000 80085c18 9461bc00 9424553c 9461bc00 943abd00 809e8560 943abd10
7f80: 94456000 80037320 80037320 809a9cc0 809e8540 800861a4 943abd00 8008606c
7fa0: 00000013 94047f14 943abd00 8008606c 00000013 00000000 00000000 00000000
7fc0: 00000000 8008a9c4 8003ecf8 00000000 943abd00 00000000 00000000 00000000
7fe0: 94457fe0 94457fe0 94047f14 8008a944 8003ecf8 8003ecf8 27ffe821 27ffec21
[<7f050ba8>] (ar5008_hw_process_ini+0x3a0/0x7e8 [ath9k_hw]) from [<7f049918>] (ath9k_hw_reset+0x434/0x10a4 [ath9k_hw])
[<7f049918>] (ath9k_hw_reset+0x434/0x10a4 [ath9k_hw]) from [<7f0d909c>] (ath9k_htc_config+0x218/0x760 [ath9k_htc])
[<7f0d909c>] (ath9k_htc_config+0x218/0x760 [ath9k_htc]) from [<7f09cfc0>] (ieee80211_hw_config+0xd8/0x148 [mac80211])
[<7f09cfc0>] (ieee80211_hw_config+0xd8/0x148 [mac80211]) from [<7f0a0dc4>] (ieee80211_scan_work+0x1ec/0x4e4 [mac80211])
[<7f0a0dc4>] (ieee80211_scan_work+0x1ec/0x4e4 [mac80211]) from [<80085c18>] (process_one_work+0x110/0x39c)
[<80085c18>] (process_one_work+0x110/0x39c) from [<800861a4>] (worker_thread+0x138/0x2f4)
[<800861a4>] (worker_thread+0x138/0x2f4) from [<8008a9c4>] (kthread+0x80/0x88)
[<8008a9c4>] (kthread+0x80/0x88) from [<8003ecf8>] (kernel_thread_exit+0x0/0x8)
Code: e59d201c e2811024 e1a00004 ebffe19b (e5993058)
ipu_init_sync_panel: disp=0, pixel_clk=51200000 50285714BUG: scheduling while atomic: kworker/u:0/1575/0x00000002
Modules linked in: uvcvideo ath9k_htc mac80211 ath9k_common ath9k_hw ath cfg80211 ov5640_camera_mipi camera_sensor_clock
[<8004497c>] (unwind_backtrace+0x0/0xf8) from [<80484910>] (__schedule+0x5d0/0x68c)
[<80484910>] (__schedule+0x5d0/0x68c) from [<80484f98>] (schedule_timeout+0x13c/0x1f0)
[<80484f98>] (schedule_timeout+0x13c/0x1f0) from [<8007c5f8>] (msleep+0x14/0x20)
[<8007c5f8>] (msleep+0x14/0x20) from [<803785b8>] (ipu_init_sync_panel+0x2b0/0x1928)
[<803785b8>] (ipu_init_sync_panel+0x2b0/0x1928) from [<8026af64>] (mxcfb_set_par+0x360/0x694)
[<8026af64>] (mxcfb_set_par+0x360/0x694) from [<8026b33c>] (mxcfb_blank+0xa4/0xb8)
[<8026b33c>] (mxcfb_blank+0xa4/0xb8) from [<802554a8>] (fb_blank+0x38/0x6c)
[<802554a8>] (fb_blank+0x38/0x6c) from [<8025f3bc>] (fbcon_blank+0x1c4/0x25c)
[<8025f3bc>] (fbcon_blank+0x1c4/0x25c) from [<802921dc>] (do_unblank_screen+0x94/0x17c)
[<802921dc>] (do_unblank_screen+0x94/0x17c) from [<80246e74>] (bust_spinlocks+0x24/0x4c)
[<80246e74>] (bust_spinlocks+0x24/0x4c) from [<800417a8>] (die+0xac/0x290)
[<800417a8>] (die+0xac/0x290) from [<8047e238>] (__do_kernel_fault.part.3+0x54/0x74)
[<8047e238>] (__do_kernel_fault.part.3+0x54/0x74) from [<8004853c>] (do_page_fault+0x234/0x2f0)[<8004853c>] (do_page_fault+0x234/0x2f0) from [<80038374>] (do_DataAbort+0x34/0x9c)
[<80038374>] (do_DataAbort+0x34/0x9c) from [<8003dbd0>] (__dabt_svc+0x70/0xa0)
Exception stack(0x94457db8 to 0x94457e00)
7da0:                                                       94661a84 00000000
7dc0: 00000000 00000000 94690000 000022cc 00000004 000022d0 000022d4 01010201
7de0: 00000145 00005b84 00000001 94457e00 7f0db51c 7f050ba8 600f0013 ffffffff
[<8003dbd0>] (__dabt_svc+0x70/0xa0) from [<7f050ba8>] (ar5008_hw_process_ini+0x3a0/0x7e8 [ath9k_hw])
[<7f050ba8>] (ar5008_hw_process_ini+0x3a0/0x7e8 [ath9k_hw]) from [<7f049918>] (ath9k_hw_reset+0x434/0x10a4 [at h9k_hw])
[<7f049918>] (ath9k_hw_reset+0x434/0x10a4 [ath9k_hw]) from [<7f0d909c>] (ath9k_htc_config+0x218/0x760 [ath9k_htc])
[<7f0d909c>] (ath9k_htc_config+0x218/0x760 [ath9k_htc]) from [<7f09cfc0>] (ieee80211_hw_config+0xd8/0x148 [mac 80211])
[<7f09cfc0>] (ieee80211_hw_config+0xd8/0x148 [mac80211]) from [<7f0a0dc4>] (ieee80211_scan_work+0x1ec/0x4e4 [mac80211])
[<7f0a0dc4>] (ieee80211_scan_work+0x1ec/0x4e4 [mac80211]) from [<80085c18>] (process_one_work+0x110/0x39c)
[<80085c18>] (process_one_work+0x110/0x39c) from [<800861a4>] (worker_thread+0x138/0x2f4)

[<800861a4>] (worker_thread+0x138/0x2f4) from [<8008a9c4>] (kthread+0x80/0x88)
[<8008a9c4>] (kthread+0x80/0x88) from [<8003ecf8>] (kernel_thread_exit+0x0/0x8)
---[ end trace 95cff65f5dcb4fb2 ]---
Unable to handle kernel paging request at virtual address fffffffc
pgd = 80004000
[fffffffc] *pgd=27ffe821, *pte=00000000, *ppte=00000000
Internal error: Oops: 17 [#2] PREEMPT SMP
Modules linked in: uvcvideo ath9k_htc mac80211 ath9k_common ath9k_hw ath cfg80211 ov5640_camera_mipi camera_sensor_clock
CPU: 0    Tainted: G      D      (3.0.35+ #218)
PC is at kthread_data+0x4/0xc
LR is at wq_worker_sleeping+0xc/0xd4
pc : [<8008ab84>]    lr : [<80086ff4>]    psr: 00000093
sp : 94457b28  ip : 00000800  fp : 94457bcc
r10: 8bc06e60  r9 : 942e6440  r8 : 942e65d8
r7 : 942e6438  r6 : 942e6440  r5 : 00000000  r4 : 00000000
r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 942e6440
Flags: nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment userControl: 10c53c7d  Table: 2467804a  DAC: 00000015
Process kworker/u:0 (pid: 1575, stack limit = 0x944562f0)
Stack: (0x94457b28 to 0x94458000)
7b20:                   94456000 804846c4 00000627 801015b8 94457b63 00000001
7b40: 943ab410 9400f1e0 80036e60 80036e60 80036e60 800361b4 00000004 94457b63
7b60: 31000001 d896f963 00000001 80073300 00000001 942e6440 941e6220 00000000
7b80: 00000000 944f0a40 944f0f44 809e2998 942e6440 80073314 942e6440 80036260
7ba0: ffffffcf 8bc06648 942e67ac ffffffff 942e6438 942e6440 942e6438 942e655c
7bc0: 94030000 00000000 942e655c 80074588 00000000 94457be8 94456000 00000001
7be0: 94457c5a 942e65a4 94457be8 94457be8 00000001 94456000 00000000 94457c5a
7c00: 00000001 7f050baa 7f050bac 00000000 00000008 80041918 944562f0 0000000b
7c20: 9469001c 00000000 65005b84 32643935 20633130 31383265 34323031 61316520
7c40: 30303030 62652034 31656666 28206239 39393565 38353033 20002029 00000145
7c60: 00005b84 01010259 00000005 00000000 94457db8 94457db8 200f0113 00000145
7c80: 00005b84 8047e238 942e6440 8004853c 00000000 803011cc 94346e9c a00f0013

7ca0: 00000004 94297800 00000000 803011cc 94346e9c 000000d0 00010035 000000d0
7cc0: 24454c00 24454d14 809e1c08 8004b45c 94454c00 00000114 ffdf9ed8 ffdf9ed8
7ce0: 946ed980 94454c00 94297800 946ed980 00000000 00000005 8004867c 809ad200
7d00: 01010259 94457db8 200f0113 00000145 00005b84 80038374 9469001c 9466180c
7d20: 00005b84 802eb0f4 00000000 8bc09490 7f0d4590 00000001 946ed980 94454c00
7d40: 946ed980 944de0c0 9466102c 940007a0 00000064 94661828 94690000 80484044
7d60: 00000020 94002180 7f0d588c 94661000 00000001 944de0c0 9466102c 94661800
7d80: 944de0c0 94690000 00000000 00000015 9469001c 9466180c 00005b84 7f0d5994
7da0: 0000040f 00000005 00000004 000022d0 000022d4 8003dbd0 94661a84 00000000
7dc0: 00000000 00000000 94690000 000022cc 00000004 000022d0 000022d4 01010201
7de0: 00000145 00005b84 00000001 94457e00 7f0db51c 7f050ba8 600f0013 ffffffff
7e00: 94457e14 00000004 00000064 00000004 7f0e45e0 00000002 946900d0 00000002
7e20: 48700000 000002ba 94690000 00000000 00000000 fffffffb fffffffb 946902a8
7e40: 01010201 946920bc 94690000 7f049918 9466102c 94661800 946f0540 94690000
7e60: 00000000 fffffffb 9469001c 9466180c 942450a0 942450a0 00000040 94244320
7e80: 942456b8 94690000 00000001 00000000 94690000 7f0d909c 94457ecb 00000001
7ea0: 000000c8 00000000 00000001 00000000 9469001c 946902a8 9469001c 7f0e45e0
7ec0: 942453cc 94244320 01000000 80067530 8099401c 942e6440 9448e680 00000000
7ee0: 942e65d4 8008ff40 94456000 809abc00 94457f94 00000000 943ba420 942449c8
7f00: 94244990 7f0e45e0 00000001 7f0a0bd8 94244a50 7f09cfc0 7f0e4628 94244320
7f20: 943ba420 7f0a0dc4 48400000 00807f4e 00021efb 94244998 00000001 00008000
7f40: 00004048 94244a50 943abd00 809e8560 9461bc00 00000000 00000001 7f0a0bd8
7f60: 94456000 80085c18 9461bc00 9424553c 9461bc00 943abd00 809e8560 943abd10
7f80: 94456000 80037320 80037320 809a9cc0 809e8540 800861a4 943abd00 8008606c
7fa0: 00000013 94047f14 943abd00 8008606c 00000013 00000000 00000000 00000000
7fc0: 00000000 8008a9c4 8003ecf8 00000000 943abd00 00000001 00000000 00000000
7fe0: 94457fe0 94457fe0 94047f14 8008a944 8003ecf8 8003ecf8 27ffe821 27ffec21
[<8008ab84>] (kthread_data+0x4/0xc) from [<80086ff4>] (wq_worker_sleeping+0xc/0xd4)
[<80086ff4>] (wq_worker_sleeping+0xc/0xd4) from [<804846c4>] (__schedule+0x384/0x68c)
[<804846c4>] (__schedule+0x384/0x68c) from [<80074588>] (do_exit+0x44c/0x6d8)
[<80074588>] (do_exit+0x44c/0x6d8) from [<80041918>] (die+0x21c/0x290)
[<80041918>] (die+0x21c/0x290) from [<8047e238>] (__do_kernel_fault.part.3+0x54/0x74)
[<8047e238>] (__do_kernel_fault.part.3+0x54/0x74) from [<8004853c>] (do_page_fault+0x234/0x2f0)
[<8004853c>] (do_page_fault+0x234/0x2f0) from [<80038374>] (do_DataAbort+0x34/0x9c)[<80038374>] (do_DataAbort+0x34/0x9c) from [<8003dbd0>] (__dabt_svc+0x70/0xa0)
Exception stack(0x94457db8 to 0x94457e00)
7da0:                                                       94661a84 00000000
7dc0: 00000000 00000000 94690000 000022cc 00000004 000022d0 000022d4 01010201
7de0: 00000145 00005b84 00000001 94457e00 7f0db51c 7f050ba8 600f0013 ffffffff
[<8003dbd0>] (__dabt_svc+0x70/0xa0) from [<7f050ba8>] (ar5008_hw_process_ini+0x3a0/0x7e8 [ath9k_hw])
[<7f050ba8>] (ar5008_hw_process_ini+0x3a0/0x7e8 [ath9k_hw]) from [<7f049918>] (ath9k_hw_reset+0x434/0x10a4 [at h9k_hw])
[<7f049918>] (ath9k_hw_reset+0x434/0x10a4 [ath9k_hw]) from [<7f0d909c>] (ath9k_htc_config+0x218/0x760 [ath9k_h tc])
[<7f0d909c>] (ath9k_htc_config+0x218/0x760 [ath9k_htc]) from [<7f09cfc0>] (ieee80211_hw_config+0xd8/0x148 [mac80211])
[<7f09cfc0>] (ieee80211_hw_config+0xd8/0x148 [mac80211]) from [<7f0a0dc4>] (ieee80211_scan_work+0x1ec/0x4e4 [mac80211])
[<7f0a0dc4>] (ieee80211_scan_work+0x1ec/0x4e4 [mac80211]) from [<80085c18>] (process_one_work+0x110/0x39c)

[<80085c18>] (process_one_work+0x110/0x39c) from [<800861a4>] (worker_thread+0x138/0x2f4)
[<800861a4>] (worker_thread+0x138/0x2f4) from [<8008a9c4>] (kthread+0x80/0x88)
[<8008a9c4>] (kthread+0x80/0x88) from [<8003ecf8>] (kernel_thread_exit+0x0/0x8)
Code: eaff985e 80488b54 808f1170 e590316c (e5130004)

[AnsonHuang]

Hi, Kai

     When you enable SMP, you meant there will be 4 cores booting up, right? To narrow down this issue, I think you can add command line of "maxcpus=1" into uboot, then kernel will boot up with SMP enabled but only one core, then redo the test, we can see whether it is caused by multi cores activity or just SMP enabled in kernel. Can you please try that?

[kaihuang]

Hi Yongcai

    the custom imx6-based board have three kind of CPU: imx6s,imx6dl and imx6q,when SMP enabled in kernel config,the same thing happen on imx6s and imx6dl

[AnsonHuang]

Hi, Kai

     Do you running the test on our BSP? I suspected this issue is caused by the atheros wifi driver, have you met same issue running other test? I meant no atheros wifi driver loaded, you can run some stress test to see whether it failed, SMP should be the basic feature of our i.MX6Q platform, our BSP did NOT meet same issue as yours.

[kaihuang]

Hi Yongcai:

   The BSP  i use is  L3.0.35_4.1.0 ,and i have tried Atheros 6302 SDIO wifi and Realtek 8179 USB wifi on the board,both of them worked fine.but my project  use 9271 wifi,so i want to fix this problem.I mailed to Qualcomm FAE,they told me this issue was caused by the BSP,because the wifi driver they give me worked fine on the PC and the other ARM platform.I tested the 9271 wifi driver on TI DM365 platform which is a ARM9 board and the kernel is 2.6.32.17, the wifi was Ok.now i don't know what should I do ? can you help me about this issue?

[AnsonHuang]

Hi, Kai

     I think this issue most likely caused by wifi driver, maybe some of their function calls do NOT considering the SMP environment. From the log, it is obvious that this atheros wifi driver has something wrong. ARM9 platform has big difference from Cortex A9 SMP, maybe you can try it on other TI board, said Panda board(Dual cortex A9 cores with SMP enabled)?

[kaihuang]

Hi Yongcai:

    Can you send your phone number to my mial-box ? I want to call you,my E-mail address :jeff-hk@126.com