88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

[yao_feng]

Platform (Host CPU): i.MX8MQ
Wi-Fi module: AW-CM276NF (interface PCIE/UART)
OS: Linux

• BSP version: Yocto 5.15.71
• Wi-Fi and Bluetooth/Bluetooth LE Firmware version: 16.92.21.p84
• Driver version: Linux 5.15.71

1. HW or SW/FW issue?  FW issue
2. Environment: In shielding room or open site? open site
3. Issue description: FW is in bad state. PCIE command no response, timeout. failed to get signal information.
4. Setup: on board module.
   
   • Linux 5.15.71 Built-in driver,

   [    4.909207] mwifiex_pcie 0000:01:00.0: no quirks enabled
   [    4.909234] mwifiex_pcie 0000:01:00.0: enabling device (0000 -> 0002)
   [    4.909326] mwifiex_pcie: PCI memory map Virt0: 0000000024acaf60 PCI memory map Virt2: 00000000050d867e
   [    5.555157] mwifiex_pcie 0000:01:00.0: info: FW download over, size 439256 bytes
   [    6.313354] mwifiex_pcie 0000:01:00.0: WLAN FW is active
   [    6.349480] mwifiex_pcie 0000:01:00.0: Unknown api_id: 5
   [    6.359375] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84)
   [    6.359393] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.92.21.p84)

5. Steps to reproduce on NXP EVK: Set iperf3 test via 2.4G WiFi router AP & Bluetooth alsa play wav file with Bluetooth speaker in the same time.  
   

   WiFi iperf test:
   iperf3 –c 133.33.33.13 –t 86400
   Audio play via bluealsa
   aplay –D bluealsa:< BT MAC> xxx.wav

6. Observation: AW-CM276NF no response.
7. Fail Rate: 70% -> 1time/8hrs
8. Additional dmesg/logcat/OTA sniffer logs, FW coredump, RF tester logs…

dmesg with FW coredump:
[30166.238363] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa4, act = 0x0
[30166.247313] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
[30166.253280] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
[30166.259137] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
[30166.264644] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
[30166.270067] mwifiex_pcie 0000:01:00.0: last_cmd_index = 1
[30166.275480] mwifiex_pcie 0000:01:00.0: last_cmd_id: 16 00 a4 00 16 00 a4 00 7f 00
[30166.282984] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[30166.290579] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 0
[30166.296434] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 16 80 7f 80 16 80 a4 80 7f 80
[30166.304366] mwifiex_pcie 0000:01:00.0: last_event_index = 2
[30166.309956] mwifiex_pcie 0000:01:00.0: last_event: 0b 00 0a 00 0b 00 0b 00 0a 00
[30166.317370] mwifiex_pcie 0000:01:00.0: data_sent=1 cmd_sent=1
[30166.323130] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
[30166.328788] mwifiex_pcie 0000:01:00.0: failed to get signal information
[30166.336084] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
[30166.337333] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[30166.343008] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84)[30166.343033] mwifiex_pcie 0000:01:00.0: PCIE register dump start
[30166.349540] mwifiex_pcie 0000:01:00.0: failed to get signal information
[30166.357569] mwifiex_pcie 0000:01:00.0: pcie scratch register:
[30166.375967] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
reg:0xcf8, value=0x4bb009e
reg:0xcfc, value=0xde00

any suggestion is welcome.
Thanks.

[yao_feng]

Hi Arthur

Please check personal email.
I attached the version.

Thanks.

[Christine_Li]

Hi, @ArthurC 

Our latest release has been available on our 88W8997 website. Please download it and verify this issue.

Generic_PCIE-WLAN-UART-BT-8997-LNX_6_6_3-IMX8-16.92.21.p119.2-16.92.21.p119.2-MM6X16437.P3-GPL

To remind you, this thread has been closed for more than 1 month. The related SFDC case can not be reopen after 1 month closed. If have any other queries, please feel free to create a new case to us for further tracking, and not continue to reply on this thread. Sorry for the inconvenience and thank you for your corporation.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

The FW crashed issue is still exist in Generic_PCIE-WLAN-UART-BT-8997-LNX_6_6_3-IMX8-16.92.21.p119.2-16.92.21.p119.2-MM6X16437.P3-GPL. 

The previous released test firmware is ok. But with tiny disconnetion in Bluetooth which can be recovered by reconnection.

Why do NXP release firmware early & closed thread without verification with us, the issue reporter?

Are you sure the thread is closed, seriously?

[Christine_Li]

Hi, @ArthurC  @yao_feng 

Sorry for the inconvenience.

Every community case will be automatically linked to a SFDC case to us. It is not easily ignored in the SFDC system. Our SFDC case will be automatically closed after we answered back to customer and not receive customer's reply in 7 days. And case can be reopen in one month after closed. But this case has been more than one month after auto closed. It could not be reopen again. I worry that this case might be ignored or missed because there is no related SFDC case. I know there is some unreasonable process in our SFDC and Community, we are also trying to improve it now.

I am very sorry about the official release still has problem. Based on the fact that this case could not be reopen again, can you please help to create a new one and we will mention in the new case that is related to this one and continue to track it on the new release? 

Best regards,

Christine.

[yao_feng]

check the new one:
https://community.nxp.com/t5/Wireless-Connectivity/88W8997-module-command-timeout-issue-interface-PC...

[ArthurC]

Hello Christine_Li,

Looks like it's internal rule in SFDC & we didn't be mentioned before.

Could you create anouther thread to migrate this issue for re-opening?

[Christine_Li]

Hi, @ArthurC @yao_feng 

Thank you so much for your corporation and understanding.

Sure, let's continue to track and discuss in the new thread.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

We will start to test.

Thank you for announcing to us about FW releasing imformation.

Although it have been late for over 1 month.

And looks like we can't discuss about this issue anymore because it's too late.

[ArthurC]

The FW crashed issue is still exist.

[Christine_Li]

Hi, @yao_feng 

Thanks for creating case to us.

It seems there are FW dump issue happened. Can you please help to provide full logs(dmesg logs, FW_dump, Driver_dump and time synced sniffer if possible) for further debug this issue?

How to capture fw_dump and driver dump?

1. load driver with drvdbg=0xa0037

issue happened

logs will automatically save in the directory like below(You can check the dmesg log)

========================================

[ 4221.173141] Create directory /data/dump_4220 error, try create dir in /var

[ 4221.173147] Wakeup device...

[ 4221.173231] Create directory /var/dump_4220 successfully

[ 4221.173232] Directory name is /var/dump_4220

[ 4221.173233] === START DRIVER INFO DUMP===

[ 4221.173289] DRV dump data in /var/dump_4220/file_drv_info

[ 4221.173306] Wakeup device...

[ 4221.173889] SDIO Func0 (0x0-0x9): 43 03 06 00 07 00 00 02 03 00

[ 4221.175173] SDIO Func1 (0x4-0xb): 00 00 00 00 ff ff ff ff

[ 4221.175879] SDIO Func1: (0x4c) 00 (0x50) 08 (0x54) 07 (0x55) 0c (0x58) 10 (0x59) 00 (0x5c) 00 (0x5d) 00

[ 4221.176284] SDIO Func1 (0xc0-0xca): dc fe 96 00 31 11 00 37 37 02 20

[ 4221.177301] |

[ 4221.275861] SDIO Func1 (0xc0-0xca): dc fe a1 00 3b 00 00 37 37 02 20

[ 4221.276501] Drv info total bytes = 263150 (0x403ee)

[ 4221.276503] === DRIVER INFO DUMP END===Wakeup device...

[ 4221.276555] ==== DEBUG MODE OUTPUT START: 4220.878602 ===

====================================================

2. If you check it doesn't print the directory in dmesg logs,

3.please get it in below way:

- echo debug_dump > /proc/mwlan/adapter0/config

- Wait for some time (1 or 2 min)

- cat /proc/mwlan/adapter0/fw_dump > file_fw_dump

4.drv_dump:

cat /proc/mwlan/adapter0/drv_dump > file_drv_dump

5.Sniffer log：

Wireshark with Wi-Fi adapter is ok.

Best regards,

Christine.

[Christine_Li]

Hi, @yao_feng 

Is there any updates on logs of this issue?

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li ,

Here is the consumer from @yao_feng

The logs generated from driver, mxm_wifiex with  drvdbg=0xa0037 setting are as attached.

Please help to fix issues.

Thank you

[ArthurC]

Hello @Christine_Li ,

Here are issue occured & dump logs in runtime.

In runtime, there is no data in fw_dump buffer.

[ArthurC]

Hello @Christine_Li ,

Issue occured in usage about Wi-Fi 2.4G using iperf3 & Bluetooth baud=3000000 A2DP profile playing audio continuously.

Wi-Fi 5G is stable.

[ArthurC]

Hello @Christine_Li ,

The issue occured & fw_dump is as attached.

[Christine_Li]

Hi, @ArthurC 

Thanks for providing us requested logs.

And sorry for my late reply because we were in new product training last week.

I have check your logs and forward to our internal expert team if necessary.

Once have any updates, I will keep you posted.

Best regards,

Christine.

[Christine_Li]

Hi, @yao_feng 

1. Can you please help to confirm with customer that current provided logs are captured with drvdbg=0xa0037? Because I didn't find this parameter in dmesg logs.

=================

[ 10.671399] mlan: loading out-of-tree module taints kernel.
[ 10.702840] wlan: Loading MWLAN driver
[ 10.706976] wlan: Register to Bus Driver...
[ 10.713553] wlan_pcie 0000:01:00.0: enabling device (0000 -> 0002)
[ 10.719900] Attach moal handle ops, card interface type: 0x204
[ 10.727185] PCIE8997: init module param from usr cfg
[ 10.732631] card_type: PCIE8997, config block: 0
[ 10.737712] cfg80211_wext=0xf
[ 10.741010] max_vir_bss=1
[ 10.744200] cal_data_cfg=none
[ 10.747270] ps_mode = 1
[ 10.749314] RTL8211F Gigabit Ethernet 30be0000.ethernet-1:00: attached PHY driver (mii_bus:phy_addr=30be0000.ethernet-1:00, irq=POLL)
[ 10.749942] auto_ds = 1
[ 10.764239] host_mlme=enable
[ 10.767152] fw_name=nxp/pcie8997_wlan_v4.bin
[ 10.771462] sta_name=wlan
[ 10.774122] drv_mode = 1
[ 10.776686] rx_work=1 cpu_num=4
[ 10.779912] Attach mlan adapter operations.card_type is 0x204.
[ 10.791376] Request firmware: nxp/pcie8997_wlan_v4.bin
[ 11.148733] FW download over, size 439256 bytes
[ 11.841311] WLAN FW is active
[ 11.844356] on_time is 11843131148
[ 11.875053] fw_cap_info=0x587c7fa3, dev_cap_mask=0xffffffff
[ 11.880676] max_p2p_conn = 8, max_sta_conn = 8
[ 11.893907] Register NXP 802.11 Adapter wlan0
[ 11.898372] wlan: version = PCIE8997--16.92.21.p84.4-MM5X16366.p5-GPL-(FP92)
[ 11.906138] wlan: Register to Bus Driver Done
[ 11.910569] wlan: Driver loaded successfully

===============

2. 

• Can you please share inline supplicant logs and sniffers with the console logs?
• Can you please share driver load parameters?

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

1. Can you please help to confirm with customer that current provided logs are captured with drvdbg=0xa0037? Because I didn't find this parameter in dmesg logs.

 The log generated by driver probe with drvdbg=0xa0037 is "wifi_crash_dmesg.log" only.

 "file_fw_dump" file is named fw_dump.

Please found the  post with tag  drvdbg=0xa0037

2. 

• Can you please share inline supplicant logs and sniffers with the console logs?

 We are using NetworkManager for wifi networking, We didn't get any sniffers.

• Can you please share driver load parameters?

Spoiler

modprobe moal mod_para=nxp/wifi_mod_para.conf drvdbg=0xa0037

[Christine_Li]

Hi, @ArthurC 

Thanks for your feedback.

Here is our initial analysis from the shared logs.

• From wifi_faild_dmsg.log:
  • From the dmesg logs deauthentication received and after that scanning continuously for STA.
  • ERR: Role is AP occurred at At the 8283.119064 and IOCTL failure observed at 8334.008714
• From wifi_error_dmesg.log:
  • From the dmesg logs kernel crash is observed. so it resulted the wakeup card timeout.
• From wifi_crash_dmesg.log:
  • From the dmesg logs failure in ADDBA response observed. After that manualy dump has taken. There is no any kernel crash observed.

From all the logs we are not able to relate them with each other. 

So we need  in-sync dmesg logs and sniffer with console logs from the startup of the device to narrow down the issue.

1. Can you please also share me the 88W8997 load parameters in lib/firmware/nxp/wifi_mod_para.conf?

2. Yes, I saw you didn't provide us sniffer logs.

Then can you please help to try to reproduce and capture Sniffer logs + fw_dump + console logs? Please help to capture dmesg/console logs from the startup of the device and load Wi-Fi driver(with drvdbg=0xa0037) so that we can understand the loading parameters you used.

Such as the wifi_error_dmesg.log, we can see what parameters you used in the logs, and it is clear that without with drvdbg=0xa0037 parameter.
============

[ 10.671399] mlan: loading out-of-tree module taints kernel.
[ 10.702840] wlan: Loading MWLAN driver
[ 10.706976] wlan: Register to Bus Driver...
[ 10.713553] wlan_pcie 0000:01:00.0: enabling device (0000 -> 0002)
[ 10.719900] Attach moal handle ops, card interface type: 0x204
[ 10.727185] PCIE8997: init module param from usr cfg
[ 10.732631] card_type: PCIE8997, config block: 0
[ 10.737712] cfg80211_wext=0xf
[ 10.741010] max_vir_bss=1
[ 10.744200] cal_data_cfg=none
[ 10.747270] ps_mode = 1
[ 10.749314] RTL8211F Gigabit Ethernet 30be0000.ethernet-1:00: attached PHY driver (mii_bus:phy_addr=30be0000.ethernet-1:00, irq=POLL)
[ 10.749942] auto_ds = 1
[ 10.764239] host_mlme=enable
[ 10.767152] fw_name=nxp/pcie8997_wlan_v4.bin
[ 10.771462] sta_name=wlan
[ 10.774122] drv_mode = 1
[ 10.776686] rx_work=1 cpu_num=4
[ 10.779912] Attach mlan adapter operations.card_type is 0x204.
[ 10.791376] Request firmware: nxp/pcie8997_wlan_v4.bin

===============

Best regards,

Christine.

[ArthurC]

Hello Christine_Li,

The config file is as attached.

Spoiler

PCIE8997 = {
cfg80211_wext=0xf
max_vir_bss=1
cal_data_cfg=none
ps_mode=1
auto_ds=1
host_mlme=1
fw_name=nxp/pcie8997_wlan_v4.bin
sta_name=wlan
drv_mode=3
}

[yao_feng]

Hi @ArthurC

Could you please help to provide information NXP ask below:

• Can you please share inline supplicant logs and sniffers with the console logs?
• Can you please share driver load parameters? (driver load steps? where you put drvdbg parameters?)
  
  Thanks.
  Yao