topic Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ) in Wi-Fi® + Bluetooth® + 802.15.4

88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

yao_feng — Wed, 17 Jan 2024 18:40:42 GMT

Platform (Host CPU): i.MX8MQ
Wi-Fi module: AW-CM276NF (interface PCIE/UART)
OS: Linux

BSP version: Yocto 5.15.71
Wi-Fi and Bluetooth/Bluetooth LE Firmware version: 16.92.21.p84
Driver version: Linux 5.15.71

HW or SW/FW issue? FW issue
Environment: In shielding room or open site? open site
Issue description: FW is in bad state. PCIE command no response, timeout. failed to get signal information.
Setup: on board module.
- Linux 5.15.71 Built-in driver,
[    4.909207] mwifiex_pcie 0000:01:00.0: no quirks enabled
[    4.909234] mwifiex_pcie 0000:01:00.0: enabling device (0000 -> 0002)
[    4.909326] mwifiex_pcie: PCI memory map Virt0: 0000000024acaf60 PCI memory map Virt2: 00000000050d867e
[    5.555157] mwifiex_pcie 0000:01:00.0: info: FW download over, size 439256 bytes
[    6.313354] mwifiex_pcie 0000:01:00.0: WLAN FW is active
[    6.349480] mwifiex_pcie 0000:01:00.0: Unknown api_id: 5
[    6.359375] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84)
[    6.359393] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.92.21.p84)
Steps to reproduce on NXP EVK: Set iperf3 test via 2.4G WiFi router AP & Bluetooth alsa play wav file with Bluetooth speaker in the same time.

WiFi iperf test:
iperf3 –c 133.33.33.13 –t 86400
Audio play via bluealsa
aplay –D bluealsa:< BT MAC> xxx.wav
Observation: AW-CM276NF no response.
Fail Rate: 70% -> 1time/8hrs
Additional dmesg/logcat/OTA sniffer logs, FW coredump, RF tester logs…

dmesg with FW coredump:
[30166.238363] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa4, act = 0x0
[30166.247313] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
[30166.253280] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
[30166.259137] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
[30166.264644] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
[30166.270067] mwifiex_pcie 0000:01:00.0: last_cmd_index = 1
[30166.275480] mwifiex_pcie 0000:01:00.0: last_cmd_id: 16 00 a4 00 16 00 a4 00 7f 00
[30166.282984] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[30166.290579] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 0
[30166.296434] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 16 80 7f 80 16 80 a4 80 7f 80
[30166.304366] mwifiex_pcie 0000:01:00.0: last_event_index = 2
[30166.309956] mwifiex_pcie 0000:01:00.0: last_event: 0b 00 0a 00 0b 00 0b 00 0a 00
[30166.317370] mwifiex_pcie 0000:01:00.0: data_sent=1 cmd_sent=1
[30166.323130] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
[30166.328788] mwifiex_pcie 0000:01:00.0: failed to get signal information
[30166.336084] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
[30166.337333] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[30166.343008] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84)[30166.343033] mwifiex_pcie 0000:01:00.0: PCIE register dump start
[30166.349540] mwifiex_pcie 0000:01:00.0: failed to get signal information
[30166.357569] mwifiex_pcie 0000:01:00.0: pcie scratch register:
[30166.375967] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
reg:0xcf8, value=0x4bb009e
reg:0xcfc, value=0xde00

any suggestion is welcome.
Thanks.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Tue, 21 Nov 2023 10:00:46 GMT

Hi, @yao_feng

Thanks for creating case to us.

It seems there are FW dump issue happened. Can you please help to provide full logs(dmesg logs, FW_dump, Driver_dump and time synced sniffer if possible) for further debug this issue?

How to capture fw_dump and driver dump?

1. load driver with drvdbg=0xa0037

issue happened

logs will automatically save in the directory like below(You can check the dmesg log)

========================================

[ 4221.173141] Create directory /data/dump_4220 error, try create dir in /var

[ 4221.173147] Wakeup device...

[ 4221.173231] Create directory /var/dump_4220 successfully

[ 4221.173232] Directory name is /var/dump_4220

[ 4221.173233] === START DRIVER INFO DUMP===

[ 4221.173289] DRV dump data in /var/dump_4220/file_drv_info

[ 4221.173306] Wakeup device...

[ 4221.173889] SDIO Func0 (0x0-0x9): 43 03 06 00 07 00 00 02 03 00

[ 4221.175173] SDIO Func1 (0x4-0xb): 00 00 00 00 ff ff ff ff

[ 4221.175879] SDIO Func1: (0x4c) 00 (0x50) 08 (0x54) 07 (0x55) 0c (0x58) 10 (0x59) 00 (0x5c) 00 (0x5d) 00

[ 4221.176284] SDIO Func1 (0xc0-0xca): dc fe 96 00 31 11 00 37 37 02 20

[ 4221.177301] |

[ 4221.275861] SDIO Func1 (0xc0-0xca): dc fe a1 00 3b 00 00 37 37 02 20

[ 4221.276501] Drv info total bytes = 263150 (0x403ee)

[ 4221.276503] === DRIVER INFO DUMP END===Wakeup device...

[ 4221.276555] ==== DEBUG MODE OUTPUT START: 4220.878602 ===

====================================================

2. If you check it doesn't print the directory in dmesg logs,

3.please get it in below way:

- echo debug_dump > /proc/mwlan/adapter0/config

- Wait for some time (1 or 2 min)

- cat /proc/mwlan/adapter0/fw_dump > file_fw_dump

4.drv_dump:

cat /proc/mwlan/adapter0/drv_dump > file_drv_dump

5.Sniffer log：

Wireshark with Wi-Fi adapter is ok.

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

yao_feng — Wed, 22 Nov 2023 02:19:44 GMT

Hi Christine

Thanks for help, will reproduce to get dump file.

In the other hand, could you please help to analyze below hex command error meaning and any register table doc. can be shared for reference. first

[30166.275480] mwifiex_pcie 0000:01:00.0: last_cmd_id: 16 00 a4 00 16 00 a4 00 7f 00

[30166.282984] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 00 00 00 00 00 00 00 00

[30166.290579] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 0

[30166.296434] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 16 80 7f 80 16 80 a4 80 7f 80

[30166.304366] mwifiex_pcie 0000:01:00.0: last_event_index = 2

[30166.309956] mwifiex_pcie 0000:01:00.0: last_event: 0b 00 0a 00 0b 00 0b 00 0a 00

[30166.317370] mwifiex_pcie 0000:01:00.0: data_sent=1 cmd_sent=1

[30166.323130] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0

[30166.357569] mwifiex_pcie 0000:01:00.0: pcie scratch register:

[30166.375967] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00

reg:0xcf8, value=0x4bb009e

reg:0xcfc, value=0xde00

Thanks.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Wed, 22 Nov 2023 07:33:53 GMT

Hi, @yao_feng

Sorry, I could not explain these without our internal expert's help. These information is encapsulated in our driver and FW. I don't have permission to check. It seems like some command actions and response between Wi-Fi Card and SOC through PCIE.

According to our corporation experience, our internal expert will ask us to provide fw_dump firstly once receive any cases related to firmware crash.

Apologize again for any inconvenience to you.

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Tue, 28 Nov 2023 01:31:15 GMT

Hi, @yao_feng

Is there any updates on logs of this issue?

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

yao_feng — Tue, 28 Nov 2023 02:01:52 GMT

Hi Christine

Customer is trying to reproduce with drvdbg level setting.
Will keep posting.

Thanks.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Thu, 30 Nov 2023 02:15:12 GMT

Hello @Christine_Li ,

Here is the consumer from @yao_feng

The logs generated from driver, mxm_wifiex with drvdbg=0xa0037 setting are as attached.

Please help to fix issues.

Thank you

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Fri, 01 Dec 2023 02:15:27 GMT

Hello @Christine_Li ,

Here are issue occured & dump logs in runtime.

In runtime, there is no data in fw_dump buffer.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Fri, 01 Dec 2023 02:45:32 GMT

Hello @Christine_Li ,

Issue occured in usage about Wi-Fi 2.4G using iperf3 & Bluetooth baud=3000000 A2DP profile playing audio continuously.

Wi-Fi 5G is stable.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

yao_feng — Fri, 01 Dec 2023 03:32:28 GMT

Hi @Christine_Li
Customer attached related logs, please help for it.

Thanks a lot.
Yao

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Fri, 01 Dec 2023 09:23:32 GMT

Hello @Christine_Li ,

The issue occured & fw_dump is as attached.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Mon, 04 Dec 2023 04:22:18 GMT

Hi, @ArthurC

Thanks for providing us requested logs.

And sorry for my late reply because we were in new product training last week.

I have check your logs and forward to our internal expert team if necessary.

Once have any updates, I will keep you posted.

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Thu, 07 Dec 2023 09:02:38 GMT

Hi, @yao_feng

1. Can you please help to confirm with customer that current provided logs are captured with drvdbg=0xa0037? Because I didn't find this parameter in dmesg logs.

=================

[ 10.671399] mlan: loading out-of-tree module taints kernel.
[ 10.702840] wlan: Loading MWLAN driver
[ 10.706976] wlan: Register to Bus Driver...
[ 10.713553] wlan_pcie 0000:01:00.0: enabling device (0000 -> 0002)
[ 10.719900] Attach moal handle ops, card interface type: 0x204
[ 10.727185] PCIE8997: init module param from usr cfg
[ 10.732631] card_type: PCIE8997, config block: 0
[ 10.737712] cfg80211_wext=0xf
[ 10.741010] max_vir_bss=1
[ 10.744200] cal_data_cfg=none
[ 10.747270] ps_mode = 1
[ 10.749314] RTL8211F Gigabit Ethernet 30be0000.ethernet-1:00: attached PHY driver (mii_bus:phy_addr=30be0000.ethernet-1:00, irq=POLL)
[ 10.749942] auto_ds = 1
[ 10.764239] host_mlme=enable
[ 10.767152] fw_name=nxp/pcie8997_wlan_v4.bin
[ 10.771462] sta_name=wlan
[ 10.774122] drv_mode = 1
[ 10.776686] rx_work=1 cpu_num=4
[ 10.779912] Attach mlan adapter operations.card_type is 0x204.
[ 10.791376] Request firmware: nxp/pcie8997_wlan_v4.bin
[ 11.148733] FW download over, size 439256 bytes
[ 11.841311] WLAN FW is active
[ 11.844356] on_time is 11843131148
[ 11.875053] fw_cap_info=0x587c7fa3, dev_cap_mask=0xffffffff
[ 11.880676] max_p2p_conn = 8, max_sta_conn = 8
[ 11.893907] Register NXP 802.11 Adapter wlan0
[ 11.898372] wlan: version = PCIE8997--16.92.21.p84.4-MM5X16366.p5-GPL-(FP92)
[ 11.906138] wlan: Register to Bus Driver Done
[ 11.910569] wlan: Driver loaded successfully

===============

Can you please share inline supplicant logs and sniffers with the console logs?
Can you please share driver load parameters?

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

yao_feng — Thu, 07 Dec 2023 09:18:58 GMT

Hi @ArthurC

Could you please help to provide information NXP ask below:

Can you please share inline supplicant logs and sniffers with the console logs?
Can you please share driver load parameters? (driver load steps? where you put drvdbg parameters?)

Thanks.
Yao

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Fri, 08 Dec 2023 01:45:05 GMT

Hello @Christine_Li,

1. Can you please help to confirm with customer that current provided logs are captured with drvdbg=0xa0037? Because I didn't find this parameter in dmesg logs.

The log generated by driver probe with drvdbg=0xa0037 is "wifi_crash_dmesg.log" only.

"file_fw_dump" file is named fw_dump.

Please found the post with tag drvdbg=0xa0037

Can you please share inline supplicant logs and sniffers with the console logs?

We are using NetworkManager for wifi networking, We didn't get any sniffers.

Can you please share driver load parameters?

modprobe moal mod_para=nxp/wifi_mod_para.conf drvdbg=0xa0037

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Fri, 08 Dec 2023 03:41:58 GMT

Hi, @ArthurC

Thanks for your feedback.

Here is our initial analysis from the shared logs.

From wifi_faild_dmsg.log:
- From the dmesg logs deauthentication received and after that scanning continuously for STA.
- ERR: Role is AP occurred at At the 8283.119064 and IOCTL failure observed at 8334.008714
From wifi_error_dmesg.log:
- From the dmesg logs kernel crash is observed. so it resulted the wakeup card timeout.
From wifi_crash_dmesg.log:
- From the dmesg logs failure in ADDBA response observed. After that manualy dump has taken. There is no any kernel crash observed.

From all the logs we are not able to relate them with each other.

So we need in-sync dmesg logs and sniffer with console logs from the startup of the device to narrow down the issue.

1. Can you please also share me the 88W8997 load parameters in lib/firmware/nxp/wifi_mod_para.conf?

2. Yes, I saw you didn't provide us sniffer logs.

Then can you please help to try to reproduce and capture Sniffer logs + fw_dump + console logs? Please help to capture dmesg/console logs from the startup of the device and load Wi-Fi driver(with drvdbg=0xa0037) so that we can understand the loading parameters you used.

Such as the wifi_error_dmesg.log, we can see what parameters you used in the logs, and it is clear that without with drvdbg=0xa0037 parameter.
============

===============

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Fri, 08 Dec 2023 09:33:31 GMT

Hello Yao,

Responsed.

BTW, we found that issue is occured in 88W8997 Wi-Fi with 2.4G connected to an external Wi-Fi AP.

Setting 88W8997 Wi-Fi as host AP to share network and play Bluetooth Audio in the same time is ok.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Mon, 11 Dec 2023 04:02:41 GMT

HI, @ArthurC

Thanks for providing us more information.

But as our comments before, we can not catch up loading Wi-Fi driver parameters in the given logs. And also the given 3 logs show different phenomenon and issue. For narrow down the issue root cause, can you please help to provide one time clean logs with drvdbg=0xa0037?

Time synced dmesg logs + console logs + fw_dump + sniffer(if possible) from the startup of the device., including loading driver process.

so that we can see detailed loading driver parameters you are using and get more clear information in logs.

If right now, Sniffer logs are difficult for you to provide, then please help to provide other logs firstly.

We only need one-time clean log.

And please make sure the logs are from device booting up and make sure including loading driver process.

Sorry for any inconveniences.

Best regards,

Christine.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

ArthurC — Tue, 12 Dec 2023 03:14:52 GMT

Hello @Christine_Li

It's hard to save mass logs produced by kernel.

In our platform, we don't have enough buffer to save mass logs when we probe with drvdbg=0xa0037.

Why is it necessary to provide?
What do experts concern?

Please help to explan the analysis result based on the provided logs.

Re: 88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

Christine_Li — Tue, 12 Dec 2023 03:21:42 GMT

Hi, @ArthurC

Please help to explain the analysis result based on the provided logs.

==>We have shared our initial analysis results of the provided logs, please see above comments.

As per our discussion with our internal team, earlier shared 3 set of logs are having multiple discrepancies. Also in the previous logs, we could not see loading Wi-Fi driver parameters as mentioned before.

So please help to share below mentioned in-sync logs with single iteration. It will help us to narrow down the issue.

dmesg logs from the driver load
Console logs to check what commands customer is using
Sniffer logs
Dumps

Best regards,

Christine.