88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ)

[yao_feng]

Platform (Host CPU): i.MX8MQ
Wi-Fi module: AW-CM276NF (interface PCIE/UART)
OS: Linux

• BSP version: Yocto 5.15.71
• Wi-Fi and Bluetooth/Bluetooth LE Firmware version: 16.92.21.p84
• Driver version: Linux 5.15.71

1. HW or SW/FW issue?  FW issue
2. Environment: In shielding room or open site? open site
3. Issue description: FW is in bad state. PCIE command no response, timeout. failed to get signal information.
4. Setup: on board module.
   
   • Linux 5.15.71 Built-in driver,

   [    4.909207] mwifiex_pcie 0000:01:00.0: no quirks enabled
   [    4.909234] mwifiex_pcie 0000:01:00.0: enabling device (0000 -> 0002)
   [    4.909326] mwifiex_pcie: PCI memory map Virt0: 0000000024acaf60 PCI memory map Virt2: 00000000050d867e
   [    5.555157] mwifiex_pcie 0000:01:00.0: info: FW download over, size 439256 bytes
   [    6.313354] mwifiex_pcie 0000:01:00.0: WLAN FW is active
   [    6.349480] mwifiex_pcie 0000:01:00.0: Unknown api_id: 5
   [    6.359375] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84)
   [    6.359393] mwifiex_pcie 0000:01:00.0: driver_version = mwifiex 1.0 (16.92.21.p84)

5. Steps to reproduce on NXP EVK: Set iperf3 test via 2.4G WiFi router AP & Bluetooth alsa play wav file with Bluetooth speaker in the same time.  
   

   WiFi iperf test:
   iperf3 –c 133.33.33.13 –t 86400
   Audio play via bluealsa
   aplay –D bluealsa:< BT MAC> xxx.wav

6. Observation: AW-CM276NF no response.
7. Fail Rate: 70% -> 1time/8hrs
8. Additional dmesg/logcat/OTA sniffer logs, FW coredump, RF tester logs…

dmesg with FW coredump:
[30166.238363] mwifiex_pcie 0000:01:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa4, act = 0x0
[30166.247313] mwifiex_pcie 0000:01:00.0: num_data_h2c_failure = 0
[30166.253280] mwifiex_pcie 0000:01:00.0: num_cmd_h2c_failure = 0
[30166.259137] mwifiex_pcie 0000:01:00.0: is_cmd_timedout = 1
[30166.264644] mwifiex_pcie 0000:01:00.0: num_tx_timeout = 0
[30166.270067] mwifiex_pcie 0000:01:00.0: last_cmd_index = 1
[30166.275480] mwifiex_pcie 0000:01:00.0: last_cmd_id: 16 00 a4 00 16 00 a4 00 7f 00
[30166.282984] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[30166.290579] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 0
[30166.296434] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 16 80 7f 80 16 80 a4 80 7f 80
[30166.304366] mwifiex_pcie 0000:01:00.0: last_event_index = 2
[30166.309956] mwifiex_pcie 0000:01:00.0: last_event: 0b 00 0a 00 0b 00 0b 00 0a 00
[30166.317370] mwifiex_pcie 0000:01:00.0: data_sent=1 cmd_sent=1
[30166.323130] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0
[30166.328788] mwifiex_pcie 0000:01:00.0: failed to get signal information
[30166.336084] mwifiex_pcie 0000:01:00.0: ===mwifiex driverinfo dump start===
[30166.337333] mwifiex_pcie 0000:01:00.0: PREP_CMD: FW is in bad state
[30166.343008] mwifiex_pcie 0000:01:00.0: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84)[30166.343033] mwifiex_pcie 0000:01:00.0: PCIE register dump start
[30166.349540] mwifiex_pcie 0000:01:00.0: failed to get signal information
[30166.357569] mwifiex_pcie 0000:01:00.0: pcie scratch register:
[30166.375967] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00
reg:0xcf8, value=0x4bb009e
reg:0xcfc, value=0xde00

any suggestion is welcome.
Thanks.

[yao_feng]

Hi @ArthurC 

For debug purpose, is it possible load driver/FW by modprobe command refer to UM11490 as below:

modprobe moal mod_para=nxp/wifi_mod_para.conf drvdbg=0xa0037

and then catch dmesg logs from device boot with FW dump after issue occurred.
sorry for inconvenient.
Thanks.
Yao 

[ArthurC]

Hello @yao_feng,

Already did it!!
There are mass logs flushed our buffer & it's hard to capture.

Is there no expert to support FW issue in NXP?

Why don't response in the post?

[Christine_Li]

Hi, @ArthurC 

I have to apologize to you because of any inconvenience.

This case is already dealt with by our NXP experts, but right now they request the logs as mentioned before.

According to our company support policy, our expert team doesn't communicate or post directly on our community. I am here supporting you, for some questions or issues I can answer, I will directly answer. For some firmware issue or too difficult issues I could not resolve, I will create an internal case to our expert team to ask help from them. They will support us for sure.

I know requesting logs may cause some inconvenience to you, but with requested logs, we can narrow down the issue and speed up resolution process. Sincerely hope you can understand.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li 

The requested log about initialization is as attached.

It should be enough to narrow the issue cause.

For log about issue occured, please find logs provided before. 

[Christine_Li]

Hi， @ArthurC 

Thanks for providing us again logs.

I will forward to our internal expert team to check, and once have any updates, I will let you know.

Best regards,

Christine.

[yao_feng]

Hi @Christine_Li 
Customer attached related logs, please help for it.

Thanks a lot.
Yao

[yao_feng]

Hi Christine

Thanks for help, will reproduce to get dump file.

In the other hand, could you please help to analyze below hex command error meaning and any register table doc. can be shared for reference. first

[30166.275480] mwifiex_pcie 0000:01:00.0: last_cmd_id: 16 00 a4 00 16 00 a4 00 7f 00

[30166.282984] mwifiex_pcie 0000:01:00.0: last_cmd_act: 00 00 00 00 00 00 00 00 00 00

[30166.290579] mwifiex_pcie 0000:01:00.0: last_cmd_resp_index = 0

[30166.296434] mwifiex_pcie 0000:01:00.0: last_cmd_resp_id: 16 80 7f 80 16 80 a4 80 7f 80

[30166.304366] mwifiex_pcie 0000:01:00.0: last_event_index = 2

[30166.309956] mwifiex_pcie 0000:01:00.0: last_event: 0b 00 0a 00 0b 00 0b 00 0a 00

[30166.317370] mwifiex_pcie 0000:01:00.0: data_sent=1 cmd_sent=1

[30166.323130] mwifiex_pcie 0000:01:00.0: ps_mode=1 ps_state=0

[30166.357569] mwifiex_pcie 0000:01:00.0: pcie scratch register:

[30166.375967] mwifiex_pcie 0000:01:00.0: reg:0xcf0, value=0xfedcba00

               reg:0xcf8, value=0x4bb009e

               reg:0xcfc, value=0xde00

 Thanks.

[Christine_Li]

Hi, @yao_feng 

Sorry, I could not explain these without our internal expert's help. These information is encapsulated in our driver and FW. I don't have permission to check. It seems like some command actions and response between Wi-Fi Card and SOC through PCIE.

According to our corporation experience, our internal expert will ask us to provide fw_dump firstly once receive any cases related to firmware crash.

Apologize again for any inconvenience to you.

Best regards,

Christine.

[ArthurC]

Hello  @Christine_Li 

The log about start up with time is as attached.

It should be enough with previous logs to narrow issue cause.  

[Christine_Li]

Hi, @ArthurC 

We had a chance to work with our internal. We have prepared a test firmware to check on your platform. Can you please help to  test with shared firmware?

Name : w8997_combo.bin

md5sum: 9e38158b494468cdf53ba9239ca38d84

Name : w8997o.bin

md5sum: d932dbfc38fc89d904149025d337c6a0

Version : 16.92.21.p255

If the issue is still observed by the customer please request the customer to share below mentioned logs.

1. Dmesg
2. supplicant Logs with 0xa0037
3. Dump

I will send the test firmware to your email because here is public community. It is not good to post here.

Best regards,

Christine.

[ArthurC]

Hello Christine_Li,
FW Version 16.92.21.p255 is received.

Here is on testing with "w8997o.bin".

About FW changing, we found that looks like the wifi buffer is kept a const in 2.4G client & Bluetooth A2DP playing.

It's make sense. And looks more stable.

Will keep testing over 1 week.

Then responsed the result.

Thank you for your help.
And thanks a lot for NXP experts' effort.

[Christine_Li]

Hi， @ArthurC 

Thanks for your quickly reply.

It is our pleasure to service you.

And sure, please take your time to do stress test on the new FW. 

Please feel free to contact us, if still face any issue on the new FW. And also this is only a test FW for verification, if there is no problem, we will share you the official one to fix the issue you reported.

Best regards,

Christine.

[Christine_Li]

Hi, @ArthurC 

Is there any test results on the new test FW?

Best regards,

Christine.

[ArthurC]

Hello, @Christine_Li ,

Yes, there is.

After tested 7 days, the test result about w8997o.bin with FW Version 16.92.21.p255 is positive.

The issue is not occured anymore.

Please help to release the official format for our production.
Thank you.

[Christine_Li]

Hi, @ArthurC 

Can you please help to double confirm :

1. whether use same test method when you verify this issue? I mean, testing the coexist of Wi-Fi and Bluetooth?

2.Are you sure you used this FW:w8997o.bin, not w8997_combo.bin?

Because as I know, w8997_combo.bin is a Wi-Fi and Bluetooth combination firmware. But w8997o.bin is Wi-Fi only firmware.

The reason we provided these two firmware is we changed in Wi-Fi firmware so shared Wi-Fi only and combo. 

I feel confused if your Bluetooth can work well with w8997o.bin.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li ,

Can you please help to double confirm :

1. whether use same test method when you verify this issue? I mean, testing the coexist of Wi-Fi and Bluetooth?

-> Yes, we use the same method to verify.

2.Are you sure you used this FW:w8997o.bin, not w8997_combo.bin?

Because as I know, w8997_combo.bin is a Wi-Fi and Bluetooth combination firmware. But w8997o.bin is Wi-Fi only firmware.

The reason we provided these two firmware is we changed in Wi-Fi firmware so shared Wi-Fi only and combo.

I feel confused if your Bluetooth can work well with w8997o.bin.

->Yes, we use w8997o.bin.

Because we need to set Bluetooth baud rate in 3M in kernel driver in stable.

uartuart8997_bt_v4.bin is from https://github.com/nxp-imx/imx-firmware/blob/lf-6.1.1_1.0.0/nxp/FwImage_8997/uartuart8997_bt_v4.bin

So we use "uartuart8997_bt_v4.bin" + "w8997o.bin" to test.

Bluetooth feature is bring-uped by https://elixir.bootlin.com/linux/v5.15.71/source/drivers/bluetooth/hci_mrvl.c

Wi-Fi feature is bring-uped by https://elixir.bootlin.com/linux/v5.15.71/source/drivers/net/wireless/marvell/mwifiex

Ported them successful by myself.

BTW, if there is a "uartuart8997_bt_v4.bin" with version matching w8997o.bin which could be provided will be prefect.

[ArthurC]

Hi, @Christine_Li

Is there any progress for the new FW delivery?



Best regards,

Arthur.

[Christine_Li]

Hi, @ArthurC 

Our AE team is still checking internally with R&D, once have any updates I will let you know.

And for now, you can use previous test FW for daily debugging and verification to make your project proceed. But for production, please wait for our official release. I will push our internal team to give an ETA firstly.

Best regards,

Christine.

[Christine_Li]

Hi, @ArthurC 

I received our internal team's feedback as below:

This issue fix will be available in the official Mass Market Q1-24 release on March 24.

Please let me know if you have any other concerns.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li ,

How is the status about new firmware releasing?

During the test in thease days, we found that the test for firmware about Wi-Fi is stable. But the Bluetooth signal is getting weak in the working situration with 2.4G Wi-Fi iperf + A2DP Audio play. Will cause disconnection often and no error logs but can be recovered by re-connection.

Please help to confirm the bluetooth firmware releasing is suitable for the released Wi-Fi firmware changing.

Thank You