Linux Kernel crashes on reboot - NXP W9098 mwifiex driver, imx-firmware - Fail download FW

[mlueer]

Hello together,

I have a problem with the NXP PCIE 9098 module. I compiled the driver (https://github.com/nxp-imx/mwifiex/tree/lf-5.15.52_2.1.0, lf-5.15.52_2.1.0) + firmware (https://github.com/nxp-imx/imx-firmware/tree/lf-5.15.52_2.1.0, lf-5.15.52_2.1.0) into the kernel (version 5.15.39). The driver works fine: the driver creates linux wireless interfaces as expected which I can use normally. But if I try to reboot ($ reboot OR $ reboot -f) without interrupting the power cycle of the board, the driver/kernel/system crashes on boot with the following errors:

Fail to read DMA CFG register.
woal_request_fw_failed
Firmware Init Failed
...
woal_pcie_probe: failed
wlan_pcie: probe of 0000:01:00.0 failed with error -14
...
Request firmware: nxp/pcieuart9098_combo_v1.bin
WLAN read winner status failed!
Read host int status register failed
wlan_dnld_fw fail ret=0xffffffff
WLAN: Fail download FW with nowwait: 0

Does anybody know how to fix this? Is this a problem with the firmware?

Thanks in advance

Max

[DanielRuvalcaba]

Hi,

If I'm understanding correctly, you are able to download the FW and use the Wi-Fi and BT. Once all is working, if you use the reboot command and then you try to download the FW again, you get this error. Am I correct?

Could you please help me with the following information?

1. What Host are you using?

2. Are you using a module?

Regards,

Daniel.

[mlueer]

Hi Daniel,
thank you for your fast response. Yes I think you got it right. I am using Debian as Linux and compiled my own kernel where I added the mwifiex driver to the kernel driver sources.
The module that I use is the M2-JODY-W3 card from ublox.
Regards,
Max

[DanielRuvalcaba]

Hi,

Thanks for the information.

I tried this on my side using an i.MX 8MQuad EVK (Linux 5.15.52) and the AW-XM458 (88W9098) module and execute the same steps as you; download the FW > reboot > download the FW again and I didn’t notice any problem while downloading the Wi-Fi firmware after the reboot.

Please let me know if you want me to try it in another way so I can try to replicate your issue.

Regards,

Daniel.

[mlueer]

Okay, I think I did not explain my procedure well enough.
My goal is to use the W9098 PCIE wireless card out of the box, without dynamically inserting the module at runtime to the kernel. I want to compile it into the kernel such that it is loaded in kernel boot process.
Therefore, I copied the firmware to the kernel sources into linux/firmware/nxp/pcieuart9098_combo_v1.bin (and modified the additional firmware sources in kernel config) and copied the mwifiex driver sources to linux/drivers/net/wireless/nxp/mwifiex with a self-written kernel-appropriate Makefile (see attachement). Then I compiled the kernel for a custom x86 computer which has the ublox Jody-W3 (NXP 88W9098 PCIE) connected via PCIE. The kernel compilation works fine and after copying the kernel image to the x86 computer and booting it, the module works as expected: It creates 2x3 wireless interfaces (2xuAP,2xWFD,2xSTA). Until here, everything works fine. If I now reboot the computer via "$ reboot" command, at the next boot, the kernel crashes with the following output:

Fail to read DMA CFG register.
woal_request_fw_failed
Firmware Init Failed
...
woal_pcie_probe: failed
wlan_pcie: probe of 0000:01:00.0 failed with error -14
...
Request firmware: nxp/pcieuart9098_combo_v1.bin
WLAN read winner status failed!
Read host int status register failed
wlan_dnld_fw fail ret=0xffffffff
WLAN: Fail download FW with nowwait: 0

In another case, when I remove the power from the computer and boot again, everything works fine as before. But rebooting without power cycle brings me to that error and the kernel hangs up.
This may be a problem with the driver->pcie card communication when the firmware from the last boot is still running on the pcie card. 

I hope this gets a better insight into my problem.
Thank you in advance!

[DanielRuvalcaba]

Hi,

Thank you for the clarification.

Could you please try to unload the drivers before the reboot command and then load them again to see if the problem still occurs?

Regards,

Daniel.

[mlueer]

Hi Daniel,

I now changed my setup a bit. I took a amd64 computer and installed Debian 12 ($uname -a: Linux debian-blue 6.1.0-11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux). Then I compiled the driver mwifiex from tag lf-6.1.1-1.0.0 and the appropriate firmware from imx-firmware repository.
Loading the modules mlan and moal for the first time works fine: 

[ 52.796739] wlan: Loading MWLAN driver

[ 52.800928] wlan: Register to Bus Driver...

[ 52.805755] PCI memory map Virt0: 00000000abbc52fb PCI memory map Virt2: 00000000a8a4694a

[ 52.814051] Attach moal handle ops, card interface type: 0x206

[ 52.819968] rps set to 0 from module param

[ 52.824101] No module param cfg file specified

[ 52.828580] magic=0x24, strap=0x3, revision_id=0x3

[ 52.833408] rx_work=1 cpu_num=4

[ 52.841759] Request firmware: nxp/pcieuart9098_combo_v1.bin

[ 53.413448] WLAN FW is active

[ 53.416451] on_time is 53415087645

[ 53.437574] Register NXP 802.11 Adapter mlan0

[ 53.442261] wlan: uap%d set max_mtu 2000

[ 53.451664] Register NXP 802.11 Adapter uap0

[ 53.461735] Register NXP 802.11 Adapter wfd0

[ 53.466637] wlan: version = PCIE9098--17.92.1.p136.13-MM5X17368.p2-GPL-(FP92)

[ 53.475001] PCI memory map Virt0: 000000003ac25f63 PCI memory map Virt2: 000000004415b7eb

[ 53.477686] gpio-keys-polled gpio-keys-polled: unable to claim gpio 0, err=-517

[ 53.483287] Attach moal handle ops, card interface type: 0x206

[ 53.496551] rps set to 0 from module param

[ 53.500732] No module param cfg file specified

[ 53.505274] rx_work=1 cpu_num=4

[ 53.513803] Request firmware: nxp/pcieuart9098_combo_v1.bin

[ 53.522563] WLAN FW is active

[ 53.525596] on_time is 53524230649

[ 53.545689] Register NXP 802.11 Adapter mmlan0

[ 53.550682] wlan: muap%d set max_mtu 2000

[ 53.575738] Register NXP 802.11 Adapter muap0

[ 53.587393] Register NXP 802.11 Adapter mwfd0

[ 53.591937] wlan: version = PCIE9098--17.92.1.p136.13-MM5X17368.p2-GPL-(FP92)

[ 53.600057] wlan: Register to Bus Driver Done

[ 53.600130] gpio-keys-polled gpio-keys-polled: unable to claim gpio 0, err=-517

[ 53.604511] wlan: Driver loaded successfully

Unloading works as well:

[ 56.360441] wlan: Unloading MWLAN driver

[ 56.716467] Free module params

[ 56.856321] Free module params

[ 56.859884] wlan: Driver unloaded

When I reboot the computer ($ reboot) without unloading the driver before and try to load it after reboot, the wifi module does not work and I get the following output:

[ 52.911159] wlan: Loading MWLAN driver

[ 52.915370] wlan: Register to Bus Driver...

[ 52.920156] PCI memory map Virt0: 00000000ebc6088e PCI memory map Virt2: 000000002c490b12

[ 52.928426] Attach moal handle ops, card interface type: 0x206

[ 52.934305] rps set to 0 from module param

[ 52.938429] No module param cfg file specified

[ 52.942942] magic=0x24, strap=0x3, revision_id=0x3

[ 52.947790] rx_work=1 cpu_num=4

[ 52.951029] Attach mlan adapter operations.card_type is 0x206.

[ 52.961872] Request firmware: nxp/pcieuart9098_combo_v1.bin

[ 52.978455] wlan_pcie 0000:01:00.0: firmware: direct-loading firmware nxp/pcieuart9098_combo_v1.bin

[ 53.550324] prog_fw: Failed to get command done interrupt retry count = 10001

[ 55.557604] FW failed to be active in time!

[ 55.561841] wlan_dnld_fw fail ret=0xffffffff

[ 55.566164] WLAN: Fail download FW with nowwait: 0

[ 55.571016] Config Space Registers:

[ 55.574560] reg:0x00 value=0x2b431b4b

[ 55.578265] reg:0x04 value=0x00100406

[ 55.582000] reg:0x10 value=0xfe20000c

[ 55.585716] reg:0x18 value=0xfe40000c

[ 55.589394] reg:0x2c value=0x2b431b4b

[ 55.593126] reg:0x3c value=0x00000100

[ 55.596836] reg:0x44 value=0x00000008

[ 55.600542] reg:0x80 value=0x10120040

[ 55.604255] reg:0x98 value=0x00000000

[ 55.607962] reg:0x170 value=0x0001001e

[ 55.611759] FW Scrach Registers:

[ 55.615039] reg:0x1c90 value=0xfedcba00

[ 55.618926] reg:0x1c98 value=0x0

[ 55.622201] reg:0x1c9c value=0x0

[ 55.724837] reg:0x1c98 value=0x0

[ 55.728089] reg:0x1c9c value=0x0

[ 55.830741] Interface registers dump from offset 0x1c98 to 0x1c9f

[ 55.836907] PCIE registers from offset 0x1c20 to 0x1c9c:

[ 55.842286] 00000000 00000002 0000004f 00000000 ffffffff 00000002 00000000 00000000 

[ 55.850126] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 55.857902] dc053000 00000000 00000010 00000000 008047e3 c8701424 00000000 00000000 

[ 55.865735] 00000000 00000000 00000000 00000000 fedcba00 00000000 00000000 00000000 

[ 55.873497] 

[ 55.875018] ADMA Tx/Rx/Event/Cmd/CmdResp registers:

[ 55.879917] ADMA registers dump from offset 0x10000 to 0x10068

[ 55.885840] 00100000 dc053000 00000000 c010ad64 00000000 00000000 00000000 00000000 

[ 55.893647] 00000000 00000001 00000000 00000000 00000000 00000000 00000000 0000f002 

[ 55.901441] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 55.909255] 80000948 00000000 00808c8c 

[ 55.913144] ADMA registers dump from offset 0x10800 to 0x10868

[ 55.919047] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 55.926885] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000f000 

[ 55.934765] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 55.942531] 80000948 00000000 00808c8c 

[ 55.946396] ADMA registers dump from offset 0x10880 to 0x108e8

[ 55.952296] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 55.960087] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 55.967879] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 55.975676] 80000948 00000000 00000000 

[ 55.979570] ADMA registers dump from offset 0x11000 to 0x11068

[ 55.985442] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 55.993222] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000f000 

[ 56.001015] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 56.008830] 80000948 00000000 00808c8c 

[ 56.012692] ADMA registers dump from offset 0x11080 to 0x110e8

[ 56.018592] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 56.026399] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 56.034173] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 56.041954] 80000948 00000000 00000000 

[ 56.046036] woal_request_fw failed

[ 56.049452] Firmware Init Failed

[ 56.055636] Free module params

[ 56.059147] woal_pcie_probe: failed

[ 56.062723] wlan_pcie: probe of 0000:01:00.0 failed with error -14

[ 56.069294] PCI memory map Virt0: 000000007e0c8309 PCI memory map Virt2: 0000000099d2d9ee

[ 56.077600] Attach moal handle ops, card interface type: 0x206

[ 56.083571] rps set to 0 from module param

[ 56.087723] No module param cfg file specified

[ 56.092228] rx_work=1 cpu_num=4

[ 56.095482] Attach mlan adapter operations.card_type is 0x206.

[ 56.106011] Request firmware: nxp/pcieuart9098_combo_v1.bin

[ 56.113804] wlan_pcie 0000:01:00.1: firmware: direct-loading firmware nxp/pcieuart9098_combo_v1.bin

[ 56.693143] prog_fw: Failed to get command done interrupt retry count = 10001

[ 58.697604] FW failed to be active in time!

[ 58.701832] wlan_dnld_fw fail ret=0xffffffff

[ 58.706159] WLAN: Fail download FW with nowwait: 0

[ 58.711006] Config Space Registers:

[ 58.714547] reg:0x00 value=0x2b441b4b

[ 58.718250] reg:0x04 value=0x00100406

[ 58.721962] reg:0x10 value=0xfe30000c

[ 58.725680] reg:0x18 value=0xfe500004

[ 58.729365] reg:0x2c value=0x2b441b4b

[ 58.733071] reg:0x3c value=0x00000200

[ 58.736780] reg:0x44 value=0x00000008

[ 58.740488] reg:0x80 value=0x10120040

[ 58.744198] reg:0x98 value=0x00000000

[ 58.747906] reg:0x170 value=0x0001001e

[ 58.751705] FW Scrach Registers:

[ 58.754974] reg:0x1c90 value=0xfedcba00

[ 58.758901] reg:0x1c98 value=0x0

[ 58.762177] reg:0x1c9c value=0x0

[ 58.864796] reg:0x1c98 value=0x0

[ 58.868052] reg:0x1c9c value=0x0

[ 58.970711] Interface registers dump from offset 0x1c98 to 0x1c9f

[ 58.976846] PCIE registers from offset 0x1c20 to 0x1c9c:

[ 58.982219] 00000000 00000002 0000004f 00000000 ffffffff 00000002 00000000 00000000 

[ 58.990057] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 58.997871] de03a000 00000000 00000010 00000000 008047e3 c8701424 00000000 00000000 

[ 59.005675] 00000000 00000000 00000000 00000000 fedcba00 00000000 00000000 00000000 

[ 59.013441] 

[ 59.014960] ADMA Tx/Rx/Event/Cmd/CmdResp registers:

[ 59.019863] ADMA registers dump from offset 0x10000 to 0x10068

[ 59.025737] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.033560] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000f000 

[ 59.041339] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 59.049122] 80000948 00000000 00808c8c 

[ 59.053014] ADMA registers dump from offset 0x10800 to 0x10868

[ 59.058918] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.066696] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000f000 

[ 59.074567] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 59.082358] 80000948 00000000 00808c8c 

[ 59.086220] ADMA registers dump from offset 0x10880 to 0x108e8

[ 59.092126] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.099935] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.107742] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 59.115527] 80000948 00000000 00000000 

[ 59.119390] ADMA registers dump from offset 0x11000 to 0x11068

[ 59.125295] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.133124] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000f000 

[ 59.140928] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 59.148696] 80000948 00000000 00808c8c 

[ 59.152553] ADMA registers dump from offset 0x11080 to 0x110e8

[ 59.158450] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.166265] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

[ 59.174039] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 

[ 59.181803] 80000948 00000000 00000000 

[ 59.185985] woal_request_fw failed

[ 59.189414] Firmware Init Failed

[ 59.195436] Free module params

[ 59.198926] woal_pcie_probe: failed

[ 59.202495] wlan_pcie: probe of 0000:01:00.1 failed with error -14

[ 59.208953] wlan: Register to Bus Driver Done

[ 59.213369] wlan: Driver loaded successfully

The only way to recover the wifi module is to remove the power from the computer.
To reproduce this error, you may have to use a amd64 computer, not the arm64.

Thank you in advance,
Max

[DanielRuvalcaba]

Hi,

Could you please make sure to unload the driver before a reset?

Regards,

Daniel.

[mlueer]

Hi Daniel,

thank you for your reply.
I have now build the driver as modules and loaded it dynamically via insmod mlan.ko and insmod moal.ko.
Everything works fine. Unloading via rmmod or force rmmod and load again via insmod also works fine. When I unload the driver and then reboot, at the next boot, I can load the driver again without problem. If I do not unload the module and trigger a reboot, the kernel crashes when loading the module via insmod with the same error message.

Regards,
Max