88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ) - reopen

yao_feng · ‎03-11-2024

The FW crashed issue is still exist in Generic_PCIE-WLAN-UART-BT-8997-LNX_6_6_3-IMX8-16.92.21.p119.2-16.92.21.p119.2-MM6X16437.P3-GPL.

follow previous case:

https://community.nxp.com/t5/Wireless-Connectivity/88W8997-module-command-timeout-issue-interface-PC...

and reopen by this case.

@ArthurC
@Christine_Li
88W8997

ArthurC

Hello @Christine_Li,

About feature support in our product, there is wifi network sharing usage.

If we configured to be STA only, the drv_mode=1, how can we switch the working mode to be AP in runtime?

Is there any design support about working mode between STA & AP mode in runtime?

If not, is there any FW adjustment for this issue?

Christine_Li

Hi, @ArthurC

From detailed analysis of your shared logs and local test observation, we found there are two instances of wpa_supplicant are created by nmcli for uap0 and wlan0/mlan0. As the issue is related to scan timeout and normally nmcli do very aggressive scan for both the interfaces, we are suspecting this is culprit.

As your use-case is only STA mode, we tried to run the STA only scenario by using drv_mode=1 for more than ~2 days and the issue is not observed.

Can you please tried out this solution at your end and share the feedback if you still facing the issue or not?

Please also take care below points before start the test.

Driver load parameters should be as below.

cfg80211_wext=0xf max_vir_bss=1 cal_data_cfg=none ps_mode=1 auto_ds=1 host_mlme=1 drv_mode=1 auto_fw_reload=0 drvdbg=0xa0037

If issue seen, please share FW dumps file, which should be auto generate and dump file path will be notified in the dmesg by the driver.

If auto dump not generated, please follow below steps to get dump manually.

echo "debug_dump" >/proc/mwlan/adapter0/config

cat /proc/mwlan/adapter0/drv_dump > file_drv_dump

cat /proc/mwlan/adapter0/fw_dump > file_fw_dump

Regards,

Christine.

Christine_Li

Hi, @ArthurC

Thank you for the confirmation.

We actively debugging to identifying the suspect of the issue.

As the "cmd 0x107 error" issue is locally reproduced, We are trying to capture the dump files to help the engineering team to identify the suspect of the issue.

We will update you soon.

Best regards,

Christine.

ArthurC

Hello @Christine_Li,

Yes, device 88w8997 is hang/crashed should be the root cause about the "cmd 0x107 error".

Please help to fix this issue.

Thank you.

Christine_Li

Hi, @ArthurC

Thank you for the logs.

We have analyzed the received both logs.

We can observe the "cmd 0x107 error" in both the dmesg logs.

From journalctl logs we could also observe the "Disconnection" but this might be because of the device is hang/crash.

Regarding local reproduction,

- We are also seen the similar customer reported issue "cmd 0x107 error" with your given nmcli.

Here main concern is the "cmd 0x107 erro" the disconnection is not a culprit because this might be because of the device will hang/crashed.

From the older you shared logs we could observe the disassociation as below.

[ 165.306659] wlan: Received disassociation request on wlan0, reason: 3

[ 165.313144] wlan: REASON: (Deauth) Sending STA is leaving (or has left) IBSS or ESS

[ 165.320869] Deauth: bc:XX:XX:XX:03:1a

Now, In the latest logs we could not observe the above disassociation.

Please understand this behavior and if you agree then we will move forward to debug the "cmd 0x107 erro" issue.

Best regards,

Christine.

Christine_Li

Hi, @ArthurC

Thanks for your reply.

Is there any test environment changes?

It seems this time the test can continue to more than 20 hours. As I remember, before the Wi-Fi disconnection issue will occur.

But right now, it seems that system hang occurs firstly, but Wi-Fi disconnection doesn't happen. Right?

Anyway, we will analyze the given logs and update to you our findings.

Best regards,

Christine.

ArthurC

Hello @Christine_Li,

The setup with drvdbg=0x80037 after 74065 seconds is caused system hang, too.

The logs before system hang is as attached.

The final screen capture by camera about dmesg window is as following.

Christine_Li

Hi, @ArthurC

Thank you so much for your feedback.

We will analyze current given logs and reply to you ASAP.

Best regards,

Christine.

ArthurC

Hello @Christine_Li,

The dmesg & Journalctrl log with "ExecStart=/usr/sbin/wpa_supplicant -u -ddd" in /lib/systemd/system/wpa_supplicant.service & drvdbg=0xa0037 is as attached.

The system hang when issue occured.

The setup with drvdbg=0x80037 is on testing.

If issue occured, will provided needed log ASAP.

Christine_Li

Hi, @ArthurC

We have run the setup for more than 2 days using nmcli but we are not observed the "disassociation" issue which you have reported.

So, It looks like the issue is only observed at your environment.

For further debugging we need at least one set of debug logs of the wpa_supplicant because the nmcli is using the wpa_supplicant in background,

You can use below steps to enable the supplicant logs:

1. boot up the board

2. killall wpa_supplicant

3. open "vi /lib/systemd/system/wpa_supplicant.service"

4. add -ddd at end "ExecStart=/usr/sbin/wpa_supplicant -u -ddd"

5. reboot the board

6. Run the scenario using nmcli

7. run journalctl -a -b to capture the supplicant logs

Please use drvdbg=0xa0037 while running the scenario.

If you use drvdbg=0xa0037, device will hang, then you can also use the drvdbg=0x80037.

But make sure to take the dump manually using below commands after the crash is observed.

echo "debug_dump" >/proc/mwlan/adapter0/config
cat /proc/mwlan/adapter0/drv_dump > file_drv_dump
cat /proc/mwlan/adapter0/fw_dump > file_fw_dump

Share the journalctl/console and dmesg logs for further debugging.

Kindly help to get the requested logs to unblock the debugging.

Please note: We need this logs ASAP to conclude the issue, If you are not supporting then we could not support this case further. Because we must need the supplicant logs to further narrow down the issue.

Best Regards,

Christine.

Christine_Li

Hi, @ArthurC

Thanks for your quick reply.

Now I started the test with nmcli and doing iperf under 2.4G and playing Music now.

And also our AE team has also started the same test.

Hope we can reproduce locally and resolve it ASAP. Until now we tested more than 12 mins, and not reproduced yet.

Thank you for your strong help until now.

Best regards,

Christine.

ArthurC

Hello @Christine_Li ,

The libnewt is as attached.

Please ignore local pcakages except CN, TW, HK & EN.

Thank you

Christine_Li

Hi, @ArthurC

Thanks for your quick reply.

I met this lib dependency (libnewt0.52), please help also to provide.

dpkg: dependency problems prevent configuration of networkmanager-nmtui:
 networkmanager-nmtui depends on libnewt0.52 (>= 0.52.21); however:
  Package libnewt0.52 is not installed.

dpkg: error processing package networkmanager-nmtui (--install):
 dependency problems - leaving unconfigured
Setting up networkmanager-src (1.36.2-r0) ...
Setting up networkmanager-wifi (1.36.2-r0) ...

Thanks

Christine.

ArthurC

Hello @Christine_Li,

Yes, in very first, we install the packages by ourselve including the dependencies.

If there is any other packages missed and needed to install.

Please let me know.

Thank you

Christine_Li

Hi, @ArthurC

Thanks for your reply.

Are you using the prebuilt Image downloaded from our website or from the yocto you build by yourself?

Because when I try to install these networkmanager deb file, it reminds me conflict with connman.

And after I remove connman, there are still some dependency libs are needed:

root@imx8mqevk:~# dpkg --install networkmanager_1.36.2-r0_arm64.deb
(Reading database ... 65783 files and directories currently installed.)
Preparing to unpack networkmanager_1.36.2-r0_arm64.deb ...
Unpacking networkmanager (1.36.2-r0) ...
dpkg: dependency problems prevent configuration of networkmanager:
 networkmanager depends on libndp (>= 1.8); however:
  Package libndp is not installed.
 networkmanager depends on nspr (>= 4.29); however:
  Package nspr is not installed.
 networkmanager depends on nss (>= 3.74); however:
  Package nss is not installed.

dpkg: error processing package networkmanager (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 networkmanager

Have you installed these dependency libs manually before install networkmanager debs?

Best regards,

Christine.

ArthurC

Hello @Christine_Li,

Please install following packages to make feature workable:

networkmanager_1.36.2-r0_arm64.deb
networkmanager-dev_1.36.2-r0_arm64.deb
networkmanager-nmcli_1.36.2-r0_arm64.deb
networkmanager-nmtui_1.36.2-r0_arm64.deb
networkmanager-wifi_1.36.2-r0_arm64.deb
networkmanager-src_1.36.2-r0_arm64.deb
networkmanager-dbg_1.36.2-r0_arm64.deb
networkmanager-bash-completion_1.36.2-r0_arm64.deb
networkmanager-locale-zh-cn_1.36.2-r0_arm64.deb
networkmanager-locale-zh-hk_1.36.2-r0_arm64.deb
networkmanager-locale-zh-tw_1.36.2-r0_arm64.deb
networkmanager-locale-en-ca_1.36.2-r0_arm64.deb
networkmanager-locale-en-gb_1.36.2-r0_arm64.deb
networkmanager-doc_1.36.2-r0_arm64.deb

Thank you

Christine_Li

Hi, @ArthurC

Can you please help to provide the exact .deb file name which you used?

Best regards,

Christine.

ArthurC

Hello @Christine_Li,

Please use the version, Linux 5.15.71_2.2.0.

Thank you

Christine_Li

Hi, @ArthurC

Thanks for your quick reply.

We have Linux 5.15.71_2.2.2, Linux 5.15.71_2.2.1 and Linux 5.15.71_2.2.0 prebuilt image versions on our website.

Please share the exact version of the 8MQ pre-built image that you have flashed to avoid any discrepancy.

Thanks,

Christine.

ArthurC

Hello @Christine_Li,

Please use command "dkpg install" to install the network manager package in official prebuild image.

And then nmcli will be working.

Please use the prebuilt image "imx-image-full-imx8mqevk.wic" to test.

Thank you.

Christine_Li

Hi， @ArthurC

We checked the zip file shared by the customer, it only contains .deb files, we need an executable nmcli binary which we can directly copy to 8MQ and use. Can you please share it to us?

Also, We have Linux 5.15.71_2.2.2, Linux 5.15.71_2.2.1 and Linux 5.15.71_2.2.0 prebuilt image versions on our website. Please share the exact version of the 8MQ pre-built image that you have flashed to avoid any discrepancy.