88W8997 module command timeout issue (interface PCIE+UART, Host: iMX8MQ) - reopen

[yao_feng]

The FW crashed issue is still exist in Generic_PCIE-WLAN-UART-BT-8997-LNX_6_6_3-IMX8-16.92.21.p119.2-16.92.21.p119.2-MM6X16437.P3-GPL.

follow previous case:

https://community.nxp.com/t5/Wireless-Connectivity/88W8997-module-command-timeout-issue-interface-PC...

and reopen by this case.  

@ArthurC
@Christine_Li
88W8997 

[ArthurC]

Hello @Christine_Li,

Yes, device 88w8997 is hang/crashed should be the root cause about the "cmd 0x107 error".

Please help to fix this issue.

Thank you.

[Christine_Li]

Hi, @ArthurC 

Thank you for the logs.

We have analyzed the received both logs.

We can observe the "cmd 0x107 error" in both the dmesg logs.

From journalctl logs we could also observe the "Disconnection" but this might be because of the device is hang/crash.

Regarding local reproduction,

- We are also seen the similar customer reported issue "cmd 0x107 error" with your given nmcli.

Here main concern is the "cmd 0x107 erro" the disconnection is not a culprit because this might be because of the device will hang/crashed.

From the older you shared logs we could observe the disassociation as below.

[ 165.306659] wlan: Received disassociation request on wlan0, reason: 3

[ 165.313144] wlan: REASON: (Deauth) Sending STA is leaving (or has left) IBSS or ESS

[ 165.320869] Deauth: bc:XX:XX:XX:03:1a

Now, In the latest logs we could not observe the above disassociation.

Please understand this behavior and if you agree then we will move forward to debug the "cmd 0x107 erro" issue.

Best regards,

Christine.

[Christine_Li]

Hi, @ArthurC 

Thanks for your reply.

Is there any test environment changes?

It seems this time the test can continue to more than 20 hours. As I remember, before the Wi-Fi disconnection issue will occur.

But right now, it seems that system hang occurs firstly, but Wi-Fi disconnection doesn't happen. Right?

Anyway, we will analyze the given logs and update to you our findings.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

The setup with drvdbg=0x80037 after 74065 seconds is caused system hang, too.

The logs before system hang is as attached.

The final screen capture by camera about dmesg window is as following.

[Christine_Li]

Hi, @ArthurC 

Thank you so much for your feedback.

We will analyze current given logs and reply to you ASAP.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

The dmesg & Journalctrl log with "ExecStart=/usr/sbin/wpa_supplicant -u -ddd" in /lib/systemd/system/wpa_supplicant.service &  drvdbg=0xa0037 is as attached.

The system hang when issue occured.

The setup with drvdbg=0x80037 is on testing.

If issue occured, will provided needed log ASAP.

[Christine_Li]

Hi, @ArthurC 

We have run the setup for more than 2 days using nmcli but we are not observed the "disassociation" issue which you have reported.

So, It looks like the issue is only observed at your environment.

For further debugging we need at least one set of debug logs of the wpa_supplicant because the nmcli is using the wpa_supplicant in background,

You can use below steps to enable the supplicant logs:

1. boot up the board

2. killall wpa_supplicant

3. open "vi /lib/systemd/system/wpa_supplicant.service" 

4. add -ddd at end "ExecStart=/usr/sbin/wpa_supplicant -u -ddd"

5. reboot the board

6. Run the scenario using nmcli

7. run journalctl -a -b to capture the supplicant logs

Please use drvdbg=0xa0037 while running the scenario.

If you use drvdbg=0xa0037, device will hang, then you can also use the drvdbg=0x80037.

But make sure to take the dump manually using below commands after the crash is observed.

• echo "debug_dump" >/proc/mwlan/adapter0/config
• cat /proc/mwlan/adapter0/drv_dump > file_drv_dump
• cat /proc/mwlan/adapter0/fw_dump > file_fw_dump

Share the journalctl/console and dmesg logs for further debugging.

Kindly help to get the requested logs to unblock the debugging.

Please note: We need this logs ASAP to conclude the issue, If you are not supporting then we could not support this case further. Because we must need the supplicant logs to further narrow down the issue.

Best Regards,

Christine.

[Christine_Li]

Hi, @ArthurC 

Thanks for your quick reply.

Now I started the test with nmcli and doing iperf under 2.4G and playing Music now.

And also our AE team has also started the same test.

Hope we can reproduce locally and resolve it ASAP. Until now we tested more than 12 mins, and not reproduced yet.

Thank you for your strong help until now.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li ,

The libnewt is as attached. 

Please ignore local pcakages except CN, TW, HK & EN.

Thank you

[Christine_Li]

Hi, @ArthurC 

Thanks for your quick reply.

I met this lib dependency (libnewt0.52), please help also to provide.

dpkg: dependency problems prevent configuration of networkmanager-nmtui:
 networkmanager-nmtui depends on libnewt0.52 (>= 0.52.21); however:
  Package libnewt0.52 is not installed.

dpkg: error processing package networkmanager-nmtui (--install):
 dependency problems - leaving unconfigured
Setting up networkmanager-src (1.36.2-r0) ...
Setting up networkmanager-wifi (1.36.2-r0) ...

 Thanks 

Christine.

[ArthurC]

Hello @Christine_Li,

Yes, in very first, we install the packages by ourselve including the dependencies.

If there is any other packages missed and needed to install.

Please let me know.

Thank you

[Christine_Li]

Hi, @ArthurC 

Thanks for your reply.

Are you using the prebuilt Image downloaded from our website or from the yocto you build by yourself?

Because when I try to install these networkmanager deb file, it reminds me conflict with connman.

And after I remove connman, there are  still some dependency libs are needed:

root@imx8mqevk:~# dpkg --install networkmanager_1.36.2-r0_arm64.deb
(Reading database ... 65783 files and directories currently installed.)
Preparing to unpack networkmanager_1.36.2-r0_arm64.deb ...
Unpacking networkmanager (1.36.2-r0) ...
dpkg: dependency problems prevent configuration of networkmanager:
 networkmanager depends on libndp (>= 1.8); however:
  Package libndp is not installed.
 networkmanager depends on nspr (>= 4.29); however:
  Package nspr is not installed.
 networkmanager depends on nss (>= 3.74); however:
  Package nss is not installed.

dpkg: error processing package networkmanager (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 networkmanager

Have you installed these dependency libs manually before install networkmanager debs? 

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

Please install following packages to make feature workable:

networkmanager_1.36.2-r0_arm64.deb
networkmanager-dev_1.36.2-r0_arm64.deb
networkmanager-nmcli_1.36.2-r0_arm64.deb
networkmanager-nmtui_1.36.2-r0_arm64.deb
networkmanager-wifi_1.36.2-r0_arm64.deb
networkmanager-src_1.36.2-r0_arm64.deb
networkmanager-dbg_1.36.2-r0_arm64.deb
networkmanager-bash-completion_1.36.2-r0_arm64.deb
networkmanager-locale-zh-cn_1.36.2-r0_arm64.deb
networkmanager-locale-zh-hk_1.36.2-r0_arm64.deb
networkmanager-locale-zh-tw_1.36.2-r0_arm64.deb
networkmanager-locale-en-ca_1.36.2-r0_arm64.deb
networkmanager-locale-en-gb_1.36.2-r0_arm64.deb
networkmanager-doc_1.36.2-r0_arm64.deb

Thank you

[Christine_Li]

Hi, @ArthurC 

Can you please help to provide the exact .deb file name which you used?

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

Please use the version, Linux 5.15.71_2.2.0.

Thank you

[Christine_Li]

Hi, @ArthurC 

Thanks for your quick reply.

We have Linux 5.15.71_2.2.2, Linux 5.15.71_2.2.1 and Linux 5.15.71_2.2.0 prebuilt image versions on our website.

Please share the exact version of the 8MQ pre-built image that you have flashed to avoid any discrepancy. 

Thanks,

Christine.

[ArthurC]

Hello @Christine_Li,

Please use command "dkpg install" to install the network manager package in official prebuild image.

And then nmcli will be working.

Please use the prebuilt image "imx-image-full-imx8mqevk.wic" to test.

Thank you.

[Christine_Li]

Hi， @ArthurC 

We checked the zip file shared by the customer, it only contains .deb files, we need an executable nmcli binary which we can directly copy to 8MQ and use. Can you please share it to us?

Also, We have Linux 5.15.71_2.2.2, Linux 5.15.71_2.2.1 and Linux 5.15.71_2.2.0 prebuilt image versions on our website. Please share the exact version of the 8MQ pre-built image that you have flashed to avoid any discrepancy. 

Best regards,

Christine.

[Christine_Li]

Hi, @ArthurC 

Thanks for your reply.

Sure, We will use the given nmcli to have a try on our local side.

Best regards,

Christine.

[ArthurC]

Hello @Christine_Li,

We are using (standalone prebuilt image + your yocto build nmcli binary) to test.

But our EVK's PMIC (pfuse100) is unexpected broken during the test procedure.

Probably it was purchased for long time ago. It can't boot anymore.

So we can't provide the results for you. 

The yocto build nmcli binary is as attached. 

Please help to use it to check.

Thank you

[Christine_Li]

Hi, @ArthurC 

Thanks for your quick reply.

1. If you have complete BSP(i.MX8MQ-EVK prebuilt image + nmcli package) then please provide us the full BSP image.

2. If you are using the (standalone prebuilt image + your yocto build nmcli binary) then please provide us the nmcli binary to install in our i.MX8MQ-EVK and also provide the steps which you followed to install the nmcli binary in our i.MX8MQ-EVK.

Best regards,

Christine.