88Q9098: Configuration Issues for Dual-MAC Operation

[jrhaws]

I am running an 88Q9098 and am trying to ensure that both MACs work properly with an AP and STA on both.  I have the device attached to my iMX6ULL EVK  via SDIO and am running the device in SDIO/UART mode (with device tree updated per this thread).

I am configuring my network devices with systemd-networkd and wpa_supplicant. I don't use NetworkManager because it interferes with my protocol timing by randomly doing scans and other "management" tasks that I don't need.

I see interesting behavior with various configurations. To my understanding I have to specify the specific band my AP will operate on as well as the interface, so that ties uap0 and muap0 to specific bands.

I only need to connect a single station at a time, but I will have to ensure that if I connect to 2.4 GHz I use the station on the same MAC as the AP on 2.4 GHz; same for 5 GHz.

About 50% of the time, I see the following kernel dump during boot:

[ 320.783247] ------------[ cut here ]------------
[ 320.796790] WARNING: CPU: 0 PID: 688 at net/wireless/util.c:1349 cfg80211_calculate_bitrate_he+0x240/0x34c [cfg80211]
[ 320.816883] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_tables x_tables snd_soc_fsl_asoc_card ov5640 snd_soc_imx_audmux snd_soc_simple_card_utils v4l2_fwnode snd_ac97_codec videodev ac97_bus mc snd_soc_wm8960 snd_soc_fsl_sai snd_soc_fsl_asrc imx_pcm_dma snd_soc_core imx_sdma snd_pcm_dmaengine snd_pcm snd_timer snd soundcore flexcan can_dev sch_fq_codel vxcan vcan moal(O) mlan(O) cfg80211 lz4 lz4_compress zram can_isotp can_gw can fuse configfs
[ 320.926651] CPU: 0 PID: 688 Comm: systemd-network Tainted: G O 5.10.93-lmp-standard #1
[ 320.935937] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[ 320.942143] Backtrace:
[ 320.944635] [<c0cb740c>] (dump_backtrace) from [<c0cb77ac>] (show_stack+0x20/0x24)
[ 320.952242] r7:00000545 r6:600f0113 r5:00000000 r4:c11f2b18
[ 320.957931] [<c0cb778c>] (show_stack) from [<c0cbb804>] (dump_stack+0x90/0xa4)
[ 320.965189] [<c0cbb774>] (dump_stack) from [<c0136be0>] (__warn+0xfc/0x158)
[ 320.972177] r7:00000545 r6:00000009 r5:bf047580 r4:bf0b40fc
[ 320.977865] [<c0136ae4>] (__warn) from [<c0cb8020>] (warn_slowpath_fmt+0x8c/0xc0)
[ 320.985378] r7:00000545 r6:bf0b40fc r5:00000000 r4:c567e000
[ 320.991267] [<c0cb7f98>] (warn_slowpath_fmt) from [<bf047580>] (cfg80211_calculate_bitrate_he+0x240/0x34c [cfg80211])
[ 321.001931] r9:c567f9a0 r8:c567f9ac r7:c567e000 r6:c567f9b8 r5:c567fb7a r4:00000000
[ 321.009896] [<bf047340>] (cfg80211_calculate_bitrate_he [cfg80211]) from [<bf047804>] (cfg80211_calculate_bitrate+0x178/0x2fc [cfg80211])
[ 321.022295] r10:c56f2480 r9:c567fc0e r8:c8c96d3c r7:c8c96cec r6:c567e000 r5:c567fb7a
[ 321.030155] r4:c56f2480
[ 321.032877] [<bf04768c>] (cfg80211_calculate_bitrate [cfg80211]) from [<bf0777f4>] (nl80211_put_sta_rate+0x60/0x334 [cfg80211])
[ 321.044395] r4:c56f2480
[ 321.047115] [<bf077794>] (nl80211_put_sta_rate [cfg80211]) from [<bf0780e8>] (nl80211_send_station+0x620/0xe90 [cfg80211])
[ 321.058204] r8:c567fc0e r7:c8c96cec r6:c2f55800 r5:c56f2480 r4:c567fb38
[ 321.065096] [<bf077ac8>] (nl80211_send_station [cfg80211]) from [<bf078a60>] (nl80211_dump_station+0x108/0x2e0 [cfg80211])
[ 321.076186] r10:c56f2480 r9:c567fc0e r8:c2f559c0 r7:c2f75e58 r6:00000000 r5:c567fb38
[ 321.084045] r4:00000000
[ 321.086688] [<bf078958>] (nl80211_dump_station [cfg80211]) from [<c0ad5ebc>] (netlink_dump+0x158/0x338)
[ 321.096125] r10:c122d9c0 r9:00001244 r8:c2f75e58 r7:00001244 r6:c567e000 r5:c56f2480
[ 321.103983] r4:c2f75c00
[ 321.106545] [<c0ad5d64>] (netlink_dump) from [<c0ad6fe8>] (__netlink_dump_start+0x17c/0x294)
[ 321.115015] r10:c122d9c0 r9:c2f75e58 r8:00000000 r7:c567fcf8 r6:c56f2540 r5:c2f75c00
[ 321.122871] r4:c2f75c64
[ 321.125429] [<c0ad6e6c>] (__netlink_dump_start) from [<c0ad9954>] (genl_rcv_msg+0x344/0x378)
[ 321.133904] r9:c567fd88 r8:00000004 r7:c567e000 r6:c56f2540 r5:bf0bb000 r4:c28f4e00
[ 321.141682] [<c0ad9610>] (genl_rcv_msg) from [<c0ad87b4>] (netlink_rcv_skb+0xcc/0x124)
[ 321.149632] r10:c2f75d84 r9:00000000 r8:0000001c r7:c28f4e00 r6:c567e000 r5:c0ad9610
[ 321.157487] r4:c56f2540
[ 321.160043] [<c0ad86e8>] (netlink_rcv_skb) from [<c0ad8e88>] (genl_rcv+0x34/0x44)
[ 321.167557] r8:c122d9c0 r7:c56f2540 r6:c15f5140 r5:c12347f8 r4:c56f2540
[ 321.174287] [<c0ad8e54>] (genl_rcv) from [<c0ad7d74>] (netlink_unicast+0x258/0x39c)
[ 321.181971] r5:c1605464 r4:c1605400
[ 321.185573] [<c0ad7b1c>] (netlink_unicast) from [<c0ad80d8>] (netlink_sendmsg+0x220/0x49c)
[ 321.193870] r10:00000000 r9:c2f75c00 r8:c56f2540 r7:c567e000 r6:0000001c r5:c28f4e00
[ 321.201725] r4:c567feb0
[ 321.204287] [<c0ad7eb8>] (netlink_sendmsg) from [<c0a4bae0>] (__sys_sendto+0xe4/0x134)
[ 321.212240] r10:00000122 r9:c567e000 r8:be8438fc r7:00000040 r6:c567e000 r5:c7ca4000
[ 321.220096] r4:00000000
[ 321.222655] [<c0a4b9fc>] (__sys_sendto) from [<c0a4bb54>] (sys_sendto+0x24/0x2c)
[ 321.230083] r8:c0100264 r7:00000122 r6:b6fb8810 r5:00000080 r4:be8438fc
[ 321.236817] [<c0a4bb30>] (sys_sendto) from [<c0100244>] (__sys_trace_return+0x0/0x1c)
[ 321.244675] Exception stack(0xc567ffa8 to 0xc567fff0)
[ 321.249752] ffa0: be8438fc 00000080 00000009 00bd88e0 0000001c 00000000
[ 321.257965] ffc0: be8438fc 00000080 b6fb8810 00000122 00000000 000002b0 00bc6138 be843d4c
[ 321.266169] ffe0: 00000122 be8438c8 b6d554e9 b6cc8996
[ 321.318572] ---[ end trace bf20526c2425181a ]---

Beyond that I'm finding that most of the time I don't get a DHCP address from my server. Sometimes I can set a manual address and gateway with these commands and get Internet access, but most times not:

ip addr add dev mlan0 <ip>/24
ip route add default via <gateway>

The most concerning issue to me is that the behavior is wildly inconsistent. I'll boot up and nothing will work, reboot and things work great, reboot again and things are broken again.

I've tried about every combination of settings, all of which give the inconsistent behavior and random driver dumps shown above:

• mlan0 all by itself (uap0/muap0/mmlan0 all turned off) - works all the time, except for random DHCP assignment issues
• mmlan0 all by itself (uap0/muap0/mlan0 all turned off) - works all the time, except for random DHCP assignment issues
• mlan0 with only uap0 - works about all the time; sometimes a DHCP assignment issue (i.e., no address assigned)
• mmlan0 with only muap0 - usually gets address assignment, but cannot access Internet (ping fails with "aborted" for external address and gateway address)
• mlan0 with both uap0/muap0 - similar issues to mmlan0+muap0 but rarely get valid address assignment
• mmlan0 with both uap0/muap0 - similar issues to mmlan0+muap0 but rarely get valid address assignment 

At the end of the day the configuration I am after is as follows:

mlan0 or mmlan0: connected to external AP for Internet - ideally a single config, but can manage multiples and only have one active at a time
uap0/muap0: Advertising single SSID on both 2.4 GHz and 5 GHz, simultaneously; for now I'd settle for different SSIDs if it always worked

My guess is that there are still some driver kinks to work out for the new device. Just let me know what logs and whatnot I can gather to provide the necessary details to help resolve this!

[achyut]

Hello @jrhaws 

There has been a new driver released for NXP chipsets including SDIO 9098 end of June. 

Could you please give it a try with it? Make sure you use the latest FW as well.

Driver: https://source.codeaurora.org/external/imx/mwifiex/tree/?h=lf-5.15.32_2.0.0

Firmware: https://github.com/NXP/imx-firmware/blob/lf-5.15.32_2.0.0/nxp/FwImage_9098_SD/sdiouart9098_combo_v1....

Thanks, Achyut

[jrhaws]

@achyut 

That is the version we are using now, commit hash 31800f43a6ee562f412d2301bb0c7e7d0c129231.

We are seeing the same behavior, with a few additional error messages as reported in another post I made on this thread.

[jrhaws]

Here are the systemd-networkd configuration files.

I have changed some configurations as well. I'll do some more rigorous testing as I get the other information.

[estephania_mart]

Hello,

We truly apologize for the delay, by any chance, is this problem solved with the fix proposed in this other post ?

https://community.nxp.com/t5/Wireless-Connectivity/88Q9098-88W8987-connectivity-problems-with-statio...

Do you still have problems ?

Regards

[jrhaws]

No, that solution does not solve the problem. In fact, that other post is mine as well.

I have these same problems on the EVK and custom hardware with the same Jody W3 chip, so I don't believe it is device tree related.

I believe there are likely some issues in how the dual MACs are handled internal to the software, especially since the default MAC (the first one) doesn't seem to exhibit poor behavior, especially when used on its own. When I use the second MAC, that is when things seem to start failing.

I'm open to try any patches or anything else to try and get to the bottom of this.  Thank you for the support!

[jrhaws]

@ocourson 

Sorry for the delay in my response.  I've been pulled off on some other hardware debugging and am just getting back to this.

As far as trying the prebuilt binaries, I'll take a look at that and report back. Same with the drvdbg=0x20037 logs.

But here is all the other information you requested for now:

• Kernel Version: 5.10.93-lmp-standard
• WiFi SW: git commit is 29b5d07aee25a1135db92e608049f6a50c5e44be (not sure where to pull the actual version from the logs)
• wifi_mod_para.conf contains:

SD9098_0 = {
        cfg80211_wext=0xf
        max_vir_bss=1
        cal_data_cfg=none
        ps_mode=1
        auto_ds=1
        host_mlme=1
        fw_name=nxp/sdiouart9098_combo_v1.bin
}

SD9098_1 = {
        cfg80211_wext=0xf
        max_vir_bss=1
        cal_data_cfg=none
        ps_mode=1
        auto_ds=1
        host_mlme=1
        fw_name=nxp/sdiouart9098_combo_v1.bin
}​

• I've got uap0 and muap0 configured as hotspots with hostapd, controlled via systemd. mlan0 is configured as a station using wpa-supplication, also controlled via systemd. systemd-networkd is managing network configuration and I will provide those configurations as well in another reply since I can only attach 5 files here.

[jrhaws]

I found the driver version in another dump I saw - this one appeared to be a driver timeout when utilizing Bluetooth, but only happens on my custom hardware - not the EVK:

SD9098----17.92.1.p98.1-MM5X17299.p1-GPL-(FP92)

Here is the full dump - maybe these things are related? Maybe not. I can start a new topic for this if necessary:

[  383.024239] Bluetooth: hci0: Frame reassembly failed (-84)
[  398.951457] wlan_wakeup_card_timeout_func: ps_state=3
[  398.956810] mlan0:
[  398.956867] Wakeup card timeout!
[  398.962391] Driver version = SD9098----17.92.1.p98.1-MM5X17299.p1-GPL-(FP92)
[  398.969658] main_state = 4
[  398.972460] ioctl_pending = 0
[  398.975524] tx_pending = 3
[  398.978326] wmm_tx_pending[0] = 0
[  398.981732] wmm_tx_pending[1] = 0
[  398.985141] wmm_tx_pending[2] = 3
[  398.988547] wmm_tx_pending[3] = 0
[  398.991954] rx_pending = 0
[  398.994753] lock_count = 79
[  398.997633] malloc_count = 66
[  399.000692] mbufalloc_count = 0
[  399.003924] hs_skip_count = 0
[  399.006987] hs_force_count = 0
[  399.010134] Media state = "Connected"
[  399.013892] carrier on
[  399.016341] tx queue 0: started
[  399.019574] tx queue 1: started
[  399.022810] tx queue 2: started
[  399.026044] tx queue 3: started
[  399.029281] mlan0: num_tx_timeout = 0
[  399.033045] uap0: num_tx_timeout = 0
[  399.036740] wfd0: num_tx_timeout = 0
[  399.040886] Block woal_cfg80211_del_key in abnormal driver state
[  399.047913] Block woal_cfg80211_del_key in abnormal driver state
[  399.056513] Block woal_cfg80211_del_key in abnormal driver state
[  399.063188] Block woal_cfg80211_del_key in abnormal driver state
[  399.069480] Block woal_cfg80211_del_key in abnormal driver state
[  399.076041] Block woal_cfg80211_del_key in abnormal driver state
[  399.112751] get fw info failed! status=-1, error_code=0x0
[  399.118232] 11D: Error setting domain info in FW
[  399.131883] 11D: Error setting domain info in FW
[  399.136645] get fw info failed! status=-1, error_code=0x0
[  399.158666] 11D: Error setting domain info in FW
[  399.163628] 11D: Error setting domain info in FW
[  399.212407] Fail to set EXTCAP IE
[  399.215814] woal_do_scan fails!

[ocourson]

Dear @jrhaws 

Please confirm:

- which kernel version are you using (5.10.X ?)

- Which WiFi SW are you using (is it Driver version = SD9098----17.92.1.p98.1-MM5X17299.p1-GPL-(FP92) ?)

Could you try to reproduce your issue with the prebuilt i.MX6ULL 5.15.5 binaries (including the SD9098----17.92.1.p98.1-MM5X17299.p1-GPL) ?

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

https://www.nxp.com/webapp/Download?colCode=L5.15.5_1.0.0_MX6UL7D&appType=license

Regards,

Olivier

[ocourson]

Dear @jrhaws 

Could you also provide the following:

- parameters used at Wifi driver init

- full list of commands used and configuration files used

- add drvdbg=0x20037 driver parameter to enable debug logs and provide us the corresponding logs

Olivier

[jrhaws]

@ocourson 

Here is a log with drvdbg=0x20037. I also attached the mmlan0 wpa_supplicant configuration file. The mlan0 one is attached to another message in this thread.

When I run wpa_supplicant-nl80211@mlan0, I don't seem to have any issues. However, when I run wpa_supplicant-nl80211@mmlan0, I appear to have a DHCP address from my router, but I cannot get online as seen in the log.

The only difference between the mlan0 and mmlan0 is that mmlan0 is set up with a 5GHz frequency list, while mlan0 has a 2.4GHz list.  My router is functional on 5GHz in case that is a question.

I have both my 2.4 GHz and 5 GHz APs running.

As you can see in the log, I have an address, I can ping myself, but I cannot ping my gateway or out to the Internet - both of those pings simply return "Aborted" instead of reporting dropped packets as I was expecting.

Hopefully there is enough information in the logs to track down whatever the issue may be. Ideally, this would be a configuration issue that I can easily adjust; but it has the feel of something bigger.

I am building my Linux image with the Yocto tools and already have a mxm_wiflex module bbappend setup, so if there are patches you want to throw my way I'm happy to test them!

Thank you for the support!

[jrhaws]

@ocourson 

More results from mmlan0.

Disabled both APs: mmlan0 worked, but there were lots of debug logs that aren't there when running just mlan0

Enabled just uap0 (on other MAC): mmlan0 worked, same mass of debug logs as with no APs

Enabled just muap0 (on same MAC): mmlan0 does NOT work, no mass of debug logs as with other two scenarios.

Basically, it appears that mmlan0 does not work when muap0 is also running. However, mlan0 and uap0 work just fine.

What I would expect is that each MAC would function 100% independent of one another, and I could have two APs and two STAs running simultaneously.

[jrhaws]

@ocourson 

I just updated to the latest branch (lf-5.15.32_2.0.0) and ran the same tests.  Behavior is very much the same. If you need new logs let me know.

I did see a few new messages with the new driver:

[ 19.254855] CMD_RESP: cmd 0x27c error, result=0x2
[ 19.259635] IOCTL failed: 79b17d1c id=0xe0000, sub_id=0xe0001 action=2, status_code=0x3
[ 19.280497] CMD_RESP: cmd 0x27c error, result=0x2
[ 19.290575] IOCTL failed: 79b17d1c id=0xe0000, sub_id=0xe0001 action=1, status_code=0x3
[ 19.417295] CMD_RESP: cmd 0x27c error, result=0x2
[ 19.422074] IOCTL failed: 127ee8e7 id=0xe0000, sub_id=0xe0001 action=2, status_code=0x3
[ 19.443955] CMD_RESP: cmd 0x27c error, result=0x2
[ 19.448737] IOCTL failed: 127ee8e7 id=0xe0000, sub_id=0xe0001 action=1, status_code=0x3

This happens about half the time on boot. I have wpa-supplicant for mlan0 and hostapd for both uap0/muap0 enabled. Things still seem to work properly, but maybe this helps to point at a potential issue?

[ocourson]

Dear @jrhaws 

Please apologize for late answer.

Regarding the BT init issue you see on you HW, remind that this 88W9098 firmware is using SDIO interface for the WiFi interface, but UART for the BT interface.

Can you please check you have a valid UART connection between 88W9098 and your host ?

Regards,

Olivier

[jrhaws]

@ocourson 

Understood on the Bluetooth; and yes, we have a working UART between the two. I bring it up here because it seemed related - accessing the Bluetooth caused the WiFi driver to hang/crash.

[jrhaws]

Some more feedback on the issues.

I've stated that mlan0 + uap0 and muap0 seems to work fine. And that is true about 80% of the time. The other 20% of the time I'll have a connection and be shelled in over SSH only to have the device lose its IP - I can verify this through the serial console I keep open. After 5-10 minutes or so, it seems to regain its brain and get reassigned its IP and I can then reconnect my SSH sessions.

I don't experience this with the 88W8987 (i.e., the u-blox Jody W2); only the 88Q9098 (Jody W3). I can simply swap those modules, leaving all other software and configuration the same, and see this behavior.

Again, my configuration files are posted, but this is running wpa_supplicant to manage my station connection and hostapd to manage my access points. This feels like an issue with the second MAC causing issues at the driver level with stuff going on in the first MAC.

[weidong_sun]

Hello @jrhaws ,

I just replied to you on another post of yours.

It seems to be the same project.

https://community.nxp.com/t5/Wireless-Connectivity/SD9098-quot-abnormal-driver-state-quot/m-p/149581...

Fitstly solve IO level issuwe between CPU usdhc1 & WiFi SDIO.please!

WiFi SDIO card is differenct from SD card, it doesn't support switching IO level.

Regards,

weidong