topic Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs in Wi-Fi® + Bluetooth® + 802.15.4

PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

martinetd — Wed, 13 Apr 2022 07:25:57 GMT

Hello,

This is probably not a main usecase, but I'm observing errors when connecting on both mlan0 and mmlan0 on different bands (5GHz / 2.4GHz) simultaneously.

It appears to work for a while, then there are sometimes connectivity troubles and it finally crashed. after the crash linux works fine but wireless connectivity is dead, even scans stop working.

Hardware:

- imx8mp evk (SCH-46370 REV A1 / 700-46370 REV X1)

- AW-XM458 (identified PCI9098 in code, 1b4b:2b43 / 1b4b:2b44 in lspci)

Software:

- v5.10.72-2.2.0 BSP

- my evk has an older chip which is not supported by the kernel, so I reverted a single patch to restore the workaround: https://github.com/atmark-techno/linux-5.10-at/commits/nxp_5.10.72-2.2.0-repro using imx_v8_defconfig
- wifi driver as per bsp, default config https://source.codeaurora.org/external/imx/mwifiex/log/?h=lf-5.10.72-2.2.0
- firmware as per bsp, I confirmed it is https://github.com/NXP/imx-firmware/blob/lf-5.10.72-2.2.0/nxp/FwImage_9098_PCIE/pcieuart9098_combo_v1.bin without modifying it

setup: two APs, one on 5GHz and one on 2.4GHz

connected manually with wpa_supplicant as follow:

$ cat wpa_supplicant_5G.conf ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=0 update_config=1 network={ ssid="xxx_5G" key_mgmt=WPA-PSK proto=WPA2 pairwise=CCMP TKIP group=CCMP TKIP psk="xxx" } $ cat wpa_supplicant_2G4.conf ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=0 update_config=1 network={ ssid="xxx" key_mgmt=WPA-PSK proto=WPA2 pairwise=CCMP TKIP group=CCMP TKIP psk="xxxxxxx" } # wpa_supplicant -i mlan0 -c wpa_supplicant_5G.conf -d & # wpa_supplicant -i mmlan0 -c wpa_supplicant_2G4.conf -d &

There was no particular workload when the crash happened, I successfully used the network before and just left it connected when it happened.

[ 449.394874] mlan: loading out-of-tree module taints kernel. [ 449.577647] wlan: Loading MWLAN driver [ 449.582482] wlan: Driver loaded successfully [ 449.582537] wlan: Register to Bus Driver... [ 449.591183] wlan_pcie 0000:01:00.0: enabling device (0000 -> 0002) [ 449.597464] Attach moal handle ops, card interface type: 0x206 [ 449.603337] No module param cfg file specified [ 449.607811] rx_work=1 cpu_num=4 [ 449.610984] Attach mlan adapter operations.card_type is 0x206. [ 449.625192] Request firmware: nxp/pcieuart9098_combo_v1.bin [ 450.285239] FW download over, size 620880 bytes [ 451.801082] WLAN FW is active [ 451.804061] on_time is 451539036627 [ 451.821678] fw_cap_info=0x81c3fa3, dev_cap_mask=0xffffffff [ 451.827209] max_p2p_conn = 8, max_sta_conn = 64 [ 451.842722] wlan: version = PCIE9098--17.92.5.p11-MM5X17283.p2-GPL-(FP92) [ 451.850236] wlan_pcie 0000:01:00.1: enabling device (0000 -> 0002) [ 451.856532] Attach moal handle ops, card interface type: 0x206 [ 451.862434] No module param cfg file specified [ 451.866968] rx_work=1 cpu_num=4 [ 451.870202] Attach mlan adapter operations.card_type is 0x206. [ 451.877137] Request firmware: nxp/pcieuart9098_combo_v1.bin [ 451.883495] WLAN FW already running! Skip FW download [ 451.888618] WLAN FW is active [ 451.891639] on_time is 451626613662 [ 451.899131] fw_cap_info=0x81c3fa3, dev_cap_mask=0xffffffff [ 451.904716] max_p2p_conn = 8, max_sta_conn = 64 [ 451.925988] wlan: version = PCIE9098--17.92.5.p11-MM5X17283.p2-GPL-(FP92) [ 451.933620] wlan: Register to Bus Driver Done .... [ 5484.493580] wlan: mmlan0 START SCAN [ 5485.453521] wlan: SCAN COMPLETED: scanned AP count=1 [ 5784.485077] wlan: mmlan0 START SCAN [ 5785.446095] wlan: SCAN COMPLETED: scanned AP count=1 [ 6084.477430] wlan: mmlan0 START SCAN [ 6085.439891] wlan: SCAN COMPLETED: scanned AP count=2 [ 6354.694300] FW trigger fw dump [ 6354.697390] =====FW trigger dump==== [ 6354.701376] ==== Start Receive FW dump event ==== [ 6355.273227] EvtRdy: failed to read REG_EVTBD_RDPTR [ 6359.684630] Timeout cmd id (6359.681175) = 0xce, act = 0x400 [ 6359.690317] ce [ 6359.690324] 00 [ 6359.692087] 16 [ 6359.693847] 00 [ 6359.695613] 16 [ 6359.697373] 00 [ 6359.699133] 00 [ 6359.700894] 00 [ 6359.702653] 00 [ 6359.704412] 04 [ 6359.706172] 42 [ 6359.707932] 1a [ 6359.709692] e4 [ 6359.711452] 78 [ 6359.713212] 0c [ 6359.714972] 0e [ 6359.716731] [ 6359.719978] Dump Rx CMD Response Buf: [ 6359.723711] BSS type = 0 BSS role= 0 [ 6359.727294] ------------Dump info----------- [ 6359.731573] Commmand Timeout [ 6359.734463] mlan_processing =0 [ 6359.737525] main_lock_flag =0 [ 6359.740501] main_process_cnt =2405571 [ 6359.744170] delay_task_flag =0 [ 6359.747232] mlan_rx_processing =0 [ 6359.750556] rx_pkts_queued=0 [ 6359.753444] more_task_flag = 0 [ 6359.756506] num_cmd_timeout = 1 [ 6359.759655] last_cmd_index = 6 [ 6359.762717] last_cmd_id = [ 6359.762726] 0x10 [ 6359.765440] 0x28 [ 6359.767374] 0xd0 [ 6359.769307] 0xd0 [ 6359.771240] 0xd0 [ 6359.773174] 0xd0 [ 6359.775107] 0xce [ 6359.777040] 0x10 [ 6359.778974] 0x28 [ 6359.780907] 0x28 [ 6359.782840] [ 6359.786259] last_cmd_act = [ 6359.786266] 0x1 [ 6359.789068] 0x213 [ 6359.790914] 0x400 [ 6359.792934] 0x400 [ 6359.794954] 0x400 [ 6359.796975] 0x400 [ 6359.798995] 0x400 [ 6359.801014] 0x1 [ 6359.803034] 0x213 [ 6359.804881] 0x293 [ 6359.806901] [ 6359.810407] last_cmd_resp_index = 5 [ 6359.813903] last_cmd_resp_id = [ 6359.813911] 0x8010 [ 6359.817060] 0x8028 [ 6359.819167] 0x80d0 [ 6359.821274] 0x80d0 [ 6359.823381] 0x80d0 [ 6359.825488] 0x80d0 [ 6359.827594] 0x8028 [ 6359.829701] 0x8010 [ 6359.831808] 0x8028 [ 6359.833915] 0x8028 [ 6359.836021] [ 6359.839615] last_event_index = 4 [ 6359.842851] last_event = [ 6359.842858] 0x73 [ 6359.845486] 0x73 [ 6359.847420] 0x73 [ 6359.849353] 0x73 [ 6359.851286] 0x73 [ 6359.853219] 0x73 [ 6359.855152] 0x73 [ 6359.857085] 0x73 [ 6359.859018] 0x73 [ 6359.860951] 0x73 [ 6359.862883] [ 6359.866304] num_data_h2c_failure = 0 [ 6359.869887] num_cmd_h2c_failure = 0 [ 6359.873384] num_alloc_buffer_failure = 0 [ 6359.877314] num_pkt_dropped = 0 [ 6359.880463] num_no_cmd_node = 0 [ 6359.883612] num_event_deauth = 8 [ 6359.886848] num_event_disassoc = 0 [ 6359.890257] num_event_link_lost = 7 [ 6359.893754] num_cmd_deauth = 1 [ 6359.896815] num_cmd_assoc_success = 17 [ 6359.900572] num_cmd_assoc_failure = 0 [ 6359.904242] num_cons_assoc_failure = 0 [ 6359.907998] cmd_resp_received=0 [ 6359.911147] event_received=0 [ 6359.914036] max_tx_buf_size=4096 [ 6359.917272] tx_buf_size=3103 [ 6359.920160] curr_tx_buf_size=3103 [ 6359.923483] data_sent=0 cmd_sent=1 [ 6359.926893] ps_mode=1 ps_state=0 [ 6359.930129] wakeup_dev_req=0 wakeup_tries=0 wakeup_timeout=0 [ 6359.935796] hs_configured=0 hs_activated=0 [ 6359.939901] pps_uapsd_mode=0 sleep_pd=0 [ 6359.943743] tx_lock_flag = 0 [ 6359.946632] scan_processing = 0 [ 6359.949781] bypass_pkt_count=0 [ 6359.952844] txbd_rdptr=0x14e4 txbd_wrptr=0x14e6 [ 6359.957382] rxbd_rdptr=0x1ed4 rxbd_wrptr=0x1f54 [ 6359.961920] evtbd_rdptr=0xd42 evt_wrptr=0xd4a [ 6359.966285] last_wr_index:102 [ 6359.969260] txrx_bd_size = 128 [ 6359.972409] Tx pkt size: [ 6359.974950] 3070 [ 6359.974957] 3070 [ 6359.976890] 3070 [ 6359.978823] 3070 [ 6359.980756] 3070 [ 6359.982689] 3070 [ 6359.984622] 3070 [ 6359.986555] 3070 [ 6359.988488] 3070 [ 6359.990421] 3070 [ 6359.992354] 3070 [ 6359.994288] 3070 [ 6359.996220] 3070 [ 6359.998153] 3070 [ 6360.000086] 3070 [ 6360.002019] 3070 [ 6360.003952] [ 6360.007372] 3070 [ 6360.007379] 3070 [ 6360.009313] 3070 [ 6360.011246] 3070 [ 6360.013179] 3070 [ 6360.015111] 3070 [ 6360.017045] 3070 [ 6360.018978] 3070 [ 6360.020911] 3070 [ 6360.022844] 3070 [ 6360.024776] 3070 [ 6360.026709] 3070 [ 6360.028642] 3070 [ 6360.030575] 3070 [ 6360.032508] 3070 [ 6360.034441] 3070 [ 6360.036373] [ 6360.039793] 3070 [ 6360.039804] 3070 [ 6360.041737] 3070 [ 6360.043671] 3070 [ 6360.045604] 3070 [ 6360.047536] 3070 [ 6360.049469] 3070 [ 6360.051402] 3070 [ 6360.053335] 3070 [ 6360.055268] 3070 [ 6360.057201] 3070 [ 6360.059133] 3070 [ 6360.061067] 3070 [ 6360.062999] 3070 [ 6360.064933] 3070 [ 6360.066866] 3070 [ 6360.068798] [ 6360.072218] 3070 [ 6360.072226] 3070 [ 6360.074160] 3070 [ 6360.076093] 3070 [ 6360.078027] 3070 [ 6360.079960] 3070 [ 6360.081893] 3070 [ 6360.083826] 3070 [ 6360.085759] 3070 [ 6360.087692] 3070 [ 6360.089625] 3070 [ 6360.091558] 3070 [ 6360.093491] 3070 [ 6360.095424] 3070 [ 6360.097356] 3070 [ 6360.099290] 3070 [ 6360.101222] [ 6360.104642] 3070 [ 6360.104649] 3070 [ 6360.106582] 3070 [ 6360.108515] 3070 [ 6360.110447] 3070 [ 6360.112380] 3070 [ 6360.114313] 3070 [ 6360.116246] 3070 [ 6360.118179] 3070 [ 6360.120113] 3070 [ 6360.122046] 3070 [ 6360.123979] 3070 [ 6360.125912] 3070 [ 6360.127845] 3070 [ 6360.129778] 3070 [ 6360.131711] 3070 [ 6360.133644] [ 6360.137063] 3070 [ 6360.137071] 3070 [ 6360.139004] 3070 [ 6360.140938] 3070 [ 6360.142870] 3070 [ 6360.144804] 3070 [ 6360.146737] 3070 [ 6360.148670] 3070 [ 6360.150603] 3070 [ 6360.152536] 3070 [ 6360.154469] 3070 [ 6360.156402] 1544 [ 6360.158336] 0100 [ 6360.160269] 0096 [ 6360.162202] 0381 [ 6360.164135] 0096 [ 6360.166068] [ 6360.169488] 0097 [ 6360.169495] 0096 [ 6360.171429] 0096 [ 6360.173362] 0128 [ 6360.175295] 0128 [ 6360.177228] 0076 [ 6360.179162] 3070 [ 6360.181095] 3070 [ 6360.183028] 3070 [ 6360.184961] 3070 [ 6360.186894] 3070 [ 6360.188827] 3070 [ 6360.190761] 3070 [ 6360.192694] 3070 [ 6360.194626] 3070 [ 6360.196559] 3070 [ 6360.198491] [ 6360.201911] 3070 [ 6360.201918] 3070 [ 6360.203851] 3070 [ 6360.205785] 3070 [ 6360.207718] 3070 [ 6360.209651] 3070 [ 6360.211584] 3070 [ 6360.213517] 3070 [ 6360.215450] 3070 [ 6360.217383] 3070 [ 6360.219316] 3070 [ 6360.221249] 3070 [ 6360.223182] 3070 [ 6360.225115] 3070 [ 6360.227049] 3070 [ 6360.228982] 3070 [ 6360.230914] [ 6360.234336] bss_index = 0, tx_pkts_queued = 0 [ 6360.238701] bss_index = 1, tx_pkts_queued = 0 [ 6360.243066] bss_index = 2, tx_pkts_queued = 0 [ 6360.247430] -------- Dump info End--------- [ 6360.251628] Driver version = PCIE9098--17.92.5.p11-MM5X17283.p2-GPL-(FP92) [ 6360.258598] main_state = 4 [ 6360.261313] ioctl_pending = 0 [ 6360.264288] tx_pending = 2 [ 6360.267003] wmm_tx_pending[0] = 0 [ 6360.270326] wmm_tx_pending[1] = 0 [ 6360.273648] wmm_tx_pending[2] = 2 [ 6360.276970] wmm_tx_pending[3] = 0 [ 6360.280292] rx_pending = 0 [ 6360.283007] lock_count = 62 [ 6360.285808] malloc_count = 62 [ 6360.288783] mbufalloc_count = 136 [ 6360.292105] malloc_cons_count = 4 [ 6360.295428] hs_skip_count = 0 [ 6360.298402] hs_force_count = 0 [ 6360.301465] Media state = "Connected" [ 6360.305135] carrier on [ 6360.307502] tx queue 0: started [ 6360.310652] tx queue 1: started [ 6360.313800] tx queue 2: started [ 6360.316949] tx queue 3: started [ 6360.320098] mlan0: num_tx_timeout = 0 [ 6360.323769] uap0: num_tx_timeout = 0 [ 6360.327352] wfd0: num_tx_timeout = 0 [ 6360.330937] fw_dump_timer fired. [ 6360.334174] Start to process hanging [ 6360.337800] Cancel all pending cmd! [ 6360.337864] Block woal_cfg80211_del_key in abnormal driver state [ 6360.347342] Block woal_cfg80211_del_key in abnormal driver state [ 6360.353368] Block woal_cfg80211_del_key in abnormal driver state [ 6360.359396] Block woal_cfg80211_del_key in abnormal driver state [ 6360.365416] Block woal_cfg80211_del_key in abnormal driver state [ 6360.371438] Block woal_cfg80211_del_key in abnormal driver state [ 6360.439350] Fail to set EXTCAP IE [ 6360.442704] Fail to set beacon proberesp assoc probereq IES [ 6360.448305] Fail to set scan request IE [ 6361.454224] Fail to set EXTCAP IE [ 6361.457590] Fail to set beacon proberesp assoc probereq IES [ 6361.463191] Fail to set scan request IE [ 6362.469122] Fail to set EXTCAP IE [ 6362.472452] Fail to set beacon proberesp assoc probereq IES [ 6362.478061] Fail to set scan request IE [ 6363.483993] Fail to set EXTCAP IE [ 6363.487352] Fail to set beacon proberesp assoc probereq IES [ 6363.492951] Fail to set scan request IE [ 6364.498902] Fail to set EXTCAP IE [ 6364.502263] Fail to set beacon proberesp assoc probereq IES [ 6364.507862] Fail to set scan request IE [ 6365.513790] Fail to set EXTCAP IE [ 6365.517156] Fail to set beacon proberesp assoc probereq IES [ 6365.522752] Fail to set scan request IE [ 6366.528662] Fail to set EXTCAP IE [ 6366.531992] Fail to set beacon proberesp assoc probereq IES [ 6366.537594] Fail to set scan request IE [ 6367.542616] Fail to set EXTCAP IE [ 6367.545978] Fail to set beacon proberesp assoc probereq IES [ 6367.551573] Fail to set scan request IE [ 6368.557481] Fail to set EXTCAP IE [ 6368.560841] Fail to set beacon proberesp assoc probereq IES [ 6368.566437] Fail to set scan request IE [ 6369.572378] Fail to set EXTCAP IE [ 6369.575709] Fail to set beacon proberesp assoc probereq IES [ 6369.581309] Fail to set scan request IE [ 6370.587211] Fail to set EXTCAP IE [ 6370.590569] Fail to set beacon proberesp assoc probereq IES [ 6370.596167] Fail to set scan request IE [ 6371.602373] Fail to set EXTCAP IE [ 6371.605731] Fail to set beacon proberesp assoc probereq IES [ 6371.611329] Fail to set scan request IE

For now our workaround is to not connect both MACs simultaneously, as I've not seen the problem yet with a single STA or STA+AP

Thank you

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

ocourson — Thu, 14 Apr 2022 15:26:36 GMT

Dear customer,

Could you please:

1/ provide the parameters used for moal.ko insmod

2/ add drvdbg=0x20037 to moal.ko init parameters to enable debug logs, and provide us the logs corresponding to your issue

Regards,

Olivier

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

martinetd — Fri, 15 Apr 2022 03:44:51 GMT

Hello @ocourson

I did not specify any parameter when loading moal, just did "modprobe moal" with no options. /etc/modprobe.d is empty so it is using default options.

I got different hangs a couple of times but finally managed to reproduce:

- in seriallog_crash.txt you can find traces with a similar fw dump

- in seriallog_timeout.txt you can find traces with a "transmit queue 2 timed out" message, after which similar scan failure messages appeared and the network became unusable.

All the while I've also done outside pings to the device and when connected to two APs like this pings almost never came back, it mostly seemed to respond when the device made outwards connections (e.g. device woke up). Sorry these are not timed:

admin@RT-AX89X-7808:/tmp/home/root# ping 192.168.50.65 PING 192.168.50.65 (192.168.50.65): 56 data bytes 64 bytes from 192.168.50.65: seq=34 ttl=64 time=285.819 ms 64 bytes from 192.168.50.65: seq=35 ttl=64 time=161.870 ms 64 bytes from 192.168.50.65: seq=37 ttl=64 time=2795.000 ms 64 bytes from 192.168.50.65: seq=38 ttl=64 time=1795.100 ms 64 bytes from 192.168.50.65: seq=39 ttl=64 time=794.993 ms 64 bytes from 192.168.50.65: seq=174 ttl=64 time=51.137 ms 64 bytes from 192.168.50.65: seq=175 ttl=64 time=4337.612 ms 64 bytes from 192.168.50.65: seq=176 ttl=64 time=3337.512 ms 64 bytes from 192.168.50.65: seq=177 ttl=64 time=2337.380 ms 64 bytes from 192.168.50.65: seq=178 ttl=64 time=1337.245 ms 64 bytes from 192.168.50.65: seq=179 ttl=64 time=337.118 ms 64 bytes from 192.168.50.65: seq=320 ttl=64 time=2301.913 ms 64 bytes from 192.168.50.65: seq=321 ttl=64 time=1403.758 ms 64 bytes from 192.168.50.65: seq=322 ttl=64 time=506.039 ms 64 bytes from 192.168.50.65: seq=323 ttl=64 time=4627.113 ms 64 bytes from 192.168.50.65: seq=324 ttl=64 time=3627.223 ms 64 bytes from 192.168.50.65: seq=325 ttl=64 time=2627.101 ms 64 bytes from 192.168.50.65: seq=326 ttl=64 time=1626.965 ms 64 bytes from 192.168.50.65: seq=327 ttl=64 time=626.943 ms 64 bytes from 192.168.50.65: seq=474 ttl=64 time=1200.235 ms 64 bytes from 192.168.50.65: seq=475 ttl=64 time=200.108 ms 64 bytes from 192.168.50.65: seq=476 ttl=64 time=4324.848 ms 64 bytes from 192.168.50.65: seq=477 ttl=64 time=3324.948 ms 64 bytes from 192.168.50.65: seq=478 ttl=64 time=2324.824 ms 64 bytes from 192.168.50.65: seq=479 ttl=64 time=1324.693 ms 64 bytes from 192.168.50.65: seq=480 ttl=64 time=324.679 ms 64 bytes from 192.168.50.65: seq=773 ttl=64 time=1043.667 ms 64 bytes from 192.168.50.65: seq=774 ttl=64 time=43.529 ms 64 bytes from 192.168.50.65: seq=775 ttl=64 time=4362.015 ms 64 bytes from 192.168.50.65: seq=776 ttl=64 time=3361.947 ms 64 bytes from 192.168.50.65: seq=777 ttl=64 time=2361.783 ms 64 bytes from 192.168.50.65: seq=778 ttl=64 time=1361.779 ms 64 bytes from 192.168.50.65: seq=779 ttl=64 time=370.374 ms 64 bytes from 192.168.50.65: seq=1071 ttl=64 time=2155.837 ms 64 bytes from 192.168.50.65: seq=1072 ttl=64 time=1155.748 ms 64 bytes from 192.168.50.65: seq=1073 ttl=64 time=155.650 ms 64 bytes from 192.168.50.65: seq=1074 ttl=64 time=1.038 ms 64 bytes from 192.168.50.65: seq=1075 ttl=64 time=3281.966 ms 64 bytes from 192.168.50.65: seq=1076 ttl=64 time=2281.874 ms 64 bytes from 192.168.50.65: seq=1077 ttl=64 time=1281.736 ms 64 bytes from 192.168.50.65: seq=1078 ttl=64 time=281.585 ms 64 bytes from 192.168.50.65: seq=1238 ttl=64 time=1234.944 ms 64 bytes from 192.168.50.65: seq=1239 ttl=64 time=234.846 ms 64 bytes from 192.168.50.65: seq=1240 ttl=64 time=106.893 ms 64 bytes from 192.168.50.65: seq=1242 ttl=64 time=2411.168 ms 64 bytes from 192.168.50.65: seq=1243 ttl=64 time=1411.193 ms 64 bytes from 192.168.50.65: seq=1244 ttl=64 time=411.069 ms 64 bytes from 192.168.50.65: seq=1371 ttl=64 time=2215.494 ms 64 bytes from 192.168.50.65: seq=1372 ttl=64 time=1217.330 ms 64 bytes from 192.168.50.65: seq=1373 ttl=64 time=217.378 ms 64 bytes from 192.168.50.65: seq=1374 ttl=64 time=87.142 ms 64 bytes from 192.168.50.65: seq=1375 ttl=64 time=3285.114 ms 64 bytes from 192.168.50.65: seq=1376 ttl=64 time=2285.416 ms 64 bytes from 192.168.50.65: seq=1377 ttl=64 time=1285.353 ms 64 bytes from 192.168.50.65: seq=1378 ttl=64 time=285.204 ms 64 bytes from 192.168.50.65: seq=1491 ttl=64 time=269.096 ms 64 bytes from 192.168.50.65: seq=1492 ttl=64 time=1.053 ms 64 bytes from 192.168.50.65: seq=1495 ttl=64 time=1287.540 ms 64 bytes from 192.168.50.65: seq=1496 ttl=64 time=287.480 ms 64 bytes from 192.168.50.65: seq=1498 ttl=64 time=5038.664 ms 64 bytes from 192.168.50.65: seq=1499 ttl=64 time=4038.590 ms 64 bytes from 192.168.50.65: seq=1500 ttl=64 time=3038.611 ms 64 bytes from 192.168.50.65: seq=1501 ttl=64 time=2038.488 ms 64 bytes from 192.168.50.65: seq=1502 ttl=64 time=1038.486 ms 64 bytes from 192.168.50.65: seq=1503 ttl=64 time=38.501 ms 64 bytes from 192.168.50.65: seq=1504 ttl=64 time=61.727 ms 64 bytes from 192.168.50.65: seq=1505 ttl=64 time=1007.178 ms 64 bytes from 192.168.50.65: seq=1506 ttl=64 time=7.121 ms 64 bytes from 192.168.50.65: seq=1508 ttl=64 time=24.935 ms 64 bytes from 192.168.50.65: seq=1509 ttl=64 time=23.754 ms 64 bytes from 192.168.50.65: seq=1671 ttl=64 time=1002.748 ms 64 bytes from 192.168.50.65: seq=1672 ttl=64 time=2.601 ms 64 bytes from 192.168.50.65: seq=1673 ttl=64 time=102.808 ms 64 bytes from 192.168.50.65: seq=1674 ttl=64 time=24.394 ms 64 bytes from 192.168.50.65: seq=1675 ttl=64 time=2259.711 ms 64 bytes from 192.168.50.65: seq=1676 ttl=64 time=1259.667 ms 64 bytes from 192.168.50.65: seq=1677 ttl=64 time=259.678 ms 64 bytes from 192.168.50.65: seq=1973 ttl=64 time=324.246 ms 64 bytes from 192.168.50.65: seq=1974 ttl=64 time=21.141 ms

so I assume this usecase is just not well supported, I plan on configuring my device to only create one STA on one mac and one AP on another (e.g. mlan0 and muap0) to avoid this problem.

(If you have a suggestion to do this, input is welcome)

Thank you,

Dominique Martinet

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

ocourson — Fri, 15 Apr 2022 13:04:35 GMT

Dear Dominique,

I cannot reproduce your issue with the closer setup I can get:

- i.MX 8MQ, Linux 5.10.72

- AW-XM458-MA (9098) module

- as I do not have 2/5GHz dedicated access points, I fix the frequency in wpa_supplicant configs

Please find attached the logs I get.

Regards,

Olivier

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

martinetd — Mon, 18 Apr 2022 00:19:45 GMT

Hello @ocourson

Sorry I wasn't clear. The connections do work: from the client's perspective, both mlan0/mmlan0 are connected, I can send packets on both (you can see pings on the timeout trace to both connections, and iperfs to one on the crash one)

There are however two problems:

- packets from the outside don't work well, illustrated by the ping in the post. Most of the time it didn't get through at all, while the client was happily saying it's connected. This is better reproduced if you have two separate subnets to selectively ping either interface, as I am not sure if it could be reproduced with a single subnet (linux tends to pick any interface in the subnet to reply, leading to the router's ARP table also having both IPs on either connection and I believe you wouldn't really be using both interfaces)

- Sometimes, after a while (from 5 minutes to 1h) the driver/firmware hangs and I cannot get anything to work anymore. As illustrated by having two logs, this doesn't reproduce all the time either, I was hoping you could work it out from the traces but I believe you might be able to reproduce something similar if you leave it up for a while. In the crash reproducing case I was running iperf, but even just leaving the client connected without doing anything for lunch break was enough to hang.

Thank you.

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

ocourson — Wed, 20 Apr 2022 15:45:10 GMT

Dear Dominique,

Could you try to disable sleep mode, by adding the following parameters to moal ? "ps_mode=2 auto_ds=2"

Regards,

Olivier

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

martinetd — Thu, 21 Apr 2022 01:46:26 GMT

Hi @ocourson,

I can confirm that it works correctly with "ps_mode=2 auto_ds=2"

I've left the board up all morning with no issue: pings from outside work well, and no crash or hang happened.

Is it something you would like to fix eventually or should we document to disable sleep on our products when using both STAs ?

Thank you,

Dominique

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

ocourson — Thu, 21 Apr 2022 07:45:16 GMT

Dear Dominique,

This is the expected behavior, usually in STA mode, we only keep AP connection when needed to save power consumption.

Your usage is a bit more specific, as you want your STA to be reachable at any time from the outside, the STA is not only a client but a bit a server.

Regards,

Olivier

Re: PCI9098 (mxm_mwifiex) trigger dump when connected on both MACs

martinetd — Thu, 21 Apr 2022 08:08:15 GMT

Hello @ocourson

Thanks for the clarification, but I'm not entirely convinced:

- I don't see this problem when connecting a single STA. In that case, the first ping is a bit slower (as you describe, the MAC was asleep and saving power), but a reply comes back in the order of 100ms, and I get no or close to no packet loss. This is compared to ~7ms when the ping is initiated from the device and I have no problem with that. (Sanity check: setting ps_mode=2 auto_ds=2 in that case lowers ping values for ~1-2ms in both direction, so we can confirm it's sleep as expected)

In the example I gave earlier with two STAs, pings would not get through for a very long time, and even connections initiated from the device would stall when the reply was a bit slow (I've had iperfs stuck at 0MB/s for 30s for example, when the connection originated from the device); that doesn't strike me as normal.

- whatever happen, having the device crash (firmware dump quoted in original question) followed by unusable device should not happen.

I appreciate using both MACs as STA on different AP is not a frequent usage (and in all honesty this is just a test setup), but the behavior observed definitely doesn't look normal to me.