WiFi crash on #SD8997 with error 'cmd53 read error=-110' during mlan0 START SCAN operation

[Ankitz_Patel]

Hello All,

We have designed custom board, which is having IMX6DL dual core processor running linux Kernel 4.1.15. we have connected eMMC and Wi-Fi module (Product:muRata ##LBEE5XV1YM having NXP chipset 88W8997) on SDIO interface

We are facing WiFi crash issue with error 'cmd53 read error=-110' during mlan0 START SCAN operation by wpa_supplicant (version v2.5). We have tried many variants of NXP drivers with these experiments to rule out the issue but we had no luck... Currently we are using latest wifi drivers (16.92.21.p76.2) but issue still persists.

We have figured out a steps to reproduce crash by removing the Antenna and configuring AP so that wpa_supplicant start running SCAN operations in background. We are getting Wi-Fi crash very often after this steps... Can anybody from the community provides pointers, suggestion or resolutions for the same?

We are planning to upgrade wpa_supplicant from v2.5 to v2.10 if that can help us to resolve this issue. We will post the updates on the same.
Is it possible to optimise the number of active or sched scan operations to minimize the issue? Please suggests

Please note that We are only using Wi-Fi interface (on SDIO) and not using bluetooth interface (on UART) in our custom board.

eMMC configurations from linux kernel:
clock: 52000000 Hz
actual clock: 49500000 Hz
vdd: 21 (3.3 ~ 3.4 V)
bus mode: 2 (push-pull)
chip select: 0 (don't care)
power mode: 2 (on)
bus width: 3 (8 bits)
timing spec: 8 (mmc DDR52)
signal voltage: 0 (3.30 V)

Wi-Fi configurations from linux kernel:
clock: 50000000 Hz
actual clock: 49500000 Hz
vdd: 21 (3.3 ~ 3.4 V)
bus mode: 2 (push-pull)
chip select: 0 (don't care)
power mode: 2 (on)
bus width: 2 (4 bits)
timing spec: 2 (sd high-speed)
signal voltage: 0 (3.30 V)

Wi-Fi Driver version:
SD-WLAN-UART-BT-8997-LNX_6_1_1-IMX8-16.92.21.p76.2-16.92.21.p76.2-MM5X16368.P2-MGPL

# uname -a
Linux mainboard-sbs 4.1.15-2.0.0-ga+yocto+g6d75df8 #1 SMP PREEMPT Mon Apr 10 20:53:27 IST 2023 armv7l armv7l armv7l GNU/Linux

# cat /etc/wpa_supplicant.conf
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1
network={
key_mgmt=WPA-PSK WPA-EAP WPA-NONE
}
network={
ssid="Arlo_VT"
psk="Arlo#2022"
}

Wi-Fi Driver Loading:

# cd /lib/modules/4.1.15-2.0.0-ga+yocto+g6d75df8/extra/
#ifconfig uap0 down; ifconfig mlan0 dow
#insmod mlan-sd8997.k
#insmod /lib/modules/4.1.15-2.0.0-ga+yocto+g6d75df8/extra/moal.ko `cat /etc/modules-load.d/moal.conf

Initiate Connection/SCAN:

# wpa_supplicant -B -D nl80211 -i mlan0 -c /etc/wpa_supplicant.conf
# udhcpc -i mlan0 -t 1 -T 3 -f -B

# lsmod
Module Size Used by
moal 618496 1
brcmfmac 172032 0
brcmutil 16384 1 brcmfmac
usb_f_ecm 16384 1
g_ether 16384 0
usb_f_rndis 24576 2 g_ether
u_ether 20480 3 usb_f_ecm,usb_f_rndis,g_ether
libcomposite 45056 3 usb_f_ecm,usb_f_rndis,g_ether
bt8xxx 86016 0
mlan 544768 1 moal
cfg80211 454656 2 moal,brcmfmac

# dmesg

[ 715.053933] wlan: mlan0 START SCAN
[ 730.032289] mmc1: Timeout waiting for hardware interrupt. retries left=0 opcode=0
[ 730.038711] cmd53 read error=-110
[ 730.040857] wlan: cmd53 read failed: -1 ioport=0x10009 retry=0
[ 730.045565] Set Term cmd53 failed
[ 740.052296] wlan: scan timeout!
[ 740.054234] ------------mlan_debug_info-------------
...
[ 740.074075] mmc1: Timeout waiting for hardware interrupt. retries left=0 opcode=0
[ 740.080359] Timeout cmd id = 0x0, act = 0x0
[ 740.083345] cmd53 read error=-110
[ 740.085451] wlan: cmd53 read failed: -1 ioport=0x10009 retry=1
[ 740.090070] Set Term cmd53 failed
...
[ 740.348056] ------------mlan_debug_info End-------------
[ 740.352079] Driver version = SD8997----16.92.21.p76.2-MM5X16368.p2-(FP92)
...
[ 740.401141] Cancel all pending cmd!
[ 740.403399] IOCTL failed: a9783000 id=0x10000, sub_id=0x10003 action=1, status_code=0x80000007
[ 750.112298] mmc1: Timeout waiting for hardware interrupt. retries left=0 opcode=0
[ 750.118754] cmd53 read error=-110
[ 750.120937] wlan: cmd53 read failed: -1 ioport=0x10009 retry=2
[ 750.125683] Set Term cmd53 failed
[ 750.127755] Card to host failed: int status=0x1
[ 750.131087] read CFG reg failed
[ 750.133132] write CFG reg failed

[ 750.135151] read CFG reg failed

cc: @sanketparekh

[Attached detailed dmesg logs]

[Ankitz_Patel]

Welcoming suggestions... If anyone has faced same issue and can share methods to tune/reduce the wifi dump occurrence will be a great pointer.