HUB USB Disconnection, detect and reload without reboot

[xgarcia]

Our machine uses WiFi & 3G communications. Both devices share the same USB connection through a HUB.
Given a time it seems to disconnect both devices and the firmware is unable to recover them.
Our goal is to recover those interfaces without needing to reboot.

Comparing a well-working machine and non-working one we appreciate these differences:

root@well-working:~# lsusb -t
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ci_hdrc/1p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/2p, 480M
|__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=ath9k_htc, 480M
|__ Port 2: Dev 4, If 0, Class=Vendor Specific Class, Driver=option, 480M
|__ Port 2: Dev 4, If 1, Class=Vendor Specific Class, Driver=option, 480M
|__ Port 2: Dev 4, If 2, Class=Vendor Specific Class, Driver=option, 480M
|__ Port 2: Dev 4, If 3, Class=Vendor Specific Class, Driver=option, 480M
|__ Port 2: Dev 4, If 4, Class=Communications, Driver=cdc_ether, 480M
|__ Port 2: Dev 4, If 5, Class=CDC Data, Driver=cdc_ether, 480M

root@well-working:~# ls -l /sys/bus/usb/devices/
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-0:1.0 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-0:1.0
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.1 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.1
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.1:1.0 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.1/1-1.1:1.0
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2:1.0 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2/1-1.2:1.0
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2:1.1 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2/1-1.2:1.1
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2:1.2 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2/1-1.2:1.2
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2:1.3 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2/1-1.2:1.3
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2:1.4 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2/1-1.2:1.4
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1.2:1.5 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1.2/1-1.2:1.5
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-1:1.0 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-1/1-1:1.0
lrwxrwxrwx 1 root root 0 Jan 1 1970 usb1 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1

root@non-working:~# lsusb -t
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ci_hdrc/1p, 480M

root@non-working:~# ls -l /sys/bus/usb/devices/
lrwxrwxrwx 1 root root 0 Jan 1 1970 1-0:1.0 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1/1-0:1.0
lrwxrwxrwx 1 root root 0 Jan 1 1970 usb1 -> ../../../devices/soc0/80080000.ahb/80090000.usb/ci_hdrc.1/usb1

When a machine loses it's WiFi & 3G all the USB devices related dissapear.
We have already tried changing it's power related params values, with no success.

root@non-working:~# ls /sys/bus/usb/devices/usb1/power/
active_duration connected_duration runtime_active_time
autosuspend control runtime_status
autosuspend_delay_ms level runtime_suspended_time

Here is an example of a disconnection event:

[1795405.092245] usb 1-1: USB disconnect, device number 2
[1795405.113015] usb 1-1.1: USB disconnect, device number 3
[1795405.287948] option1 ttyUSB3: option_instat_callback: error -71
[1795405.893171] ath: phy0: Chip reset failed
[1795405.915900] ath: phy0: Unable to reset channel (2412 Mhz) reset status -22
[1795405.948908] ath: phy0: Unable to set channel
[1795406.054290] ath: phy0: Failed to wakeup in 500us
[1795406.134293] ath: phy0: DMA failed to stop in 10 ms AR_CR=0xa83aa0c7 AR_DIAG_SW=0xa83aa0c7 DMADBG_7=0xa83aa0c7
[1795406.262963] ath: phy0: Failed to wakeup in 500us
[1795407.423023] ath: phy0: DMA failed to stop in 10 ms AR_CR=0x803aa0c7 AR_DIAG_SW=0x803aa0c7 DMADBG_7=0x803aa0c7
[1795408.171040] usb 1-1.1: ath9k_htc: USB layer deinitialized
[1795408.341405] usb 1-1.2: USB disconnect, device number 4
[1795408.451976] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
[1795408.527829] option 1-1.2:1.0: device disconnected
[1795408.612111] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
[1795408.662332] option 1-1.2:1.1: device disconnected
[1795408.759173] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
[1795408.809315] option 1-1.2:1.2: device disconnected
[1795408.913052] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3
[1795408.965253] option 1-1.2:1.3: device disconnected
[1795409.026255] cdc_ether 1-1.2:1.4 eth0: unregister 'cdc_ether' usb-ci_hdrc.1-1.2, CDC Ethernet Device

At this time, our only way to recover those interfaces is to reboot the whole machine... When the OS starts again and loads its modules the interfaces are detected again:

[ 5.107845] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 5.114635] ehci-platform: EHCI generic platform driver
[ 5.123058] usbcore: registered new interface driver usb-storage
[ 5.131111] usbcore: registered new interface driver usbserial
[ 5.137924] usbcore: registered new interface driver option
[ 5.145798] usbserial: USB Serial support registered for GSM modem (1-port)
[ 5.202460] ci_hdrc ci_hdrc.1: EHCI Host Controller
[ 5.209070] ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 1
[ 5.224634] ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
[ 5.236611] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 5.243661] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.251115] usb usb1: Product: EHCI Host Controller
[ 5.256220] usb usb1: Manufacturer: Linux 3.17.2-fslc+g6725f55 ehci_hcd
[ 5.263414] usb usb1: SerialNumber: ci_hdrc.1
[ 5.288768] hub 1-0:1.0: USB hub found
[ 5.293575] hub 1-0:1.0: 1 port detected

[ 5.632202] usb 1-1: new high-speed USB device number 2 using ci_hdrc
[ 5.785868] usb 1-1: New USB device found, idVendor=0424, idProduct=2422
[ 5.793740] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 5.826896] hub 1-1:1.0: USB hub found
[ 5.833666] hub 1-1:1.0: 2 ports detected
[ 6.136125] usb 1-1.1: new high-speed USB device number 3 using ci_hdrc
[ 6.257841] usb 1-1.1: New USB device found, idVendor=0cf3, idProduct=9271
[ 6.265532] usb 1-1.1: New USB device strings: Mfr=16, Product=32, SerialNumber=48
[ 6.274210] usb 1-1.1: Product: SKW17
[ 6.279177] usb 1-1.1: Manufacturer: SKYLAB
[ 6.284601] usb 1-1.1: SerialNumber: 12345
[ 6.426135] usb 1-1.2: new high-speed USB device number 4 using ci_hdrc
[ 6.528004] usb 1-1.2: New USB device found, idVendor=05c6, idProduct=9003
[ 6.535563] usb 1-1.2: New USB device strings: Mfr=4, Product=3, SerialNumber=5
[ 6.544201] usb 1-1.2: Product: UMTS/HSPA Module
[ 6.550141] usb 1-1.2: Manufacturer: Quectel, Incorporated
[ 6.580942] option 1-1.2:1.0: GSM modem (1-port) converter detected
[ 6.606089] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB0
[ 6.636850] option 1-1.2:1.1: GSM modem (1-port) converter detected
[ 6.661615] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB1
[ 6.689028] option 1-1.2:1.2: GSM modem (1-port) converter detected
[ 6.711116] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB2
[ 6.742213] option 1-1.2:1.3: GSM modem (1-port) converter detected
[ 6.765995] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB3

Hardware related observed details:

We are unable to define what causes those desconnections. We suspect about a HW related issue.
Analising the HUB's crystal clock with an oscillator we can observe an anomalous behaviour:

https://ibb.co/c2py54j

When the firmware is unable to detect & recover the interfaces, HUB's crystal clock seems to loop.
Only with the kernel module reload at boot seems to be reinitialised and works properly.

[Bio_TICFSL]

Hello xgarcia,

May the following can help:

https://itectec.com/ubuntu/ubuntu-how-do-you-reset-a-usb-device-from-the-command-line/

Regards

[xgarcia]

I tried crosscompiling that code and using it, with no success...

root@non-working:~# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

root@non-working:~# resetUSB /dev/bus/usb/001/001
Resetting USB device /dev/bus/usb/001/001
Error in ioctl: Is a directory

The "Is a directory" error is normal when using usb reset on root hubs... this root hub is the only displayed usb device when lsusb command is executed.

Our goal is to recover the remaining missing usb devices without needing to reboot the OS.