i.MX6 USB host port can lock up if there are CRC errors in incoming isochronous stream

[HenrikJacobsen]

Reference board: i.MX6 SabreLite

Kernel version: 3.10.17, yocto. SD card mage delivered by Freescale for testing this issue.

Additional software: “uvccapture” utlity. (Any UVC cam streaming SW, e.g. “cheese” can be used).

Additional hardware: Webcam supporting UVC (USB Video Class). Logitech conference (and others) cam used for test. And one or more USB extender cables.

Observed behavior: When USB extender cables of certain length and quality are used, it is possible to stream video from the webcam, but with errors (XactErr in the isochronous transactions indicate CRC error in the data stream). Under these conditions, the i.MX6 USB host port can lock up: It cannot be “stopped”, it doesn’t generate any interrupts, Current Connect Status is not updated). The port is of course expected to keep functioning under bit error conditions.

Steps to reproduce:

You need a connection to the webcam that is good enough to allow enumeration etc., but still generates CRC errors during streaming. It may require some experimenting to find a combination of webcam and extender cable to obtain this situation. To help in this, it is usefull to add a line of test output from uvccapture for each captured image, showing the image size in bytes. When there are CRC errors, you will either see the frame rate drop, or the image size sometimes be very small. Many other methods and tools can be used too.

Under this condition – non-zero CRC error rate in the isochronous data stream, the streaming will stop completely within some minutes. Now, the USB host port (USB Core 1) in i.MX6 is no longer functioning at all.

Now, we try to stop the USB controller by setting RS=0  (bit 0 in the USBCMD register). This should result in the HCH becoming 1 (bit 12 in the USBSTS register):

root@imx6qsabrelite:~# ./mr 0x2184384

0x18001A15

root@imx6qsabrelite:~# ./mr 0x2184344

0x000C4088   < - STS, HCH=0

root@imx6qsabrelite:~# ./mr 0x2184340

0x00010B15  <- CMD, RS=1

root@imx6qsabrelite:~# ./mw 0x2184340 0x00010B14  <- CMD, set RS=0

root@imx6qsabrelite:~# ./mr 0x2184344

0x000C4088  <- STS, HCH still 0

(mr/mw are simply memory read/write utilities).

Trying the above sequence before running the test, sets bit 12 (HCH) in the USBSTS register as expected.

The RS->HCH relation should work, no matter what. I would like very much to be proven wrong, but it seems to me that the fact that setting RS=0 does not set HCH=1 is a “smoking gun” indication of a very low level issue in the USB host controller.

An easy way to prove something is wrong is to try accessing a device (USB stick) on the other USB host port on the SabreLite. This port is “dead” too, of course, as it connects through the hub to the same USB host port on i.MX6.

More observations on the error condition:

• - The USB controller stops generating interrupts
• - The CCS (Current Connect Status) bit in the PORTSC1 register does not change when the “device” is disconnected. (The “device” is the hub chip; it can be “disconnected” by asserting it’s reset signal).
• - The FRINDEX register is still updating.
• - Often, if not always, the “lock up” occurs in connection with XactErr being reported in not just 1, but 2 transactions, within the same iTD.
• - A “port reset” (setting bit 8 in PORTSC1) can get the USB controller out of the deadlock: It can be stopped/started again etc. But it doesn’t make the USB port operational for the driver (quite naturally).
• - The issue was originally found on our own hardware design, and with kernel 3.0.35. The point to notice is that our hardware design is similar to SabreLite, except that we use a Cypress hub chip. SabreLite uses a Microchip hub. So, the issue is not related to interaction with a specific type of hub.

The only way we found to make the USB host operational again in relation to software, is to reboot.

Thanks in advance for comments and suggestions regarding this issue.

Best regards,

Henrik Jacobsen

[j_mazura]

Hi,

in our device we have probably same issue. We have "main" board with i.MX6 cpu + usb hub USB2517 and "power" board with usb hub USB2514 which is connected to first hub. On the first hub is also connected touchscreen controller, emv reader, external USB port, optionally usb modem. On the second hub there is only usb printer. During some noise the usb host port in cpu can stop working. In this state it is not possible communicate with any from peripheral devices or enumerate new device in external USB port(USB flashdisk for example). Only unbind/bind is way from this state. 

This issue I can reliable trigger if I take another our device switched to mass storage(ums 0 ... in u-boot) and connect them to USB port on USB2517 with longer cable(2 - 3 meters).

Kernel log shows:

root@cv4-0:~ # [   51.142846] usb 1-1.7: new high-speed USB device number 7 using ci_hdrc
[   56.348009] usb 1-1.7: device descriptor read/all, error -110
[   57.387945] usb 1-1-port7: cannot disable (err = -110)
[   58.428042] usb 1-1-port7: cannot reset (err = -110)
[   59.467949] usb 1-1-port7: cannot reset (err = -110)
[   60.507940] usb 1-1-port7: cannot reset (err = -110)
[   61.547945] usb 1-1-port7: cannot reset (err = -110)
[   62.587952] usb 1-1-port7: cannot reset (err = -110)
[   62.592989] usb 1-1-port7: Cannot enable. Maybe the USB cable is bad?
[   63.627945] usb 1-1-port7: cannot disable (err = -110)
[   63.633171] usb 1-1-port7: attempt power cycle
[   67.147949] usb 1-1-port7: cannot reset (err = -110)
[   68.187961] usb 1-1-port7: cannot reset (err = -110)
[   69.227967] usb 1-1-port7: cannot reset (err = -110)
[   70.267953] usb 1-1-port7: cannot reset (err = -110)
[   71.307953] usb 1-1-port7: cannot reset (err = -110)
[   71.312984] usb 1-1-port7: Cannot enable. Maybe the USB cable is bad?
[   72.347943] usb 1-1-port7: cannot disable (err = -110)
[   73.387940] usb 1-1-port7: cannot reset (err = -110)
[   74.427952] usb 1-1-port7: cannot reset (err = -110)
[   75.467961] usb 1-1-port7: cannot reset (err = -110)
[   76.507950] usb 1-1-port7: cannot reset (err = -110)
[   77.547945] usb 1-1-port7: cannot reset (err = -110)
[   77.552971] usb 1-1-port7: Cannot enable. Maybe the USB cable is bad?
[   78.587944] usb 1-1-port7: cannot disable (err = -110)
[   78.593167] usb 1-1-port7: unable to enumerate USB device
[   79.627943] usb 1-1-port7: cannot disable (err = -110)

After that USB is not usable. In dynamic debug of ci_hdrc I can see some XactErr errors.

I tried kernel 4.19 from kernel.org and kernel imx_4.19.35_1.1.0 from linux-imx - i.MX Linux kernel . In both kernels it is same.

Jan

[PeterKamp]

Someone found out more about this issue? I have the same problem with a USB2517B hub (lock of i.mx6 root hub) when using a long cable (3 - 5  meters) (kernel 3.10.77) and frequent connect/disconnect of tablet. Failure seems to be the same as above, hardware reset of hub does not help. The only thing to get it back working is reboot or rebinding the driver.

echo ci_hdrc.1 > /sys/bus/platform/drivers/ci_hdrc/unbind

echo ci_hdrc.1 > /sys/bus/platform/drivers/ci_hdrc/bind

[chandler_grisco]

I've been experiencing the same issue on latest kernel (5.0-r7) on a couple i.MX6-based machines with webcams.  It seems to be induced by various things like electrical noise in the room or failing devices / bad cables.  Rebinding is also the only way to reset in my case.  I've reported it on the linux-usb mailing list, will see where that goes: 'Bus noise periodically causes ci_hdrc IRQ lockup' - MARC 

[jontsmith]

Has NXP issued a fix for this, beyond just sending the unbind/bind ?

[tim_plant]

Apologies if I'm approaching this the wrong way, I've not raised an issue in the community before, but does anyone know if this issue has been fixed? I'm currently experiencing a similar looking issue on Linux and WinCE products.

[bjornbosell]

Hi,

We have seen similar issues with a cx231xx based USB device.

Have you found a solution or work around?

Regards,

Bjorn Bosell