i.MX7D USB HSIC Hub (USB3503) enumeration variability

[billgessaman]

Hardware / Software Summary:

• Custom i.MX7D board based on MCIMX7SABRE board
• Silicon rev 1.2 (MCIMX7D5EVM10SC)
• BSP L4.1.15-1.0.0-GA primarily but also tested with L4.1.15-2.0.0-GA with difference seen
• Microchip USB3503T-I/ML 3-port USB Hub with HSIC interface
• Variation in cold boot / reboot enumeration of USB Hub
• Two out of 50 systems *always* fail to enumerate USB Hub on cold boot / reboot, but those two systems *always* enumerate USB Hub properly on resume from suspend-to-RAM state (either PMIC standby or LPSR mode).

Our application required more USB Host ports than were implemented by the i.MX7 SABRE board, so the Microchip USB3503 Hub was included to add the two additional ports that we needed. The guidelines for hardware design for both the i.MX7D and the USB3503 were taken into account and the critical routing of the DATA and STROBE signals was implemented to be less than 1" with their length matched to within just a little more than 5 mils. Most boards work correctly 99% of the time, but once in a while I have seen a system fail to enumerate the USB Hub. In these cases, a reboot (equivalent to a cold boot because the PMIC is forced off then turned back on again) will result in the USB Hub enumerating correctly.

This was true until I discovered two boards that fail 99.9% of the time on a cold boot / reboot. With these boards, I actually have something to try to find the root cause. First the high level visibility of the issue in the console serial output is:

ci_hdrc ci_hdrc.2: EHCI Host Controller
ci_hdrc ci_hdrc.2: new USB bus registered, assigned bus number 2
ci_hdrc ci_hdrc.2: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 4.1.15+ ehci_hcd
usb usb2: SerialNumber: ci_hdrc.2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
usb 2-1: new high-speed USB device number 2 using ci_hdrc
usb 2-1: device no response, device descriptor read/64, error -71
usb 2-1: device no response, device descriptor read/64, error -71
usb 2-1: new high-speed USB device number 3 using ci_hdrc
usb 2-1: device no response, device descriptor read/64, error -71

.......

A normal system with a USB Flash drive on a USB Hub downstream port looks like this:

ci_hdrc ci_hdrc.2: EHCI Host Controller
ci_hdrc ci_hdrc.2: new USB bus registered, assigned bus number 2
ci_hdrc ci_hdrc.2: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 4.1.15+ ehci_hcd
usb usb2: SerialNumber: ci_hdrc.2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
usb 2-1: new high-speed USB device number 2 using ci_hdrc
usb 2-1: New USB device found, idVendor=0424, idProduct=3503
usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 2 ports detected
usb 2-1.2: new high-speed USB device number 3 using ci_hdrc
usb 2-1.2: New USB device found, idVendor=0781, idProduct=5595
usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-1.2: Product: Ultra USB 3.0
usb 2-1.2: Manufacturer: SanDisk
usb 2-1.2: SerialNumber: 4C531001641115101474
usb-storage 2-1.2:1.0: USB Mass Storage device detected
.......

Over the last 2 to 3 weeks I have taken the following steps to isolate the cause of the problem:

1. Verified that the accuracy and jitter of the 24 MHz reference clock from the i.MX7 to the USB3503 is within the specifications for the USB3503.
2. Verified all related power supplies on both i.MX7 and USB3503.
3. Verified related signal levels and general timing relationships against a known "good" board.
4. Technician inspected USB3503 solder quality on both boards that always fail, then both Hubs were reflowed to make sure there were no bad solder joints.  HINT: When these boards were booted very soon after reflow, they both worked correctly!  As they cooled off to room temp again, they returned to always failing.  Further experimentation with heat / cold showed a strong thermal relationship between working / failing respectively.
5. New USB3503T-I/ML parts were purchased and the Hubs on both "bad" boards were replaced.  There was no change in behavior and they continued to be sensitive to temperature in the same way.
6. I then started trying to understand what would trigger the "error -71" which is an EPROTO error in the Linux kernel.  I'm not sure why I tried to put a "bad" system into suspend-to-RAM and then resume, but this lead to the revelation that the USB Hub *always* enumerates properly during resume!  I have tried this with Linux configured to do a powered "standby" on the USB Hub, and I have also used LPSR mode to suspend /resume which turns off all power supplies to the USB Hub.  Both work equally well on the "bad" boards to re-initialize the USB Host interface to the USB3503!
7. I chased the source of the "error -71" to lower levels of the USB drivers and found that the Chipidea HDRC driver sees the following error and retries 32 times before returning the -EPROTO error.
   

   ci_hdrc ci_hdrc.2: detected XactErr len 0/8 retry 1

8. Looking deeper into ehci_hcd.c and getting some visibility on what ehci register read / write operations are being done, I found that a "bad board" is detecting that the UEI bit of the USB2_USBSTS register is being set to indicate that an USB error interrupt has been detected. The following debug output (non-standard code added by me) shows the first few register read / write operations when it tries to enumerate the USB3503 Hub.
   On a "bad board":

   usb 2-1: new high-speed USB device number 2 using ci_hdrc
   ehci_readl: 0xf5b30144 = 0x00000080
   ehci_writel: 0xf5b30140 = 0x00010b25
   ehci_readl: 0xf5b30140 = 0x00010b25
   ehci_readl: 0xf5b30144 = 0x00008082           <<-- read of USB2_USBSTS with UEI bit set
   ehci_writel: 0xf5b30144 = 0x00000002           <<-- write of USB2_USBSTS to clear UEI bit
   ehci_readl: 0xf5b30140 = 0x00010b25
   ci_hdrc ci_hdrc.2: detected XactErr len 0/8 retry 1
   ehci_readl: 0xf5b30144 = 0x00008082
   ehci_writel: 0xf5b30144 = 0x00000002
   ehci_readl: 0xf5b30140 = 0x00010b25

   On a normally good board or during a resume from suspend:

   usb 2-1: new high-speed USB device number 6 using ci_hdrc
   ehci_readl: 0xf5b30144 = 0x00000080
   ehci_writel: 0xf5b30140 = 0x00010b25
   ehci_readl: 0xf5b30140 = 0x00010b25
   ehci_readl: 0xf5b30144 = 0x00048081
   ehci_writel: 0xf5b30144 = 0x00000001
   ehci_readl: 0xf5b30140 = 0x00010b25
   usb 2-1: usb_start_wait_urb length=18, retval=0
   ehci_readl: 0xf5b30184 = 0x0a001205
   ehci_writel: 0xf5b30184 = 0x0a001301
   ehci_readl: 0xf5b30140 = 0x00010b25
   usb usb2: usb_start_wait_urb length=0, retval=0

At this point, I've run out of places to look for new information so I'm stuck and need someones deep insight into what can trigger this error and particularly why it only happens on a cold boot / reboot.  What is different for a suspend / resume than what is done on a cold boot / reboot?  Why does the interface to the USB3503 work flawlessly if you can get past the initial enumeration, but that part is variable from board to board and between boot cycles?

Thanks,

Bill Gessaman

[demoniacmilk]

Hello!

I am currently designing an MX7D based system as well, needing 3 USB ports +1 for config and one for an RF module. It seemed to be a good solution to use a 3 port HSIC HUB and the two OTG ports.

When looking for a 3 Port+HSIC HUB I ran into the USB3503 as well and thought it'd be the perfect device for my needs.

However, on running into this thread, some issues are to be expected that were not being resolved despite the effort you guys put in. So there are three options for me:

1. use another HSIC HUB
2. dont use HSIC
3. use the USB3503 anyways

The USB3503 is exactly what I was looking for and using HSIC is definitively my prefered approach here, but I don't want to hunt down issues for mutliple days/weeks ofc. 

2) Yes we do have customer's using this interface.

I wonder: What do these customers use this interface for? Any Hub? What device would that be?

Does the HSIC interface enumerate during power up only? Or may a reset of the attached device trigger a new numeration as well?

[gopinathsriniva]

Hi Bill Gessaman,

Thank you very much for your prompt and detailed response.

Regards,

Gopinath S

[art]

Setting of the USB_OTG2_USBSTS[UEI] bit indicates that the last USB transaction resulted in an error for some reason.

To make me able to carefully study the case, please provide the complete schematic of your board.

Also, please specify some more details on the issue.

1) You mentioned that you tried both L4.1.15-1.0.0-GA and L4.1.15-2.0.0-GA BSPs. Is there any difference in the behaviour between these BSPs?

2) When the issue occurs (as far as I understand, it occurs at power-on boot only), is any USB Device connected to the Hub's downstream port(s)?

3) If the answer to 2) is "yes", does any issue occur if there is no device connected at the boot time?

4) If the answer to 2) is "yes", does the issue occurence depend on an USB Device(s) connected?

Have a great day,
Artur

[billgessaman]

Hi Artur,

I appreciate your interest in this issue.  I'm reluctant to share the "complete" schematic on a public forum and I think it would be necessary for me to leave out a couple of details that might compromise our product security.  I will work on this and follow up.  I think the board layout is as critical as the schematic, so an image (or images) of the i.MX7 to USB3503 layout may also be of interest.

Answers to your questions:

1) The observed behavior was the same with the two BSPs that I listed above.  I didn't see any Git repo commits that made me think that the 2.0.0 release had significant changes in this area, but I thought it was worth trying anyway.

2) You are correct that it happens only at power-on boot.  Technically it also happens when a Linux "reboot" command is executed, because our system turns a reboot into a power off / power on sequence by turning off the PMIC for approximately 200ms and then turning it back on.  The enumeration error with the external USB3503 Hub does not seem to be affected by whether a USB device is connected to a downstream port on the USB3503 Hub.  I have tested with an without having a USB flash drive on one of the downstream ports.

3) Yes - the error happens when no device is connected at boot time.

4) I think you are asking whether the error is dependent on the type of USB device connected to the hub.  I have tested with several flash drives as well as our custom measurement modules which are effectively USB devices and use a downstream port on the Hub to connect into the system.

Thanks,

Bill

[mwinker]

Hello,

I know this thread is very old, but I finally got the USB today running with Kernel 5.10.

Basically, our issue was the following:

In the device tree, we specified the reference clock. The code actually seems to stop and restart the clock with clk_prepare_enable(). Therefore, we added 4ms sleep afterwards.

Another thing is that we needed to initialize the reset GPIO as low to bring the chip through a proper reset after restarting the clock.

I believe, other folks which do have a crystal or other "stable" clock input do not face any issue. Only if a clock output of the CPU is used.

This is the patch I needed:

diff --git a/drivers/usb/misc/usb3503.c b/drivers/usb/misc/usb3503.c
index 48099c6bf04c..a003a71c8e11 100644
--- a/drivers/usb/misc/usb3503.c
+++ b/drivers/usb/misc/usb3503.c
@@ -217,6 +217,9 @@ static int usb3503_probe(struct usb3503 *hub)
return err;
}

+ /* Wait T_HUBINIT == 4ms for hub logic to stabilize */
+ usleep_range(4000, 10000);
+
property = of_get_property(np, "disabled-ports", &len);
if (property && (len / sizeof(u32)) > 0) {
int i;
@@ -247,7 +250,7 @@ static int usb3503_probe(struct usb3503 *hub)
if (hub->connect)
gpiod_set_consumer_name(hub->connect, "usb3503 connect");

- hub->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_HIGH);
+ hub->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
if (IS_ERR(hub->reset))
return PTR_ERR(hub->reset);
if (hub->reset) {