i.MX6 “PHY link never came up”

[eckhoffj]

Periodically we are seeing an issue which causes the Wi-Fi to fail to load due to a PCI bus link issue.

The tell-tale sign of the issues is seeing the “PHY link never came up” message in the console output when booting. When this message is displayed the i.MX6 was unable to establish a link with the AW-CH397 WiFi module.

Device is a Custom board using the Variscite  VAR-SOM-MX6 http://www.variscite.com/products/system-on-module-som/cortex-a9/var-som-mx6-cpu-freescale-imx6

We began seeing a significant failure rate on the PCIe link initialization. After weeks of testing we were unable to triage the issue. Once the issue occurred on a board/SOM combo, it would often persist until that board and SOM were left alone (as if an environmental or temperature issue impacted it). Trying the same board and/or SOM a day later, and it would work.

A fix came across in uboot from Variscite concerning core voltage settings. This solved our problem at the time, and we no longer saw the link failures in a significant amount. Although they do still happen, just more sporadically and more often on specific boards.

Software and Core Voltages

Looking through Variscite’s commit history a commit to uboot was made regarding core voltages:

https://github.com/varigit/uboot-imx/commit/06fd56c8f6fe4d7781f9e72852f284f81542d0ff

https://github.com/varigit/uboot-imx/commit/05534a1e791699b365e8445819460ef6ceca5bf2

Applying these patches made the PCIe problem appear to go away. In reality the problem became less frequent but persisted.

Hardware and PHYs

The Freescale i.MX6 PCIe PHY is compatible to PCIe v2.0.

The Azurewave AW-CH397 PCIe PHY is compatible to PCIe v3.0. The Azurewave part is based on the Marvell 88W8897.

The Azurewave interface speed is 2.5Gbps so it only requires a PCIe v1.0 compatible link partner. Freescale

Signal Integrity and Impedance

The PCIe v3.0 recommended impedance range is 80-120 ohm differential. Freescale support person mentioned 85 ohms in the Freescale forum discussion below and in their Design guide, but Marvell mentions 100 ohms differential. Most importantly it was confirmed with Variscite that they routed their board at 100 ohms differential and so the Carrier Board was designed to match that.

Signal Integrity was measured with a high speed scope and discussed at length in the following two posts:

https://community.freescale.com/message/537250#537250

http://electronics.stackexchange.com/questions/180012/pcie-diagnosing-and-improving-an-eye-diagram

Initial investigation into the issue lead to posts on the internet with the same error message. Some contained suggest patches:

http://www.variscite.com/support-forum/viewtopic.php?t=109

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c

index 1c781f9..d7f1a40 100644

--- a/drivers/pci/host/pci-imx6.c

+++ b/drivers/pci/host/pci-imx6.c

@@ -442,7 +442,7 @@ static int imx6_pcie_wait_for_link(struct pcie_port *pp)

        int count = 200;

        while (!dw_pcie_link_up(pp)) {

-               udelay(100);

+               udelay(1000);

                if (--count)

                        continue;

Message was edited by: Josh Eckhoff

[selamicandar]

I have a custom board based on sabreauto board. RTL8111 gigabit ethernet controller ic is connected to pcie interface (gen1, 2.5G) on imx6quad. Kernel version is 3.14.28. Most of boots, no problem, link ok, second ethernet interface works normaly. But speacily some boards have pcie link issue.

I tried to many solution, increasing VDD_SOC voltage, add delay to link wait function and many. I have tested more than 15 boards. Some boards works without problem, %100 success. But some of them %80, some of %50, unstable.

I have revized my PCB desing and re-design PCIE route. imx6 to RTL8111 length is less than 35mm. (rx, tx and refclk) Differential pair design ok. (impedance, length diff. between pairs less than 5 mils, the design was made in accordance with NXP "Hardware Design Considerations for PCI Express®andSGMII AN307" document)

I suspect the reference clock. I know internal clock is no compience Gen2. Is it realy suitable for pcie gen1?

Is there any new suggestion/solution?

Thanks.

[eckhoffj]

Curt and George will take over this issue - if Freescale could comment on their existing replies and I will update the thread going forward

[karina_valencia]

Yuri​ can you help to continue with the follow up?

[Yuri]

Hello,

  Please try adjust parameters of PCIe_PHY  by changing the IOMUXC_GPR8 register.

You may refer to app note http://cache.nxp.com/files/32bit/doc/app_note/AN4784.pdf

for details.

Regards,

Yuri.

[eckhoffj]

We had adjusted these parameters in the past as part of polling compliance testing (see PCIe, diagnosing and improving eye diagram for more details) using recommended values from AN4784, as well as exhaustive adjustments to deemph in order to improve our results.

We were definitely able to make it fail more often by adjusting these parameters, but wasn't able to improve.  Will revisit this as well..

Any ideas on the VDD_SOC change and how/why that appears to have solved it? Unfortunately we are now out of spec for VDD_SOC@1.425

[cblack]

Been playing with the Core Voltage settings, which does seem to be affecting the behavior of the link. Originally sourced changes from Variscite:

https://github.com/varigit/uboot-imx/commit/05534a1e791699b365e8445819460ef6ceca5bf2

https://github.com/varigit/uboot-imx/commit/06fd56c8f6fe4d7781f9e72852f284f81542d0ff

These 2 commits raise the setting of SW1AB and SW1C of the PMIC.  (VDDARM and VDDSOC)

I raised them a bit more since the comments didn't match the value - from 0x29 (1.325) to 0x2B (1.375), and then the PCIe started working.  Although, it still does fail once in a while so the problem is not fixed.

In an effort to understand what is going on here, any ideas as to what might be going on here?

[robyf]

Hi Curt,

I'm having similar problem with a TI XIO2001 PCIe-to-PCI bridge, never able to get it up, still link never came up.

I've also played a bit with SW1AB/C, default was 0x2d, but also dropping down to 0x29 or 0x2b doesn't change

anything.

U-Boot 2014.04-imx_v2014.04_3.14.28_1.0.0_ga+g88123ea (Feb 05 2016 - 16:57:25)

CPU:   Freescale i.MX6Q rev1.5 at 792 MHz

CPU:   Temperature 38 C, calibration data: 0x5624d569

Reset cause: POR

Board: Janas iMX6Q (ID:e315c0641d0f31d4)

I2C:   ready

DRAM:  2 GiB

MMC:   FSL_SDHC: 0, FSL_SDHC: 1

*** Warning - bad CRC, using default environment

phy link never came up

DEBUG_R0: 0x00c80b00, DEBUG_R1: 0x08200000

LTSSM current state: 0x0 (S_DETECT_QUIET)

PIPE transmit K indication: 0

PIPE Transmit data: 0xc80b

Receiver is receiving logical idle: no

Second symbol is also idle (16-bit PHY interface only): no

Currently receiving k237 (PAD) in place of link number: no

Currently receiving k237 (PAD) in place of lane number: no

Link control bits advertised by link partner: 0x0

Receiver detected lane reversal: no

TS2 training sequence received: no

TS1 training sequence received: no

Receiver reports skip reception: no

LTSSM reports PHY link up: no

A skip ordered set has been transmitted: no

Link number advertised/confirmed by link partner: 0

Application request to initiate training reset: no

PIPE transmit compliance request: no

PIPE transmit electrical idle request: yes

PIPE receiver detect/loopback request: no

LTSSM-negotiated link reset: yes

LTSSM testing for polarity reversal: no

LTSSM performing link training: no

LTSSM in DISABLE state; link inoperable: no

Scrambling disabled for the link: no

In:    serial

Out:   serial

Err:   serial

Found PFUZE100! deviceid=10,revid=21

mmc1(part 0) is current device

Net:   FEC [PRIME]

Cheers,

Roberto Fichera.

[cblack]

More information on DEBUG_R0 & DEBUG_R1.  DEBUG_R0 changes a bit from boot to boot

Boot #1:

[    0.412169] imx6q-pcie 1ffc000.pcie: phy link never came up

[    0.412197] imx6q-pcie 1ffc000.pcie: DEBUG_R0: 0x002cf742, DEBUG_R1: 0x08000000

Boot #2:

[    0.408695] imx6q-pcie 1ffc000.pcie: phy link never came up

[    0.408723] imx6q-pcie 1ffc000.pcie: DEBUG_R0: 0x00000602, DEBUG_R1: 0x08000000

...

[Yuri]

Hello,

  What about earlier comment ?

" Your clock solution may be used for the gen1, but for the gen2 external

PCIe 2.0/3.0 clock generator with 2 HCSL outputs should be applied".

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[georgekellerman]

As mentioned above the Azurewave peripheral is PCIe v3.0  compatible but it only operates at the Gen 1 rate (2.5Gbps). Since it operates at the Gen 1 rate I don't see why we would need the 2 HCSL outputs that I believe you are saying is necessarily to achieve max Gen 2 and Gen 3 rates. Do you agree?

How can we debug this further to actually determine how and why the link is failing?

[Yuri]

Hello,

Is it possible to check the PCIe with one of tested devices,

described in Chapter "i.MX 6 PCI Express Root Complex Driver"

in Linux RM ?

Regards,

Yuri.

[georgekellerman]

Yuri,

Does the debug register dump provide any insight into what our issue might be?

We have exploring this in greater detail and have found that adjustments to the output of SW1CLX on the PMIC (which feeds VDDSOC_IN on the i.MX6) impacts the PCIe PHY Link establishment.

We increased this voltage to 1.425. The VDDSOC_IN feeds the LDO_SOC LDO. For the SOM we are using this LDO is set to bypass mode, and the drop across it was about 30mV, so the voltage that actually gets to PCIE_VP and PCIE_VPTX is 1.395V. This is above the max input voltage for those pins (1.3V) but yet still improves the link establishment. During this PCIE_VPH is left at 2.5V.

• Does this provide any clues as to what could be the issues?
• How are PCIE_VP, PCIE_VPTX, and PCIE_VPH related?

[Yuri]

Hello,

  let me look at the schematic of used VAR-SOM-MX6, AW-CH397 WiFi,

connection between them and datasheet of the chipset with PCIe interface.

You can create request to send the documents :

1)	Please open www.nxp.com
2)	On the top level menu, select Support > Sales and Support (http://www.nxp.com/support/sales-and-support:SUPPORTHOME).
3)	On the bottom of the page, select Hardware & Software.
4)	Register with your business email to access the technical NXP online support.
5)	A verification email will be sent to your account. Click the embedded link to verify your access.
6)	On the NXP online support page, select Contact Support from the top menu and click “submit a new case” to start the process.

Regards,

Yuri.

[georgekellerman]

Yuri, I sent the file. As mentioned I do not have the schematics for the 3rd party SOM we are using.

I would also re-direct attention to my comment about the PCIE-VPTX voltage increase appearing to solve the issue for us. What does that voltage supply in the PCIe peripheral? What underlying issue could be solved by raising that voltage? Will keeping it at 1.395V damage the peripheral?

[georgekellerman]

No we can't. The implementation is effectively chip-to-chip and does not use a PCIe connector. Even if we could I don't think it would be much help for our problem. Our problem is that the PCIe link does not work every time. It does work most of the time and so we are trying to debug what is happening on the occasions it fails. We've printed out the supposed debug register information, but it does not appear to give any indication where things failed.

[frankba]

Hi George,

here is a little iMX6 PCIe debug register interpreter
which might help you to understand what’s going on:

imx6-pcie/imx6-pcie-decoder.c at master · xobs/imx6-pcie · GitHub

Regards

Frank

[cblack]

We've ran that at boot time after a failure, and see

LTSSM current state: 0x2 (S_POLL_ACTIVE)

PIPE transmit K indication: 0

PIPE Transmit data: 0x4a4a

Receiver is receiving logical idle: no

Second symbol is also idle (16-bit PHY interface only): no

Currently receiving k237 (PAD) in place of link number: no

Currently receiving k237 (PAD) in place of lane number: no

Link control bits advertised by link partner: 0x0

Receiver detected lane reversal: no

TS2 training sequence received: no

TS1 training sequence received: no

Receiver reports skip reception: no

LTSSM reports PHY link up: no

A skip ordered set has been transmitted: no

Link number advertised/confirmed by link partner: 0

Application request to initiate training reset: no

PIPE transmit compliance request: no

PIPE transmit electrical idle request: no

PIPE receiver detect/loopback request: no

LTSSM-negotiated link reset: yes

LTSSM testing for polarity reversal: no

LTSSM performing link training: no

LTSSM in DISABLE state; link inoperable: no

Scrambling disabled for the link: no

Although we are by no means PCI experts, so any help in decoding the decoder output would be much appreciated!

thanks,

Curt