Troubleshooting/Initializing the PCIE up on imx6

[athomas]

I am having trouble getting the PCIE up on the IMX6 and interfacing with a TW6869 Video IC and I cannot seem to find sufficient detail on how to setup HW/SW and get it configured to be able to use PCIE with the imx6q.  On the hardware side I see design notes on being able to CLK1_N/P pins C7/D7 for the clock references plus the standard PCIe bus pins and see that implemented on the Sabrelite reference designs but I am having trouble understanding what needs to be setup in software and how to ensure the hardware is properly functioning. The problem seems to be compounded by the pci-imx driver and associated 3rd party PCIe devices.

After enabling PCIe in my associated imx6qdl-customboard.dtsi (loosely based of the sabrelite dtsi) and imx6qdl.dtsi and building the Freescale PCI driver into the 3.10.53 Kernel I am using, I can see the driver attempt to load in syslog/dmesg, but it fails to initialize w/ the common error message that does not provide much context, seen here https://community.freescale.com/thread/366689 (there are multiple threads seeing the same issue). It is the same error people have reported on the Sabrelite/Boundary Devices boards with certain PCIe cards or when no card is available, see the comments on PCIe at https://boundarydevices.com/ubuntu-trusty-for-i-mx6-boards-june-2015-kernel-3-10-53/. I have replicated the issue on the Sabrelite when I do not have a PCIe card installed.

After reviewing imx6 forums I tried a number of things without any luck, including:

- Disabling MSI/MSI-X Interrupts in the kernel

- Increasing the sleeps in pci-imx.c

- Tried the pci-imx.c from the 4.3.5 Kernel

The Linux Kernel 4.3.5 version of the driver appeared to provide better error handling and improve initialization. The new driver built with little hassle on 3.10.53, but upon being loaded reported an invalid or missing clock. Investigating the driver more I saw the clock names appear hardcoded to the clocks in clks-imx6q.c and per this link, http://lists.infradead.org/pipermail/linux-arm-kernel/2014-March/243809.html, there has been a naming convention change (at least in the main line Linux). I am going to pull out the patch linked with the clocking name convention change, so the clocks it wants should be defined in mach-imx/clk-imx6q.c. But I cannot seem to put together sufficient information on how I should setup and configure the HW + kernel/device tree/driver to use PCIe with the imx6. I have been able to dig through design & data sheets, kernel, drivers, and device tree examples for the imx6 with decent success, but I can’t seem to bridge the gap on getting the PCIe to work.

Is there any additional insight/direction that I can try? I will find a yocto build that has the updated pci-imx driver, if my back port does not work.

[Yuri]

Hello,

  Please look at the following discussion Re: PCIe, diagnosing and improving eye diagram

Also, please try adjust parameters of PCIe_PHY  by changing the IOMUXC_GPR8 register.

You may refer to app note http://cache.nxp.com/files/32bit/doc/app_note/AN4784.pdf

for details.

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[athomas]

Yuri,

I am still having difficulties with getting the PCIe link up. I was able to find that the PCI Reset was not properly firing/wired and have resolved that issue. I can see data on the clock/TX/RX signals that look ok. I can try to adjust the levels per your previous comment, but I am having trouble getting insight into what is happening. It has been a little difficult digging through the IMX PCI and clock drivers, code, and documentation, but I am starting to have better feeling for how all that is handled. I have limited experience with PCIe at this level, but I have started to investigate the PCIE Debug registers when the driver tries to bring the link up.

Per the print outs below - In the function imx6_pcie_link_up () of file pci-imx.c of the, I added an extra print right after Debug Register 1 is first read and turned up the debug printing and I am seeing the following prints out of the imx6_pcie_link_up function. I had also extended some of the delays in the process.

Is there any insight you can see from the debug registers during the link-up process?

[    6.947872] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=004abc43 debug_r1=08000000

[    6.976156] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08000000

[    6.981953] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=004abc43 debug_r1=08000000

[    7.010225] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08000000

[    7.016022] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08000000

[    7.044307] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08000000

[    7.050090] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08000000

[    7.078373] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08100000

[    7.084169] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08100000

[    7.112454] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08100000

[    7.118236] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08100000

[    7.146521] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08100000

[    7.152317] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08100000

[    7.180589] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08000000

[    7.186371] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08000000

[    7.214660] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08000000

[    7.220457] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08000000

[    7.248729] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08000000

[    7.254525] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b5bc43 debug_r1=08000000

[    7.282808] imx6q-pcie 1ffc000.pcie: CLU -  debug_r1=08100000

[    7.288592] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=004abc43 debug_r1=08100000

After investigating the registers by hand and finding the i.MX6 “PHY link never came up” page I ran the decoder program someone through put on GitHub.

syn-3.10.53-1.1.2$ ./imx6-pcie-decoder

LTSSM current state: 0x3 (S_POLL_COMPLIANCE)

PIPE transmit K indication: 1

PIPE Transmit data: 0x4abc

Receiver is receiving logical idle: no

Second symbol is also idle (16-bit PHY interface only): no

Currently receiving k237 (PAD) in place of link number: no

Currently receiving k237 (PAD) in place of lane number: no

Link control bits advertised by link partner: 0x0

Receiver detected lane reversal: no

TS2 training sequence received: no

TS1 training sequence received: no

Receiver reports skip reception: no

LTSSM reports PHY link up: no

A skip ordered set has been transmitted: no

Link number advertised/confirmed by link partner: 0

Application request to initiate training reset: no

PIPE transmit compliance request: yes

PIPE transmit electrical idle request: no

PIPE receiver detect/loopback request: no

LTSSM-negotiated link reset: yes

LTSSM testing for polarity reversal: no

LTSSM performing link training: no

LTSSM in DISABLE state; link inoperable: no

Scrambling disabled for the link: no

I am going to continue to research the problem and PCIe, but I would appreciate any assistance/help.

[athomas]

We had a problem with board rework and I cleaned up the debug printouts I had. I can now see register 1 change as it tries to go through the link process.

[    2.629954] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=002cf742 debug_r1=08000000

[    2.638242] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=0033b800 debug_r1=08200000

[    2.646529] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=004a4a02 debug_r1=08000000

[    2.654818] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00f7bcc2 debug_r1=08000000

[    2.663106] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00000202 debug_r1=08000000

[    2.671393] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=0012de00 debug_r1=08200000

[    2.679667] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00951900 debug_r1=08200000

[    2.687955] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00000202 debug_r1=08000000

[    2.696242] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=004a4a02 debug_r1=08000000

...

[athomas]

I have had success in using HW with my current by building in PCIe support into u-boot, but I am having trouble with Linux.

U-Boot 2014.07-08914-gbb9dde5-dirty (Feb 19 2016 - 14:27:45)

CPU:   Freescale i.MX6Q rev1.2 at 792 MHz

Reset cause: POR

Board: Custom

I2C:   ready

DRAM:  1 GiB

MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2

SF: Detected SST25VF016B with page size 256 Bytes, erase size 4 KiB, total 2 MiB

  00:01.0     - 16c3:abcd - Bridge device

   01:00.0    - 1797:6869 - Build before PCI Rev2.0

I had to take out the PCIe support in u-boot to prevent the Kernel from hanging around the time it is loading PCIe and see basically the same problem I had before:

[    2.503619] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=008e5b00 debug_r1=08200000

[    2.511906] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00a3af00 debug_r1=08200000

[    2.520181] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00995f00 debug_r1=08200000

[    2.528469] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=002e0d00 debug_r1=08200000

[    2.536756] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=003d8200 debug_r1=08200000

[    2.545045] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=006df500 debug_r1=08200000

[    2.553332] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00b47600 debug_r1=08200000

[    2.561619] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=003f6f00 debug_r1=08200000

[    2.569893] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00d72c00 debug_r1=08200000

[    2.578181] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00828c00 debug_r1=08200000

[    2.586468] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=005f9a00 debug_r1=08200000

[    2.594755] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00f89200 debug_r1=08200000

[    2.603041] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=00503600 debug_r1=08200000

[    2.611329] imx6q-pcie 1ffc000.pcie: TG2 - debug_r0=007f9400 debug_r1=08200000

[    2.619599] imx6q-pcie 1ffc000.pcie: phy link never came up

[    2.625217] imx6q-pcie 1ffc000.pcie: DEBUG_R0: 0x00674f00, DEBUG_R1: 0x08200000

I will go back to disabling the MSI/X support.

[fabio_estevam]

Could you try to use kernel 4.5-rc4 (or 4.4.2)?

[athomas · Accepted solution]

Fabio/Yuri,

Thanks for your help. I was eventually able to resolve the issue. First in U-Boot then in Linux. I think the real culprit was our hardware and the rework we did to our boards, to include re-assigning the line we used for the PCIe Reset Pin. After the hardware issues were fully resolved, I had a problem with the PCI Reset Line assignment in my device tree. I accidentally reverted to using the old reset line when I rolled out some of the changes I made to the driver and device tree for testing.

I will say that between the reference manual and at least the 3.10.53 versions of the pci-imx.c driver, it is difficult to understand and diagnosis problems with PCIe. Particularly related to the Clock Control Manager/CLK1&2 and the limitations of the PCIe bus speeds. PCIe problems and reliability issues related to the imx6 still seem to be paint point for others:

i.MX6 “PHY link never came up”

[tengri]

Hi Thomas, 

We are also having an issue of interfacing TW6865 chip with IMX6Q Nitrogen 6Q board from Boundary Devices. The PCIE ports comes with CLK1N/P signals. I am using following drivers (TW68, tw6869 and tw686x) shown in following link for interfacing.

linux-imx6/drivers/media/pci at boundary-imx_3.14.52_1.1.0_ga · boundarydevices/linux-imx6 · GitHub 

The issue is we cannot atleast get the pcie clock up and running. We're using kernel 3.14.52 ga with ubuntu trusty version, and could you please kindly elaborate (step by step) the workaround that got your issue solve ? 

[athomas]

Yuri,

Thanks for pointing me to that information, it looks to pull it all together and I had not seen it yet. I will see if I can get it going.

Thanks,

Aaron