Problem training PCIe on i.MX6SOLO custom board

[jeremygarff]

I'm having a problem getting the PCIe to train with a PLX 8712 PCIe switch on a custom board.  I've verified the following are working properly:

  - Verified with scope, PCIe clock is being generated at 100Mhz (not 125).  This feeds a clock buffer to downstream PCIe switch and slots.

  - PCIe GPIO reset pin is being properly asserted/released for PCIe switch.  Kernel debug added to following trace to indicate sequence.

  - The PCIe switch is able to link with downstream slots properly, so it's coming out of reset and strapped properly.  We can communicate with the switch over i2c.

  - The board is routing the PCIe lane on a single layer (top) with 85 ohm characteristic differential impedance.  We're using DC blocking caps on both RX and TX lanes to the PCIe switch.  There are no connectors or vias in the path, and the routes are length balanced, isolated and very clean.

I'm running stock u-boot (19210ae9838a867ce0243c41eafe928317c15b10) patched to support our board.  The following is a recent Linux 3.0.35 kernel log from Yocto/Poky which has been patched to support our board as well as support PCIe bridges.  I've added debug in the link training loop in pcie.c to show the PCIE_PL_DEBUG0 and PCIE_PL_DEBUG1 registers during link training.  In the debug, the first column is DEBUG0, the second DEBUG1.

From what I can tell (very little), it appears that the LTSSM is working and attempting to train.  I've used this same kernel on a SABRESD Quad, which trains and links just fine with a downstream switch (PLX 8603) connected to the mini-PCIe.  I have the following questions:

1)  Is the solo different than the quad on the SABRESD with regard to PCIe?  Is something not being properly initialized for the solo?

2)  From the debug, is it possible to tell why the link isn't training?

3)  Is there any better documentation to help debug link training issues?

[    0.000000] Linux version 3.0.35-ionewt+g760b88a (jer@jers) (gcc version 4.6.2 20110630 (prerelease) (Freescale M4
[    0.000000] CPU identified as i.MX6DL/SOLO, silicon rev 1.1
[    0.000000] Kernel command line: console=ttymxc3,115200 loglevel=7 root=/dev/nfs ip=dhcp
[    0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Memory: 256MB = 256MB total
[    0.000000] Memory: 254788k/254788k available, 7356k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     DMA     : 0xf4600000 - 0xffe00000   ( 184 MB)
[    0.000000]     vmalloc : 0xd0800000 - 0xf2000000   ( 536 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xd0000000   ( 256 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .init : 0xc0008000 - 0xc0029000   ( 132 kB)
[    0.000000]       .text : 0xc0029000 - 0xc0484970   (4463 kB)
[    0.000000]       .data : 0xc0486000 - 0xc04bae80   ( 212 kB)
[    0.000000]        .bss : 0xc04baea4 - 0xc04f12d4   ( 218 kB)
[    0.000000] SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] NR_IRQS:496
[    0.000000] MXC GPIO hardware
[    0.000000] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 1431655ms
[    0.000000] Set periph_clk's parent to pll2_pfd_400M!
[    0.000000] MXC_Early serial console at MMIO 0x21f0000 (options '115200')
[    0.000000] bootconsole [ttymxc3] enabled
[    0.171833] Calibrating delay loop... 1581.05 BogoMIPS (lpj=7905280)
[    0.264678] pid_max: default: 32768 minimum: 301
[    0.269506] Mount-cache hash table entries: 512
[    0.274404] CPU: Testing write buffer coherency: ok
[    0.279442] hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counters available
[    0.287999] devtmpfs: initialized
[    0.298031] print_constraints: dummy:
[    0.301967] NET: Registered protocol family 16
[    0.310740] print_constraints: vddpu: 725 <--> 1300 mV at 1150 mV fast normal
[    0.318391] print_constraints: vddcore: 725 <--> 1300 mV at 1150 mV fast normal
[    0.326142] print_constraints: vddsoc: 725 <--> 1300 mV at 1200 mV fast normal
[    0.333943] print_constraints: vdd2p5: 2000 <--> 2775 mV at 2400 mV fast normal
[    0.341699] print_constraints: vdd1p1: 800 <--> 1400 mV at 1100 mV fast normal
[    0.349356] print_constraints: vdd3p0: 2800 <--> 3150 mV at 3000 mV fast normal
[    0.361421] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[    0.368234] hw-breakpoint: 1 breakpoint(s) reserved for watchpoint single-step.
[    0.375726] ------------[ cut here ]------------
[    0.380380] WARNING: at arch/arm/kernel/hw_breakpoint.c:242 enable_monitor_mode+0x98/0xb4()
[    0.388973] ---[ end trace da227214a82491b7 ]---
[    0.393717] hw-breakpoint: halting debug mode enabled. Assuming maximum watchpoint size of 4 bytes.
[    0.402808] L310 cache controller enabled
[    0.406880] l2x0: 16 ways, CACHE_ID 0x410000c8, AUX_CTRL 0x02050000, Cache size: 524288 B
[    0.423444] bio: create slab <bio-0> at 0
[    0.428938] SCSI subsystem initialized
[    0.432962] spi_imx imx6q-ecspi.1: probed
[    0.437680] usbcore: registered new interface driver usbfs
[    0.443323] usbcore: registered new interface driver hub
[    0.448864] usbcore: registered new device driver usb
[    0.456504] Switching to clocksource mxc_timer1
[    0.463100] NET: Registered protocol family 2
[    0.467618] IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.475225] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    0.482459] TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
[    0.488992] TCP: Hash tables configured (established 8192 bind 8192)
[    0.495382] TCP reno registered
[    0.498539] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.504411] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.510985] NET: Registered protocol family 1
[    0.515614] RPC: Registered named UNIX socket transport module.
[    0.521610] RPC: Registered udp transport module.
[    0.526329] RPC: Registered tcp transport module.
[    0.531046] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.537992] Static Power Management for Freescale i.MX6
[    0.543276] wait mode is enabled for i.MX6
[    0.547496] cpaddr = d0880000 suspend_iram_base=d0818000
[    0.552938] PM driver module loaded
[    0.556490] training stopped
[    0.559880] plx reset low
[    0.671150] plx reset high
[    0.681908] training started
[    0.684800] 000eb300 08200000
[    0.690795] 00cd5100 08200000
[    0.696806] 004a4a02 08000000
[    0.702802] 0c454504 1800f700
[    0.708796] 0c2cf744 1800f700
[    0.714801] 0c454504 1800f700
[    0.720793] 0c2cf744 1800f700
[    0.726799] 08454504 18004a00
[    0.732794] 08454504 1800b500
[    0.738786] 08f7bcc4 18004a00
[    0.744791] 08f7bcc4 1800b500
[    0.750782] 504b2c01 0860b500
[    0.756786] 0c454504 1800f700
[    0.762780] 0c454504 1800f700
[    0.768772] 0c454504 1800f700
[    0.774776] 0c000604 1800f700
[    0.780769] 0c454504 1800f700
[    0.786773] 0c454504 1800f700
[    0.792767] 0cf7bcc4 1800f700
[    0.798759] 0c2cf744 1800f700
[    0.804764] 0cf1a301 0860f700
[    0.810757] 0c2cf744 1800f700
[    0.816761] 0c454504 1800f700
[    0.822756] 0cf7bcc4 1800f700
[    0.828747] 0c454504 1800f700
[    0.834752] 0c454504 1800f700
[    0.840745] 0cf7bcc4 1800f700
[    0.846748] 0c000604 1800f700
[    0.852742] 0c454504 1800f700
[    0.858734] 0c956901 0860f700
[    0.864739] 0c000604 1800f700
[    0.870731] 0c454504 1800f700
[    0.876736] 0c454504 1800f700
[    0.882729] 0c454504 1800f700
[    0.888720] 0c454504 1800f700
[    0.894726] 0c454504 1800f700
[    0.900717] 0c454504 1800f700
[    0.906721] 0c000604 1800f700
[    0.912715] 0c2cf744 1800f700
[    0.918707] 0c454504 1800f700
[    0.924713] 0cf7bcc4 1800f700
[    0.930705] 0cf7bcc4 1800f700
[    0.936709] 0c454504 1800f700
[    0.942702] 0c000604 1800f700
[    0.948694] 0c454504 1800f700
[    0.954698] 0c000604 1800f700
[    0.960690] 0c2cf744 1800f700
[    0.966694] 0c454504 1800f700
[    0.972688] 0c1c1cc4 1800f700
[    0.978681] 0cf7bcc4 1800f700
[    0.984685] 0c454504 1800f700
[    0.990677] 0c454504 1800f700
[    0.996681] 0c454504 1800f700
[    1.002675] 0c2cf744 1800f700
[    1.008669] 0c454504 1800f700
[    1.014673] 0cf7bcc4 1800f700
[    1.020665] 0c454504 1800f700
[    1.026671] 0c000604 1800f700
[    1.032665] 0c454504 1800f704
[    1.038657] 0c2cf744 1800f700
[    1.044661] 0cf7bcc4 1800f700
[    1.050653] 0c454504 1800f700
[    1.056659] 0c2cf744 1800f700
[    1.062653] 0c454504 1800f700
[    1.068645] 0c2cf744 1800f700
[    1.074650] 0c454504 1800f700
[    1.080642] 0c454504 1800f700
[    1.086646] 0c454504 1800f700
[    1.092641] 0c454504 1800f700
[    1.098633] 0c454504 1800f700
[    1.104637] 0c000604 1800f700
[    1.110629] 0c000604 1800f700
[    1.116633] 0c454504 1800f700
[    1.122627] 0c454504 1800f700
[    1.128619] 0c454504 1800f704
[    1.134625] 0c454504 1800f700
[    1.140618] 0c454504 1800f700
[    1.146622] 0c454504 1800f700
[    1.152616] 0c2cf744 1800f700
[    1.158608] 0c454504 1800f700
[    1.164612] 0c454504 1800f700
[    1.170604] 0c454504 1800f700
[    1.176609] 0c7b7000 0820f700
[    1.182604] 0c454504 1800f700
[    1.188596] 0cf7bcc4 1800f700
[    1.194601] 0c000604 1800f704
[    1.200594] 0c000604 1800f700
[    1.206601] 0cf7bcc4 1800f700
[    1.212595] 0c454504 1800f700
[    1.218586] 0c454504 1800f700
[    1.224590] 0c000604 1800f700
[    1.230582] 0c2a3400 0820f700
[    1.236586] 0c454504 1800f700
[    1.242580] 0c454504 1800f700
[    1.248572] 0cf7bcc4 1800f700
[    1.254577] 0cf7bcc4 1800f700
[    1.260569] 0c2cf744 1800f700
[    1.266573] 0c454504 1800f700
[    1.272567] 0c454504 1800f700
[    1.278559] 0c454504 1800f700
[    1.284563] 0c39ba00 0820f700
[    1.290555] 0cf7bcc4 1800f700
[    1.296560] 0cf7bcc4 1800f700
[    1.302555] 0c454504 1800f700
[    1.308547] 0cf7bcc4 1800f700
[    1.314551] 0c454504 1800f700
[    1.320542] 0cf7bcc4 1800f700
[    1.326547] 0c454504 1800f700
[    1.332541] 0c454504 1800f700
[    1.338532] 0c0e3701 0860f700
[    1.344537] 0c454504 1800f700
[    1.350528] 0c2cf744 1800f700
[    1.356533] 0c454504 1800f700
[    1.362527] 0c454504 1800f700
[    1.368518] 0c2cf744 1800f700
[    1.374523] 0c454504 1800f700
[    1.380516] 0c000604 1800f700
[    1.386520] 0c454504 1800f700
[    1.392515] 0c546601 0860f700
[    1.398508] 0c454504 1800f700
[    1.404512] 0c454504 1800f704
[    1.410504] 0c2cf744 1800f700
[    1.416507] 0c2cf744 1800f700
[    1.422501] 0c454504 1800f700
[    1.428494] 0c454504 1800f700
[    1.434498] 0c2cf744 1800f700
[    1.440490] 0c454504 1800f700
[    1.446494] 0ceac501 0860f700
[    1.452487] 0c454504 1800f700
[    1.458479] 0c454504 1800f708
[    1.464483] 0c2cf744 1800f700
[    1.470475] 0c454504 1800f700
[    1.476479] 0c454504 1800f700
[    1.482474] 0c454504 1800f700
[    1.488466] 0c000604 1800f700
[    1.494472] 0c2cf744 1800f700
[    1.500463] 0c479001 0860f700
[    1.506467] 0c454504 1800f700
[    1.512462] 0c454504 1800f700
[    1.518454] 0c454504 1800f700
[    1.524459] 0c000604 1800f700
[    1.530451] 0c454504 1800f700
[    1.536455] 0c000604 1800f700
[    1.542449] 0c454504 1800f700
[    1.548440] 0c000604 1800f700
[    1.554445] 0c454504 1800f700
[    1.560436] 0c2cf744 1800f704
[    1.566441] 0cf7bcc4 1800f700
[    1.572435] 0c454504 1800f700
[    1.578427] 0c000604 1800f700
[    1.584432] 0c000604 1800f700
[    1.590424] 0c454504 1800f700
[    1.596428] 0c454504 1800f700
[    1.602422] 0c454504 1800f700
[    1.608413] 0c454504 1800f700
[    1.614418] 0c000604 1800f700
[    1.620410] 0c454504 1800f700
[    1.626414] 0c454504 1800f700
[    1.632408] 0c454504 1800f700
[    1.638399] 0c2cf744 1800f700
[    1.644403] 0c000604 1800f700
[    1.650394] 0c454504 1800f700
[    1.656399] 0c000604 1800f700
[    1.662394] 0cf7bcc4 1800f704
[    1.668387] 0c454504 1800f700
[    1.674391] 0cf7bcc4 1800f700
[    1.680383] 0c2cf744 1800f700
[    1.686386] 0c000604 1800f700
[    1.692380] 0c454504 1800f700
[    1.698372] 0c454504 1800f700
[    1.704376] 0c454504 1800f700
[    1.710368] 0c454504 1800f700
[    1.716373] 0c454504 1800f700
[    1.722368] 0c2cf744 1800f700
[    1.728359] 0c454504 1800f704
[    1.734363] 0c454504 1800f700
[    1.740355] 0c454504 1800f700
[    1.746361] 0c2cf744 1800f700
[    1.752355] 0c000604 1800f700
[    1.758347] 0c454504 1800f700
[    1.764352] 0c454504 1800f700
[    1.770344] 0c454504 1800f700
[    1.776348] 0c000604 1800f700
[    1.782342] 0c000604 1800f700
[    1.788334] 0c454504 1800f700
[    1.794340] 0c454504 1800f700
[    1.800332] 0c454504 1800f700
[    1.806336] 0c2cf744 1800f700
[    1.812331] 0c000604 1800f700
[    1.818323] 0cdb0d00 0820f700
[    1.824328] 0c454504 1800f700
[    1.830319] 0c454504 1800f700
[    1.836324] 0c454504 1800f700
[    1.842318] 0c454504 1800f704
[    1.848310] 0c454504 1800f700
[    1.854315] 0cf7bcc4 1800f700
[    1.860307] 0cf7bcc4 1800f700
[    1.866312] 0c454504 1800f700
[    1.872306] 0c5bd800 0820f700
[    1.878298] 0c2cf744 1800f704
[    1.884303] IMX PCIe port: link down!
[    1.905324] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.912682] msgmni has been set to 497
[    1.918054] alg: No test for stdrng (krng)
[    1.922360] io scheduler noop registered
[    1.926298] io scheduler deadline registered
[    1.930673] io scheduler cfq registered (default)
[    1.936230] imx-sdma imx-sdma: loaded firmware 1.1
[    1.945450] imx-sdma imx-sdma: initialized
[    2.085913] Serial: IMX driver
[    2.089086] imx-uart.3: ttymxc3 at MMIO 0x21f0000 (irq = 61) is a IMX
[    2.095634] console [ttymxc3] enabled, bootconsole disabled
[    2.095634] console [ttymxc3] enabled, bootconsole disabled
[    2.110615] loop: module loaded
[    2.114490] FEC Ethernet Driver
[    2.119331] fec_enet_mii_bus: probed
[    2.124271] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.131070] Initializing USB Mass Storage driver...
[    2.136148] usbcore: registered new interface driver usb-storage
[    2.142235] USB Mass Storage support registered.
[    2.147067] ARC USBOTG Device Controller driver (1 August 2005)
[    2.153737] rtc-test rtc-test.0: rtc core: registered test as rtc0
[    2.160339] rtc-test rtc-test.1: rtc core: registered test as rtc1
[    2.166597] i2c /dev entries driver
[    2.171291] sdhci: Secure Digital Host Controller Interface driver
[    2.177477] sdhci: Copyright(c) Pierre Ossman
[    2.182733] usbcore: registered new interface driver usbhid
[    2.188310] usbhid: USB HID core driver
[    2.192688] TCP cubic registered
[    2.195922] NET: Registered protocol family 17
[    2.200411] Registering the dns_resolver key type
[    2.205211] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    2.212911] ThumbEE CPU extension supported.
[    2.217309] Bus freq driver module loaded
[    2.221370] Bus freq driver Enabled
[    2.224986] DVFS driver module loaded
[    2.229405] regulator_init_complete: vddpu: incomplete constraints, leaving on
[    2.236745] rtc-test rtc-test.0: setting system clock to 1970-01-01 00:00:02 UTC (2)
[    2.246572] eth0: Freescale FEC PHY driver [Generic PHY] (mii_bus:phy_addr=1:07, irq=-1)
[    4.241321] PHY: 1:07 - Link is Up - 1000/Full
[    4.271144] Sending DHCP requests ., OK
[    4.324714] VFS: Mounted root (nfs filesystem) on device 0:12.
[    4.330854] devtmpfs: mounted
[    4.333910] Freeing init memory: 132K

[thomasgraf]

Hi,

Did you ever get it working? I've got a similar problem. Link does not go up and DEBUG regs are similar to yours.

Do you have a solution? (Same Kernel is working here on SABRESD-Board with quad code and not working on custom board using the imx6solo)

Thomas

[YixingKong]

Jeremy

We have not got your response yet and will close the discussion in 3 days. If you still need help, please feel free to reply with an update to this discussion.

Thanks,

Yixing

[YixingKong]

Jeremy

Below is the comment from one of engineer.

-----------------------------------------------

From what I can tell (very little), it appears that the LTSSM is working and attempting to train. I've used this same kernel on a SABRESD Quad, which trains and links just fine with a downstream switch (PLX 8603) connected to the mini-PCIe. I have the following questions:

1) Is the solo different than the quad on the SABRESD with regard to PCIe? Is something not being properly initialized for the solo?

[Richard] As I know that there is not difference between imx6 solo pcie and imx6q pcie.

2) From the debug, is it possible to tell why the link isn't training?

[Richard] Here are some suggestions, that maybe helpful.

The details, please refer to https://community.freescale.com/message/363300#363300

3) Is there any better documentation to help debug link training issues?

[Richard] No.

[YixingKong]

Jeremy

Had your issue got resolved? If yes, we are going to close the discussion in 3 days. If you still need help, please feel free to reply with an update to this discussion.

Thanks,
Yixing

[jamesbone]

1)  Is the solo different than the quad on the SABRESD with regard to PCIe?  Is something not being properly initialized for the solo?

ANSWER. The only main difference between the i.MX6Solo and i.MX6Quad regarding PCIe are the errata, since for the i.MX6Solo we have the following:

ERR003747 PCIe: 9000436491—Reading the Segmented Buffer Depth Port

Logic registers returns all zeros

But I do not think it is the case since, you are using a FSL BSP.

2)  From the debug, is it possible to tell why the link isn't training? Do not know let me ask, and /or research about it.

3)  Is there any better documentation to help debug link training issues? . Unfortunately I do not think so.