MX6 and PCIE implementation question?

[GregT]

Has anyone got this working?  I have the NOVPEK board from NOVTECH and built their kernel with PCIE enabled.  We connected a switch (PEX 8604BA-AIC1U1D RDK from PLX) up to it but do not see anything with the "lspci" command.  Should I see something?  Where would I look?  In the /sys or /proc trees?

I did plug the switch into a Windows XP machine and it recognized it under System Devices so I don't think it is the switch.

Thanks,

Greg

[GregT]

I've got the PCIe working on the NOVPEK board.  We had to add 12V supply to the connector, put a couple of resistors to terminate a clock and change the Linux kernel to use a different clock source.  Ask and I can send the details.

We have been able to connect end point devices to the PCIe connector including a PCIe to USB card and an Altera FPGA development card.

We still have a problem with switches.  The kernel crashes trying to access the configuration space of the ports of the switch.  The switch itself is recognized and enumerated but when the first connection port of the switch configuration space is read it crashes.  It is trying to access 0x01F10000.  FYI, the switch configuration space is at 0x01F00000.  This offset scheme somewhat conflicts with the reference manual that seems to indicate that a new bus should be at 1M offset (1 << 20) rather than 64k offset (1 << 16).  (1 << 16) is the PCI standard.  Anybody looking into this?

Thanks,

Greg

[GregT · Accepted solution]

We have the PCIe switch working now with the MX6 as root, a Pericom switch, and three end points (two altera FPGAs and one USB controller).  We had help from Nils who got us past the Cfg Type 1 problem using the ATU.  The code is attached. It is hard coded for this specific bus topology. You will have to modify it fit your topology.  It also still has debug code in it that is commented out.  Here is the lspci output:

root@freescale ~$ lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0604: 12d8:2404

02:01.0 Class 0604: 12d8:2404

02:02.0 Class 0604: 12d8:2404

02:03.0 Class 0604: 12d8:2404

03:00.0 Class ff00: 1172:e001

04:00.0 Class 0c03: 1b21:1042

05:00.0 Class ff00: 1172:e001

To get the driver to handle multiple bus topologies a data abort handler is needed so when a device doesn't exist the Linux kernel will return all F's and the PCI driver system will not enumerate it. I'm going to try calling hook_fault_code for the specific fault code I am seeing.

Regards,

Greg

[SergioSolis]

Thank you for posting the answer Greg, it will surely help someone else in the near future!

[kitz36]

I would be interested in hearing what specifically you had to do to get PCIE working.  I'm working on a design that connects an i.MX6 to a single endpoint, an FPGA.  I've compiled Freescale's Linux kernel to enable PCIE.  We are also sourcing the clock externally.  When the board comes up, it reports that the PCIE link is down.  "lspci" does not return any output - not even the root complex - which sounds similar to what you experienced initially.  I am particularly interested in what resistors you had to place.

Thanks!

-Charlie

[GregT]

I believe the resistors were put in to terminate the ref clock lines for the PCIe interface on the development board. This was suggested by the maker of the develoment board, Novtech. There was also a clock change in the BSP source. I can get you details of the BSP change as well as a pic of the terminating resistors on Monday. The pic and BSP change may not help if you don't have the Novtech board.

Greg

[kitz36]

Thanks, Greg.  I get the sense that the resistor change to terminate the PCIe reference clock may apply to our design.  The PCIe clock has a requirement for differential impedance that is slightly more strict than we anticipated.  Perhaps we aren't meeting the requirement.  Also, we added an external clock to drive the PCIe, but I'm not positive that I'm configuring the i.MX6 properly to use it.  The method I used to tell the i.MX6 to use the external reference (at the CLK1_N and CLK1_P pins) is as follows:

static void __init set_pcie_external_reference(void)

{

    void __iomem *pmu_reg = IO_ADDRESS(0x020C8000);

    u32 tmpdata;

    BUG_ON(0xDEADBEEF == (u32)pmu_reg);

    tmpdata = readl(pmu_reg + 0x160); /* PMU_MISC1n */

    tmpdata &= ~(0x1F);  /* clear clock 1 select */

    tmpdata |= (0xA);    /* CLK1: select PCIe Reference */

    tmpdata &= ~(1<<10); /* CLK1: disable output */

    tmpdata |= (1<<12);  /* CLK1: enable input */

    writel(tmpdata, pmu_reg + 0x160);

}

Do you have something similar?

-Charlie

[GregT]

The resistors we put in were 49.9 Ohm.

We are using the MX6 generated clock rather than having an external clock. We did modify the PMU_MISC1 register but modified the LVDS_CLK2 configurations, not the LVDS_CLK1 configurations as you have below. I would try modifying the LVDS2 configurations in your function.

FYI, below is our code for setting up the MX6 to generate the PCIe clock.

static int clkpcie_enable(struct clk *clk)

{

unsigned int reg;

/* Activate LVDS CLK2 (the PCIe clock input) */

/* Disable the clock first */

reg = __raw_readl(ANADIG_MISC1_REG);

reg &= ~ANATOP_LVDS_CLK2_IBEN_MASK;

__raw_writel(reg, ANADIG_MISC1_REG);

reg = __raw_readl(ANADIG_MISC1_REG);

reg |= (ANATOP_LVDS_CLK1_SRC_SATA << 5); /* shift over to LVDS_CLK2 */

__raw_writel(reg, ANADIG_MISC1_REG);

reg = __raw_readl(ANADIG_MISC1_REG);

reg |= ANATOP_LVDS_CLK2_OBEN_MASK;

__raw_writel(reg, ANADIG_MISC1_REG);

/* Enable PCIE ref clock */

reg = __raw_readl(PLL8_ENET_BASE_ADDR);

reg |= ANADIG_PLL_ENET_EN_PCIE;

__raw_writel(reg, PLL8_ENET_BASE_ADDR);

clkenable(clk);

return 0;

}

Greg Topmiller

Staff Software Engineer

240-404-2620

Greg.topmiller@jdsu.com

[danielmi]

I'm also working on getting PCI-E to work for our setup. And when reading the current BSP source and those lines you posted I am wondering why the LVDS1 (CLK1_P/N) or in your case CLK2_P/N needs the SATA reference clock output when the intention is to clock a PCI-E card on another end. I was thinking of using the *SRC_PCIE macro (which is 0xa 0b1010) but that didn't work but the *SRC_SATA macro (which is 0xb 0b1011) works.

What's the reason for using *SATA here?

[kitz36]

The i.MX6DQ reference manual sort of uses the terms "PCIe clock" and "SATA clock" interchangeably with "125MHz clock" and "100MHz clock", respectively.  Reference the following sections in th IMX6DQ reference manual (revision O):

- 18.5.1.3.6  Ethernet PLL: "...Ref_PCIe = 125MHz, Ref_SATA=100MHz..."

- 18.7.15  Analog ENET PLL Control Register (CCM_ANALOG_PLL_ENETn): clock enable bits are called "ENABLE_100M" and "ENABLE_125M"

I've had this confirmed by Freescale.  Basically, the external PCIe clock can configured to be either 125MHz or 100MHz, corresponding to the 125MHz and 100MHz outputs of the ENET PLL.  Freescale's BSP defaults to routing the 100MHz clock (a.k.a. the SATA clock) to the CLK1 pins for output to the PCIe link partner.  It sounds like the NOVPEK wants to use the CLK2 pins instead of CLK1, but still wants a 100MHz reference clock.

Changing the source in PMU_MISC1n from SATA to PCIe would result in changing the reference clock output by the i.MX6 from 100MHz to 125MHz.

-Charlie

[danielmi]

Yep, I was aware of the fact that SATA is a 100Mhz clock and PCI-E is a 125Mhz clock. I was just thinking that the PCI-E standard requires a 125Mhz clock at the slot. Though, as I don't have access to that document, I'm merely guessing.

[kitz36]

I'm guessing the Freescale documentation calls the 125MHz clock "PCIe" because it's the reference clock for the PCIe PHY in the i.MX6.

[GregT]

On the NOVPEK the 100Mhz is for the downstream pcie devices not the board.  In the 2.1 PCIe spec in section 4.3.7.4 it refers to a Refclk of 100 MHz +- 300 ppm.  In 3.0 PCIe spec section 4.3.8.4 it mentions the Refclk with nominal frequency of 100 MHza nd +- 300 ppm offset.  I don't see any mention of a Refclk of 125 MHz.

[danielmi]

Ok, then it makes perfect sense. Thank you for looking that up.

[GregT]

This code was given to me by Novtech who I believe got it directly from Freescale. The clocking architecture is not well documented and we rely on Freescale to provide support. That can be difficult to get and usually no explanation (this case) is given.

Greg Topmiller

Staff Software Engineer

240-404-2620

Greg.topmiller@jdsu.com

[danielmi]

Maybe some Freescale member could drop in and provide us with an explanation here. :smileyhappy:

[kitz36]

Thanks for the help, Greg.  We're still trying to figure this out, but I wanted to check in.  I expect that part of our problem is dealing with the PCIe reference clock, which is affected by the source code above.  I don't have access to the NOVPEK schematics, but I expect it routes a clock from the i.MX6 CLK2 pins to the PCIe switch reference clock pins.  I'm guessing the PCIe switch wants a 100MHz clock.  Your source code routes the SATA reference clock out the CLK2 pins.  Since the PCIe reference clock (125MHz) and the SATA reference (100MHz) have the ENET_PLL as a clock parent, this is valid.

Our design uses a common external 100MHz reference for both the i.MX6 and the PCIe endpoint.  For us, the trick is getting the reference from the CLK1 pins to the PCIe PHY, then applying the right PLL multipliers.  Right now I think we have the reference getting to the PCIe PHY, but I'm having trouble getting the PCIe PHY PLL to lock.

So, that's my current story.  If anyone has any experience using an external reference to drive the i.MX6 PCIe PHY, I'd love to hear about it.

-Charlie

[iWave]

PCIe switch functionality is validated. For more details check the below link:

PCIe Switch Functionality on i.MX6 Board | iWave Systems

iWave Systems