P1013 - unable to enable PCI Express

[chrisdeming]

I am using the P1013 and attempting to enable the three PCI Express controllers.  The DEVDISR register shows that PCIe1, PCIe2 and PCIe3 are all disabled following power-on / reset.  When I attempt to clear any of these bits in the DEVDISR, there is no effect on the register.  According to the Reference Manual, these bits' reset values depend on POR configuration signals at reset. 

  By reading PORDEVSR, I see that IO_SEL_SRDS is configured with 0b11000, which should be SerDes1: PCIe1 (x2), PCIe2 (x2),    SerDes2: PCIe3 (x2).

  By reading PORBMSR, I see that Host-Agent configuration (HA field) is set to 0b100, which should be Agent on PCIe1, PCIe2 and Host on PCIe3.

We are booting using the I2C for a Boot Sequencer, which attempts to set the DEVDISR register so that PCIe1, PCIe2 and PCIe3 are not disabled.

However I cannot get the corresponding bits in DEVDISR to clear.  Also, the CCSR Register ranges for PCIe controllers and the SerDes controllers all read back as all zeroes.

Does anyone know what I might be doing wrong?

[chrisdeming]

We are using the ICS9FG108 frequency generator for the 100MHz SerDes reference clocks.  It's the same one that is used on the Freescale P1022DS Development System.  We have measured the clocks and have'nt found anything wrong with them.

I will try only enabling a single port.

[genuap]

I assume you’ve been dumping all of this from u-boot so far? Do you have a debugger? What if you just connect with the debugger (i.e. issue a reset and then read these same registers)? Do you get the same results?

[chrisdeming]

Just an update... There's a problem with a power supply. Until that's fixed & tested we are assuming that is the cause of our SerDes / PCIe problem.  Thanks for your help.

-Chris

[chrisdeming]

No - we aren't using u-boot.  Yes - we are using a Wind River ICE2 JTAG emulator.  The registers I've described are what the ICE2 shows immediately after reset.  We are looking at the registers before any software has run. (Also, enabling only a single port did not have any apparent effect.)

[genuap]

Can you dump that whole section(PORPLLSR, PORBMSR, PORDEVSR, PORDEVSR2?

Also, check your serdes clock and make sure that the cfg_srds1_refclk and cfg_srds2_refclk match what your serdes clock is set to.

Last recommendation - does this only happen on cold boot? (i.e. does it actually come up properly on a warm reset?) If so, it's likely an issue with your clocks not being ready and stable prior to HRESET going away.

[chrisdeming]

Thanks for your reply!

I am includinng a screen-shot of the Global Utility registers, including the ones you mentioned.  cfg_srds1_refclk and cfg_srds2_refclk both appear to be set for 100MHz, which is the SerDes reference clock.  Also, this happens both on cold boot and warm reset - we've never seen it working yet.  The electrical engineers have looked at the reference clocks and voltages, and they say they look ok.   -- I just realized that screen-shot may not be readable here - these are the values:  PORPLLSR - 0x46440E08

            PORBMSR - 0xCF270000

            PORDEVSR- 0x1B600840

            PORDBGMSR - 0x05000000

            PORDEVSR2 - 0xCBFFF5F0

            GPPORCR - 0x00F00000

            PMUXCR - 0x00400000

            DEVDISR - 0x260C3011

[chrisdeming]

We connected some switches to the cfg_serdes_ports inputs so that we are able to try different configurations easily.

Although we are not using Ethernet, as an experiment I tried to see if we are able to enable Ethernet.

I set cfg_serdes_ports to 0b11000 (SerDes1 - PCIe1 (x2), PCIe2 (x2);   SerDes2 - PCIe3 (x2) ), at power-up, PORDEVSR has value - 0x1960_0040 - both SGMII1 and SGMII2 are disabled. DEVDISR has value - 0x260C_3011, which means TSEC1 and TSEC2 – the Ethernet controllers are enabled.

If I set cfg_serdes_ports to 00011 (SerDes1 – powered down;   SerDes2: SGMII1, SGMII2), after power up, PORDEVSR has a value of 0x010C_0040, which means SGMII1 and SGMII2 are both enabled.  DEVDISR comes up with a value of 0x260C_30D1, which means TSEC1 and TSEC2 – the Ethernet controllers are both disabled.

If I set cfg_serdes_ports to 00100 (SerDes1 – powered down;  SerDes2 – powered down), after power up, PORDEVSR has a value of 0x1910_0040, which means SGMII1 and SGMII2 are both disabled, which is expected.  DEVDISR comes up with a value of 0x260C_3011, which means TSEC1 and TSEC2 – the Ethernet controllers are both enabled.

So it seems something about SerDes may be preventing Ethernet (and PCIe?) from being enabled...

[genuap]

It looks like everything is set up correctly. My guess is the SerDes clock.

The REF_CLK’s Vmax is specified as 800mV in the P1013EC. See AN4311 about interfacing a SerDes Ref Clock. What are you using to drive the clock into the P1013?

As another experiment, you could try another IO option as well, possibly only enabling a single PCI port – but I believe it may be a clock issue still.

… Paul

[genuap]

I would take a look at the SerDes clocks. This is typical of clocks not being there, or being the wrong frequency. I'll look through the register dumps you gave later on today, but my first thought is clocks.