i.MX6 - How to enable PCIe 2.0 ?

[brianptl]

Hi All,

I have a custom board design based on the i.MX6Q Sabresd board (MCIMX6Q-SDB) running Android 8.0.

The board has a full-size PCIe 2.0 x1 connection. I have confirmed that this is working by plugging in a PCIe 2.0 to USB 3.0 card (SilverStone SST-EC04-P, link). In Android, via the command line, I can mount an external USB SSD (MX500 120Gb) and see the files, write files etc.

I am seeing an issue whereby the PCIe connection is only coming up as Gen 1.

board_01:/ # dmesg | grep -i pci
[    0.723168] PCI: CLS 0 bytes, default 64
[    0.879982] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
[    0.879994] OF: PCI:   No bus range found for /soc/pcie@0x01000000, using [bus 00-ff]
[    0.880025] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
[    0.880041] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
[    0.935078] imx6q-pcie 1ffc000.pcie: Link up, Gen1
[    0.935287] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
[    0.935301] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.935311] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.935320] pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]
[    0.935358] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[    0.935382] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[    0.935400] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    0.935471] pci 0000:00:00.0: supports D1
[    0.935480] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[    0.935741] PCI: bus0: Fast back to back transfers disabled
[    0.935954] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
[    0.936062] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    0.936546] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    0.960535] PCI: bus1: Fast back to back transfers disabled
[    0.960722] pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]
[    0.960738] pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]
[    0.960749] pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]
[    0.960765] pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x01101fff 64bit]
[    0.960829] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.960842] pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]
[    0.961106] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    0.961117] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    0.961129] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    0.961269] aer 0000:00:00.0:pcie002: service driver aer loaded
[    0.961444] pci 0000:01:00.0: enabling device (0140 -> 0142)
[    2.136127] ehci-pci: EHCI PCI platform driver

board_01:/ # lspci -k                                                         
00:00.0 Class 0604: 16c3:abcd pcieport
01:00.0 Class 0c03: 1912:0014 xhci_hcd

Looking at the driver code (pci-imx6.c), it appears that the process for entering PCIe 2.0 mode is as follows:

static int imx6_pcie_establish_link(struct imx6_pcie *imx6_pcie)
{
    struct pcie_port *pp = &imx6_pcie->pp;
    struct device *dev = pp->dev;
    u32 tmp;
    int ret;

    /*
     * Force Gen1 operation when starting the link.  In case the link is
     * started in Gen2 mode, there is a possibility the devices on the
     * bus will not be detected at all.  This happens with PCIe switches.
     */
    if (!IS_ENABLED(CONFIG_PCI_IMX6_COMPLIANCE_TEST)) {
        tmp = dw_pcie_readl_rc(pp, PCIE_RC_LCR);
        tmp &= ~PCIE_RC_LCR_MAX_LINK_SPEEDS_MASK;
        tmp |= PCIE_RC_LCR_MAX_LINK_SPEEDS_GEN1;
        dw_pcie_writel_rc(pp, PCIE_RC_LCR, tmp);
    }

    /* Start LTSSM. */
    if (imx6_pcie->variant == IMX7D)
        regmap_update_bits(imx6_pcie->reg_src, 0x2c, BIT(6), BIT(6));
    else
        regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
                IMX6Q_GPR12_PCIE_CTL_2, 1 << 10);

    ret = imx6_pcie_wait_for_link(imx6_pcie);
    if (ret) {
        dev_info(dev, "Link never came up\n");
        goto err_reset_phy;
    }

    if (imx6_pcie->link_gen == 2) {

        /* Allow Gen2 mode after the link is up. */
        tmp = dw_pcie_readl_rc(pp, PCIE_RC_LCR);
        tmp &= ~PCIE_RC_LCR_MAX_LINK_SPEEDS_MASK;
        tmp |= PCIE_RC_LCR_MAX_LINK_SPEEDS_GEN2;
        dw_pcie_writel_rc(pp, PCIE_RC_LCR, tmp);
    } else {
        dev_info(dev, "Link: Gen2 disabled\n");
        goto out;
    }

    /*
     * Start Directed Speed Change so the best possible speed both link
     * partners support can be negotiated.
     */
    tmp = dw_pcie_readl_rc(pp, PCIE_LINK_WIDTH_SPEED_CONTROL);
    tmp |= PORT_LOGIC_SPEED_CHANGE;
    dw_pcie_writel_rc(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, tmp);

    ret = imx6_pcie_wait_for_speed_change(imx6_pcie);
    if (ret) {
        dev_info(dev, "Roll back to GEN1 link!\n");
    }

    /* Make sure link training is finished as well! */
    ret = imx6_pcie_wait_for_link(imx6_pcie);
    if (ret) {
        dev_err(dev, "Failed to bring link up!\n");
        goto err_reset_phy;
    }

out:
    tmp = dw_pcie_readl_rc(pp, PCIE_RC_LCSR);
    dev_info(dev, "Link up, Gen%i\n", (tmp >> 16) & 0xf);
    return 0;

err_reset_phy:
    dev_dbg(dev, "PHY DEBUG_R0=0x%08x DEBUG_R1=0x%08x\n",
        dw_pcie_readl_rc(pp, PCIE_PHY_DEBUG_R0),
        dw_pcie_readl_rc(pp, PCIE_PHY_DEBUG_R1));
    imx6_pcie_reset_phy(imx6_pcie);

    if (!IS_ENABLED(CONFIG_PCI_IMX6_COMPLIANCE_TEST)) {
        clk_disable_unprepare(imx6_pcie->pcie);
        if (!imx6_pcie->ext_osc)
            clk_disable_unprepare(imx6_pcie->pcie_bus);
        clk_disable_unprepare(imx6_pcie->pcie_phy);
        if (imx6_pcie->variant == IMX6SX)
            clk_disable_unprepare(imx6_pcie->pcie_inbound_axi);
        release_bus_freq(BUS_FREQ_HIGH);
        if (imx6_pcie->pcie_phy_regulator != NULL)
            regulator_disable(imx6_pcie->pcie_phy_regulator);
        if (imx6_pcie->pcie_bus_regulator != NULL)
            regulator_disable(imx6_pcie->pcie_bus_regulator);
    }

    return ret;
}

I think this equates to:

1. Enter PCIe 1.0 mode

2. Wait for link to come up

3. Attempt to Enter PCI 2.0 mode

4. Report actual mode

For debug I have put print statements in the code, and I can see that it does enter the "if (imx6_pcie->link_gen == 2)" statement.

So I am wondering why I do not get Gen2 as the result in dmesg.

I see elsewhere that there is an issue with Jitter, whereby it is recommended that an external clock source is used for PCIe 2.0. This was not noticed when we were designing the board.

My questions are as follows:

1) It looks like the negotiation for PCIe 2.0 speeds is ultimately done by the hardware, and that it will automatically fall back to PCIe 1.0 if this does not succeed. Is this correct?

2) Would the jitter problem be the reason we cannot negotiate PCIe 2.0? i.e. the system is testing whether it is PCIe 2.0 capable and determining we are not as a result of the jittery clock?

Thanks,

Brian.

[igorpadykov]

Hi Brian

1. right

2. right, reason may be weak or noisy signals, one can try to improve signals

shape using AN4784 PCIe Certification Guide

https://www.nxp.com/docs/en/application-note/AN4784.pdf 

Also one can try with several pcie cards.

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[brianptl]

Hi Igor,

Thank you for your reply. We get the same results with a PCIe 2.0 SATA card as well as the PCIe 2.0 USB 3.0 card.

I see that AN4784.pdf references a compliance load board (CLB x1/x16 revision 2.0) for testing. I cannot find any further information information about this.

I do see the note:

See document #CLB_USAGE_DOC_Rev_1_4_DSA72000b, PCI Express 2.0 CEM System Signal Quality and Reference Clock Jitter Test Methodology using Tektronix MSO/DPO/DSA72004 (20GHz), MSO/DPO/DSA71604 (16GHz) and MSO/DPO/DSA71254 (12.5GHz) Real Time Oscilloscopes, rev. 1.4 (PCI-SIG, 2011; register at pcisig.com to download this document).

On the pcisig.com website I see the following:

Membership in PCI-SIG is open to everyone. Membership gives your company access to a variety of services that will help your company get to market quickly with PCI compliant devices. For a $4,000 USD annual fee you are entitled to:

...

To become a member of PCI-SIG, please submit a completed New Member Form along with the $4,000 annual membership fee to PCI-SIG Administration.

Am I correct in saying that we would need to pay the fee to perform the procedure detailed in AN4784.pdf ?

Thanks,

Brian.

[igorpadykov]

Hi Brian

customer can get help with certification own boards using

NXP Professional Services|NXP 

As for results with a PCIe 2.0 SATA card and PCIe 2.0 USB 3.0 card

one can try with i.MX6 Sabre reference board. Issues with custom board may be

related to layout and signal losses in board material.

Best regards
igor