i.MX6Q - Software changes required for PCIe external clock?

cancel
Showing results for 
Search instead for 
Did you mean: 

i.MX6Q - Software changes required for PCIe external clock?

Jump to solution
1,586 Views
brianptl
Contributor III

Hi All,

 

I have a custom board design based on the i.MX6Q Sabresd board (MCIMX6Q-SDB) running Android 8.0.

We have changed our board to use an external PCIe 2.0 compliant clock as per the advice in this thread: https://community.nxp.com/message/1065467

We have received the board. I disabled the PCIe module in the device tree to avoid contention, and I can see that the external clock chip is outputting the required clock. Now I am looking at the software changes required for the PCIe module to support the external clock.

I have searched the forum and I can see a whole host of different threads with different advice, some of it targetting older kernel versions etc.

My question is: Is there example code, an app note or a go-to forum post with the most up to date procedure for disabling the internal PCIe clock and enabling the use of the external reference clock?

Thanks,

Brian.

Labels (2)
0 Kudos
1 Solution
541 Views
brianptl
Contributor III

Hi Igor + other interested parties,

We have received support through our supplier and internally within NXP to understand why we were not seeing Gen2 reported from the driver.

The summary ( as it looks just now ) is that there is a bug in the pci-imx6.c reporting.

If you read the register PCIE_RC_LCSR [IMX6QDRM.pdf, Rev 5, 06/2018 - page 4254] when you are fully booted, it will have the correct speed, even though the read during the driver initialisation has reported Gen1.

e.g. We inserted two cards, a Gen1 card and a Gen2 card and got the following results for PCIE_RC_LCSR when booted:

Gen1: 0x30110040
Gen2: 0x30120040
We used the utility Devmem2 to read this value. For our Android build we had to enable the following in the kernel:
CONFIG_DEVMEM=y
CONFIG_DEVKMEM=y
We then compiled the source for Devmem2.
The result looks like the following:
# devmem2 0x01ffc080
/dev/mem opened.
Memory mapped at address 0xb5fb8000.
Value at address 0x1FFC080 (0xb5fb8080): 0x30120040
You can also run lspci -vv and it will tell you the PCIe speed, with 5GT/s = Gen2, 2.5GT/s = Gen1.
01:00.0 Class 0106: Device 1b4b:9215 (rev 11) (prog-if 01)
        Subsystem: Device 1b4b:9215
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 306
        Region 0: I/O ports at 1020 [size=8]
        Region 1: I/O ports at 1030 [size=4]
        Region 2: I/O ports at 1028 [size=8]
        Region 3: I/O ports at 1034 [size=4]
        Region 4: I/O ports at 1000 [size=32]
        Region 5: Memory at 01100000 (32-bit, non-prefetchable) [size=2K]
        Expansion ROM at 01200000 [size=256K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: 28534000  Data: 0001
        Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [e0] SATA HBA v0.0 BAR4 Offset=00000004
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Kernel driver in use: ahci
Note the lspci built in to Android does not support the -vv switches. I had to copy the Yocto built lspci over, along with it's dependencies.
Find and copy the following files to the Android system (need to remount the / to be read/write)
Create a /lib folder in Android
├── lib
│   ├── ld-linux-armhf.so.3
│   ├── libcap.so.2
│   ├── libc.so.6
│   ├── libgcc_s.so.1
│   ├── libkmod.so.2
│   ├── libpci.so.3
│   ├── libpthread.so.0
│   ├── libresolv.so.2
│   ├── librt.so.1
│   ├── libudev.so.1
│   └── libz.so.1
└── lspci

If you need to make the Android filesystem writeable:

mount -o rw,remount /

Thanks,

Brian.

View solution in original post

13 Replies
541 Views
igorpadykov
NXP TechSupport
NXP TechSupport

Hi Brian

for external clock one can try dts as in linux/arch/arm/boot/dts/imx6qp-sabresd-ldo-pcie-cert.dts:

&pcie {
    ext_osc = <1>..

imx6qp-sabresd-ldo-pcie-cert.dts\dts\boot\arm\arch - linux-imx - i.MX Linux kernel 

Schematic i.MX6QP (has similar PCIe module as i.MX6Q) Sabre SD with external clock option

Schematics
Design files, including hardware schematics, Gerbers, and OrCAD files for i.MX 6QuadPlus

i.MX 6QuadPlus SABRE Development Board|NXP 

General overview

IMX6 PCI with external clocks 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

0 Kudos
541 Views
brianptl
Contributor III

Hi Igor,

Thank you for your response. Our configuration is identical to the Schematic of the i.MX6QP that you provided.

I am still a little confused about the software configuration. I have implemented the C changes to pci-imx6.c as per the link.

In the "General Overview" link you provided, it has the following clock configuration:

clocks = <&clks IMX6QDL_CLK_PCIE_AXI>,
                 <&clks IMX6QDL_CLK_LVDS1_IN>,
                 <&clks IMX6QDL_CLK_SATA_REF_100M>;

clock-names = "pcie", "pcie_bus", "pcie_phy";

However the imx6qp.dtsi file has the following:

clocks = <&clks IMX6QDL_CLK_PCIE_REF_125M>, <&clks IMX6QDL_PLL6_BYPASS>,                      <&clks IMX6QDL_PLL6_BYPASS_SRC>,                      <&clks IMX6QDL_CLK_LVDS1_GATE>, <&clks IMX6QDL_CLK_PCIE_AXI>;
clock-names = "pcie_phy", "pcie_ext", "pcie_ext_src", "pcie_bus", "pcie";

This equates to the following:

Clock nameGeneral Overviewimx6qp.dtsi
pcieIMX6QDL_CLK_PCIE_AXI
IMX6QDL_CLK_PCIE_AXI
pcie_busIMX6QDL_CLK_LVDS1_IN
IMX6QDL_CLK_LVDS1_GATE
pcie_phyIMX6QDL_CLK_SATA_REF_100M
IMX6QDL_CLK_PCIE_REF_125M
pcie_ext
N/A
IMX6QDL_PLL6_BYPASS
pcie_ext_src
N/A
IMX6QDL_PLL6_BYPASS_SRC

If I try to use the setup in the "General Overview" link, I get the following error in dmesg:

pcie_ext clock source missing or invalid

If I try to use the imx6qp.dtsi then the kernel hangs on boot

Starting kernel ...

I suspect that the imx6qp.dtsi file is closer to the settings that I need, but it would be good to have this confirmed. Can you please advise?

Thanks,

Brian.

0 Kudos
541 Views
igorpadykov
NXP TechSupport
NXP TechSupport

Hi Brian

what bsp used in the case, in nxp bsps located at source.codeaurora.org/external/imx

all changes for using ext_osc = <1> are already implemented, for hardware one can follow

i.MX6QP Sabre SD schematic SPF-28857.

"General Overview" link was given for understanding how it works, software implementation

can be different for different kernels.

Best regards
igor

0 Kudos
541 Views
brianptl
Contributor III

Hi Igor,

Our BSP is from here: https://www.nxp.com/support/developer-resources/run-time-software/i.mx-developer-resources/i.mx-soft...

Android 8.0.0 Oreo (O8.0.0_1.0.0, 4.9 kernel)[Current Release] Supports i.MX 6QuadPlus, i.MX 6Quad, i.MX 6DualPlus, i.MX 6Dual, i.MX 6DualLite, i.MX 6Solo, i.MX 6SoloX, i.MX 6SoloLite, i.MX 7Dual

Indeed our kernel folder does come from your link:

cd ~/android_build/vendor/nxp-opensource/kernel_imx/arch/arm/boot/dts

git remote -v

android-imx     https://source.codeaurora.org/external/imx/linux-imx (fetch)

android-imx     https://source.codeaurora.org/external/imx/linux-imx (push)

Here are our current settings for PCIe:

imx6q.dtsi

Our version: No Changes

imx6qdl.dtsi

Our version:

(Bold copied from imx6qp )

       pcie: pcie@0x01000000 {
            compatible = "fsl,imx6q-pcie", "snps,dw-pcie";
            reg = <0x01ffc000 0x04000>,
             <0x01f00000 0x80000>;
            reg-names = "dbi", "config";
            #address-cells = <3>;
            #size-cells = <2>;
            device_type = "pci";
            ranges = <0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */
                 0x82000000 0 0x01000000 0x01000000 0 0x00f00000>; /* non-prefetchable memory */
            num-lanes = <1>;
            interrupts = <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;
            interrupt-names = "msi";
            #interrupt-cells = <1>;
            interrupt-map-mask = <0 0 0 0x7>;
            interrupt-map = <0 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>,
             <0 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>,
             <0 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>,
             <0 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;

            clocks = <&clks IMX6QDL_CLK_PCIE_REF_125M>, <&clks IMX6QDL_PLL6_BYPASS>,
                 <&clks IMX6QDL_PLL6_BYPASS_SRC>,
                 <&clks IMX6QDL_CLK_LVDS1_GATE>, <&clks IMX6QDL_CLK_PCIE_AXI>;
            clock-names = "pcie_phy", "pcie_ext", "pcie_ext_src", "pcie_bus", "pcie";

            fsl,max-link-speed = <2>;
            status = "disabled";
        }

;

imx6qdl-sabresd.dtsi

Our version:

    &pcie {
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_pcie>;
        reset-gpio = <&gpio5 31 GPIO_ACTIVE_LOW>;
        ext_osc = <1>;
        status = "okay";
    };

imx6q-sabresd.dts

Our version: No changes

Our new board does not see our PCIe card:

[    0.713733] PCI: CLS 0 bytes, default 64
[    0.871138] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
[    0.871151] OF: PCI:   No bus range found for /soc/pcie@0x01000000, using [bus 00-ff]
[    0.871176] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
[    0.871192] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
[    1.120275] imx6q-pcie 1ffc000.pcie: phy link never came up
[    1.120287] imx6q-pcie 1ffc000.pcie: Link never came up
[    1.123368] imx6q-pcie 1ffc000.pcie: failed to initialize host
[    1.123455] imx6q-pcie: probe of 1ffc000.pcie failed with error -110
[    2.246048] ehci-pci: EHCI PCI platform driver

Our old board, using the internal IMX clock did see the same card however.

[    0.723840] PCI: CLS 0 bytes, default 64
[    0.881480] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
[    0.881494] OF: PCI:   No bus range found for /soc/pcie@0x01000000, using [bus 00-ff]
[    0.881520] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
[    0.881536] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
[    0.936481] imx6q-pcie 1ffc000.pcie: Link up, Gen1
[    0.936695] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00

I do see on other posts that the IMX6QDL_CLK_LVDS1_GATE setting is set to IMX6QDL_CLK_LVDS1_IN? I have tried this with no success however.

Can you please check my settings and recommend how to further debug this?

Thanks,

Brian.

0 Kudos
541 Views
igorpadykov
NXP TechSupport
NXP TechSupport

Hi Brian

one can attach jtag and test settings for clocks, example below:

route the signal into LVDS1 => PMU_MISC1 bit 12 LVDSCLK1_IBEN = 1b  (Input enabled)
                                             => PMU_MISC1 bit 10 LVDSCLK1_0BEN = 0b  (Output disabled)

LVDS1_CLK_SEL is for output only and does not need to be set.
To route the signal into the PCIE_CLK:

=> CCM_ANALOG_PLL_ENET bits 15:14 BYPASS_CLK_SRC = 01b (CLK1 selected)
=> CCM_ANALOG_PLL_ENET bit 16  BYPASS = 1b (PLL in bypass)
=> CCM_ANALOG_PLL_ENET bit 19  ENABLE_125M = 1b (Enable PCIE_CLOCK)

If LVDS1 is being used to input an external signal into the processor for use as the PCIE_CLK,

please verify that the same signal being routed externally to the PCIE device.

Also may be useful

https://community.nxp.com/message/954737?commentID=954737#comment-954737 

https://community.nxp.com/message/1064360 

Best regards
igor

0 Kudos
541 Views
brianptl
Contributor III

Hi Igor,

I have made some more progress. With the following configuration:

imx6qdl.dtsi

Our version:

        pcie: pcie@0x01000000 {
            compatible = "fsl,imx6q-pcie", "snps,dw-pcie";
            reg = <0x01ffc000 0x04000>,
                  <0x01f00000 0x80000>;
            reg-names = "dbi", "config";
            #address-cells = <3>;
            #size-cells = <2>;
            device_type = "pci";
            ranges = <0x81000000 0 0          0x01f80000 0 0x00010000 /* downstream I/O */
                  0x82000000 0 0x01000000 0x01000000 0 0x00f00000>; /* non-prefetchable memory */
            num-lanes = <1>;
            interrupts = <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;
            interrupt-names = "msi";
            #interrupt-cells = <1>;
            interrupt-map-mask = <0 0 0 0x7>;
            interrupt-map = <0 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>,
                            <0 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>,
                            <0 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>,
                            <0 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;

            clocks = <&clks IMX6QDL_CLK_PCIE_REF_125M>, <&clks IMX6QDL_PLL6_BYPASS>,
                 <&clks IMX6QDL_PLL6_BYPASS_SRC>,
                 <&clks IMX6QDL_CLK_LVDS1_IN>, <&clks IMX6QDL_CLK_PCIE_AXI>;
            clock-names = "pcie_phy", "pcie_ext", "pcie_ext_src", "pcie_bus", "pcie";

            fsl,max-link-speed = <2>;
            status = "disabled";
        };

imx6qdl-sabresd.dtsi

Add in:

&clks {
        assigned-clocks = <&clks IMX6QDL_PLL6_BYPASS_SRC>,
                          <&clks IMX6QDL_PLL6_BYPASS>;                          
        assigned-clock-parents = <&clks IMX6QDL_CLK_LVDS1_IN>,
                                 <&clks IMX6QDL_PLL6_BYPASS_SRC>;
        assigned-clock-rates = <100000000>, <100000000>;
};

I now get the following:

PMU_MISC1n, 0x20C8160

devmem2 0x20C8160

/dev/mem opened.
Memory mapped at address 0xaa931000.
Value at address 0x20C8160 (0xaa931160): 0x8000100B

BitValue
Bit 31 - IRQ_DIG_BO1 = digital regulator brownout interrupts asserted
Bit 12 - LVDSCLK1_IBEN1 = Input buffer enabled
Bit 10 - LVDSCLK1_OBEN0 = Output buffer disabled
Bits 9–5 LVDS2_CLK_SEL0b01011 = SATA_REF — SATA ref clock (100M)

CCM_ANALOG_PLL_ENETn, 0x20C80E0

devmem2 0x20C80E0                                                
/dev/mem opened.
Memory mapped at address 0xb3e3b000.
Value at address 0x20C80E0 (0xb3e3b0e0): 0x17003

BitValue
Bit 19 - ENABLE_125M0 = off?
Bit 16 - BYPASS1 = Bypass the PLL
Bit 15 - 14 - BYPASS_CLK_SRC0b01 = CLK1 — Select the CLK1_N / CLK1_P as source.
Bit 13 - ENABLE1 = Enable the ethernet clock output.
Bit 12 - POWERDOWN1 = Powers down the PLL.
Bit 1 - 0 0 DIV_SELECTethernet reference clock, 0b11 = 125MHz

So my only issue is that Bit 19 is not set compared to your recommendations. I will continue to look at this but I would appreciate any input you have to set bit 19.

Thanks,

Brian.

0 Kudos
541 Views
igorpadykov
NXP TechSupport
NXP TechSupport

Hi Brian

seems bit 19 does not matter as pll is bypassed using external clock.

Best regards
igor

0 Kudos
541 Views
brianptl
Contributor III

Hi Igor,

I am still looking for ways to debug this.

I was looking at this post: https://community.nxp.com/thread/304283

It recommends adjusting the registers ATEOVRD and MPLL_OVRD_IN_LO, however it seems that on our original board with the internal PCIe clock you can only read these registers when a PCIe board is connected, otherwise the system hangs.

Edit:

It would seem that this post: https://community.nxp.com/thread/448174 says that you must use the "IMX6QDL_CLK_SATA_REF_100M" setting as opposed to the "IMX6QDL_CLK_PCIE_REF_125M" setting.

Studying the imx6qp dts and dtsi files does not help either. The schematic clearly shows an external clock source, however the imx6qp.dtsi has the IMX6QDL_CLK_LVDS1_GATE setting which I think is wrong and should be IMX6QDL_CLK_LVDS1_IN.

I am running out of options.

Is there local support (central scotland) apps engineers that we can access to help solve the problem?

Thanks,

Brian.

0 Kudos
541 Views
igorpadykov
NXP TechSupport
NXP TechSupport

Hi Brian

PCIe with external clock was verified on i.MX6QP Sabre SD board, so one can run image on it

and dump necessary registers, compare them with custom board.

For helping customers with porting own boards NXP has special service: Professional Services:

NXP Professional Services|NXP 

Best regards
igor

0 Kudos
541 Views
brianptl
Contributor III

Hi Igor,

We have managed to get PCIe Gen1 running. There is a critical piece of code in the function "imx6_pcie_deassert_core_reset":

    case IMX6QP:
        regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
                   IMX6Q_GPR1_PCIE_SW_RST, 0);

        udelay(200);

        /* Configure the PHY when 100Mhz external OSC is used as input clock */
        if (!imx6_pcie->ext_osc)
            break;

        mdelay(4);
        pcie_phy_read(imx6_pcie, SSP_CR_SUP_DIG_MPLL_OVRD_IN_LO, &val);
        /* MPLL_MULTIPLIER [8:2] */
        val &= ~(0x7F << 2);
        val |= (0x19 << 2);
        /* MPLL_MULTIPLIER_OVRD [9:9] */
        val |= (0x1 << 9);
        pcie_phy_write(imx6_pcie, SSP_CR_SUP_DIG_MPLL_OVRD_IN_LO, val);
        mdelay(4);

        pcie_phy_read(imx6_pcie, SSP_CR_SUP_DIG_ATEOVRD, &val);
        /* ref_clkdiv2 [0:0] */
        val &= ~0x1;
        /* ateovrd_en [2:2] */
        val |=  0x4;
        pcie_phy_write(imx6_pcie, SSP_CR_SUP_DIG_ATEOVRD, val);
        mdelay(4);

        break;
    case IMX6Q:
        regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
                   IMX6Q_GPR1_PCIE_SW_RST, 0);
        /*
         * some delay are required by 6qp, after the SW_RST is
         * cleared, before access the cfg register.
         */
        udelay(200);
        break;

As you can see, the code is only active for the IMX6QP case, not the IMX6Q case.

When we enable the same code for the IMX6Q we can see our PCIe board come up in Gen1 mode.

We are now investigating why it does not come up in Gen2 mode - do you have any thoughts on this? Is this purely down to eye-diagram tweaking via swing settings etc at this point, or is there some configuration we need to set?

Thanks,

Brian.

0 Kudos
541 Views
igorpadykov
NXP TechSupport
NXP TechSupport

Hi Brian

this may be casued by signal quality issues (noise, weak signal due to board layout or material losses)

one can use for invstigation AN4784 PCIe Certification Guide

https://www.nxp.com/docs/en/application-note/AN4784.pdf 

and test with several pcie cards.

Best regards
igor

0 Kudos
542 Views
brianptl
Contributor III

Hi Igor + other interested parties,

We have received support through our supplier and internally within NXP to understand why we were not seeing Gen2 reported from the driver.

The summary ( as it looks just now ) is that there is a bug in the pci-imx6.c reporting.

If you read the register PCIE_RC_LCSR [IMX6QDRM.pdf, Rev 5, 06/2018 - page 4254] when you are fully booted, it will have the correct speed, even though the read during the driver initialisation has reported Gen1.

e.g. We inserted two cards, a Gen1 card and a Gen2 card and got the following results for PCIE_RC_LCSR when booted:

Gen1: 0x30110040
Gen2: 0x30120040
We used the utility Devmem2 to read this value. For our Android build we had to enable the following in the kernel:
CONFIG_DEVMEM=y
CONFIG_DEVKMEM=y
We then compiled the source for Devmem2.
The result looks like the following:
# devmem2 0x01ffc080
/dev/mem opened.
Memory mapped at address 0xb5fb8000.
Value at address 0x1FFC080 (0xb5fb8080): 0x30120040
You can also run lspci -vv and it will tell you the PCIe speed, with 5GT/s = Gen2, 2.5GT/s = Gen1.
01:00.0 Class 0106: Device 1b4b:9215 (rev 11) (prog-if 01)
        Subsystem: Device 1b4b:9215
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 306
        Region 0: I/O ports at 1020 [size=8]
        Region 1: I/O ports at 1030 [size=4]
        Region 2: I/O ports at 1028 [size=8]
        Region 3: I/O ports at 1034 [size=4]
        Region 4: I/O ports at 1000 [size=32]
        Region 5: Memory at 01100000 (32-bit, non-prefetchable) [size=2K]
        Expansion ROM at 01200000 [size=256K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: 28534000  Data: 0001
        Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [e0] SATA HBA v0.0 BAR4 Offset=00000004
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Kernel driver in use: ahci
Note the lspci built in to Android does not support the -vv switches. I had to copy the Yocto built lspci over, along with it's dependencies.
Find and copy the following files to the Android system (need to remount the / to be read/write)
Create a /lib folder in Android
├── lib
│   ├── ld-linux-armhf.so.3
│   ├── libcap.so.2
│   ├── libc.so.6
│   ├── libgcc_s.so.1
│   ├── libkmod.so.2
│   ├── libpci.so.3
│   ├── libpthread.so.0
│   ├── libresolv.so.2
│   ├── librt.so.1
│   ├── libudev.so.1
│   └── libz.so.1
└── lspci

If you need to make the Android filesystem writeable:

mount -o rw,remount /

Thanks,

Brian.

View solution in original post

541 Views
brianptl
Contributor III

Hi Igor,

Thanks for your reply. It seems like we might be on the right track here. I compliled devmem2 to allow me to view registers and I see the following:

For PMU_MISC1n the address is 20C_8000h base + 160h offset + (4d × i), where i=0d to 3d = 0x20C8160

devmem2 0x20C8160                                             
/dev/mem opened.
Memory mapped at address 0xb6b90000.
Value at address 0x20C8160 (0xb6b90160): 0x8000000B

This gives:

BitValue
Bit 31 - IRQ_DIG_BO1 = digital regulator brownout interrupts asserted
Bit 12 - LVDSCLK1_IBEN0 = Input buffer disabled
Bit 10 - LVDSCLK1_OBEN0 = Output buffer disabled
Bits 9–5 LVDS2_CLK_SEL01011 = SATA_REF — SATA ref clock (100M)

So it looks like I need to enable the LVDSCLK1_IBEN bit.

I will investigate how to do this via the device tree etc. Please let me know if you already know how to do this as I find the device tree / driver abstraction difficult to navigate.

Thanks,

Brian.

0 Kudos