T4241, PCIe enumeration issue

[immbalu]

Hello,

We are using T4241, with the following RCW configurations, also added uboot prints and lspci utility prints.  In RCW PCIe1 is selected to be 5GT/s with 8 lane. But link width is downgrading to x1 lane.

Thanks for any information!

Reset Configuration Word (RCW):
00000000: 0c080012 0e12120e 00000000 08000000
00000010: 04360848 30547a00 ec03f000 f9000000
00000020: 00080000 00000000 00000000 0003e000
00000030: 00000000 50000000 00000000 00000038

Here are the uboot prints:

U-Boot 2016.012.0+ga9b437f (Mar 28 2019 - 16:23:11 +0530)

CPU0: T4241, Version: 2.0, (0x82400020)
Core: e6500, Version: 2.0, (0x80400120)
Clock Configuration:
CPU0:1800 MHz, CPU1:1800 MHz, CPU2:1800 MHz, CPU3:1800 MHz,
CPU4:1800 MHz, CPU5:1800 MHz, CPU6:1800 MHz, CPU7:1800 MHz,
CPU8:1800 MHz, CPU9:1800 MHz, CPU10:1800 MHz, CPU11:1800 MHz,
CCB:600 MHz,
DDR:800 MHz (1600 MT/s data rate) (Asynchronous), IFC:150 MHz
FMAN1: 700 MHz
FMAN2: 600 MHz
QMAN: 300 MHz
PME: 466.667 MHz
L1: D-cache 32 KiB enabled
I-cache 32 KiB enabled
Reset Configuration Word (RCW):
00000000: 0c080012 0e12120e 00000000 08000000
00000010: 04360848 30547a00 ec03f000 f9000000
00000020: 00080000 00000000 00000000 0003e000
00000030: 00000000 50000000 00000000 00000038
I2C: ready
Board: T4241, SERDES Reference Clocks:
SERDES1=125MHz SERDES2=125MHz
SERDES3=100MHz SERDES4=100MHz
SPI: ready
DRAM: Initializing DDR
10 GiB left unmapped
12 GiB (DDR3, 64-bit, CL=11, ECC on)

Value From Fuse Register = 10250

Corresponding value to be Written = 42

VID Write Successfull
Flash: 256 MiB
L2: 2 MiB enabled
enable l2 for cluster 1 fec60000
enable l2 for cluster 2 feca0000
Corenet Platform Cache: 1.5 MiB enabled
A007186 Serdes PLL locked
A007186 Serdes PLL locked
NAND: 0 MiB
MMC: FSL_SDHC: 0
PCIe1: Root Complex, x1 gen2, regs @ 0xfe240000
01:00.0 - 10ee:7028 - Memory controller
PCIe1: Bus 00 - 01
PCIe3: Root Complex, no link, regs @ 0xfe260000
PCIe3: Bus 02 - 02
PCIe4: Root Complex, no link, regs @ 0xfe270000
PCIe4: Bus 03 - 03
In: serial
Out: serial
Err: serial
Net:
Fman1: Uploading microcode version 108.4.5
Could not get PHY for FM_TGEC_MDIO: addr 16
Failed to connect

Fman2: Uploading microcode version 108.4.5
Could not get PHY for FSL_MDIO0: addr 1
Failed to connect
Could not get PHY for FSL_MDIO0: addr 2
Failed to connect
PHY reset timed out
FM1@TGEC1, FM2@DTSEC1, FM2@DTSEC2, FM2@DTSEC5 [PRIME], FM2@DTSEC6, FM2@DTSEC9, FM2@DTSEC10

POST Start:

POST Complete

U-boot Software Version : 1.0

tap env addr 0
Updating Boot Status Successful
Hit any key to stop autoboot: 0
Q4241_uboot>

lspci output prints

root@t4240rdb:~# ./lspci -vvv -d 10ee:7028
0000:01:00.0 Class 0580: Device 10ee:7028
        Subsystem: Device 10ee:0007
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 40
        IOMMU group: 60
        Region 0: Memory at c00000000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [9c] MSI-X: Enable- Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000000
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

Regards,

Balaji.G

[ufedor]

Please provide the processor connection schematics as searchable PDF.

[immbalu]

Hello @ufedor ,

Thanks for the reply.,

i'm sharing the schematic connections between processor and fpga. please find the attached pdf.

Thanks,

Balaji.G

[ufedor]

In the provided schematics T4241 PCIe lanes are connected to the FPGA in arbitrary order.

Please consider that order of lanes is very important.

For the x8 configuration it could be either (T4241 <-> FPGA):

0 <-> 0

1 <-> 1

..

6 <-> 6

7 <-> 7

or

0 <-> 7

1 <-> 6

...

6 <-> 1

7 <-> 0

As result only T4241 lane 7 connected to the FPGA lane 0 works.

[immbalu]

Hello @ufedor ,

In the schismatics,

lane 0 is connected as,

SD3_TX0 is connected to MGTXRXP3_116 and SD3_TX0_B is connected to MGTXRXN3_116,

MGTXTXN3_116 is connected to SD3_RX0_B and MGTXTXP3_116 is connected to SD3_RX0 

lane 1 is connected as,

SD3_TX1 is connected to MGTXRXP2_116 and SD3_TX1_B is connected to MGTXRXN2_116,

MGTXTXN2_116 is connected to SD3_RX1_B and MGTXTXP2_116 is connected to SD3_RX1 

and same as other lanes which is in lane reversal order. This can be confirmed by referring to https://www.xilinx.com/support/documentation/boards_and_kits/kc705_Schematic_xtp132_rev1_1.pdf schismatics page no 9.

Also, some times pcie is detecting as x2 gen 1 and sometimes its detecting as x1 gen 2.

Thanks,

Balaji.G.  

[ufedor]

How the PCIe lanes are implemented and what are the lanes lengths?

Did you simulate the PCIe lanes in HyperLynx?

Have you reported the issue to Xilinx and what is their response?

[immbalu]

Found the issue with SI analysis

Thanks,

Balaji.G