T4241, PCIe enumeration issue

immbalu — Wed, 05 Jan 2022 15:52:12 GMT

Hello,

We are using T4241, with the following RCW configurations, also added uboot prints and lspci utility prints. In RCW PCIe1 is selected to be 5GT/s with 8 lane. But link width is downgrading to x1 lane.

Thanks for any information!

Reset Configuration Word (RCW):
00000000: 0c080012 0e12120e 00000000 08000000
00000010: 04360848 30547a00 ec03f000 f9000000
00000020: 00080000 00000000 00000000 0003e000
00000030: 00000000 50000000 00000000 00000038

Here are the uboot prints:

U-Boot 2016.012.0+ga9b437f (Mar 28 2019 - 16:23:11 +0530) CPU0: T4241, Version: 2.0, (0x82400020) Core: e6500, Version: 2.0, (0x80400120) Clock Configuration: CPU0:1800 MHz, CPU1:1800 MHz, CPU2:1800 MHz, CPU3:1800 MHz, CPU4:1800 MHz, CPU5:1800 MHz, CPU6:1800 MHz, CPU7:1800 MHz, CPU8:1800 MHz, CPU9:1800 MHz, CPU10:1800 MHz, CPU11:1800 MHz, CCB:600 MHz, DDR:800 MHz (1600 MT/s data rate) (Asynchronous), IFC:150 MHz FMAN1: 700 MHz FMAN2: 600 MHz QMAN: 300 MHz PME: 466.667 MHz L1: D-cache 32 KiB enabled I-cache 32 KiB enabled Reset Configuration Word (RCW): 00000000: 0c080012 0e12120e 00000000 08000000 00000010: 04360848 30547a00 ec03f000 f9000000 00000020: 00080000 00000000 00000000 0003e000 00000030: 00000000 50000000 00000000 00000038 I2C: ready Board: T4241, SERDES Reference Clocks: SERDES1=125MHz SERDES2=125MHz SERDES3=100MHz SERDES4=100MHz SPI: ready DRAM: Initializing DDR 10 GiB left unmapped 12 GiB (DDR3, 64-bit, CL=11, ECC on) Value From Fuse Register = 10250 Corresponding value to be Written = 42 VID Write Successfull Flash: 256 MiB L2: 2 MiB enabled enable l2 for cluster 1 fec60000 enable l2 for cluster 2 feca0000 Corenet Platform Cache: 1.5 MiB enabled A007186 Serdes PLL locked A007186 Serdes PLL locked NAND: 0 MiB MMC: FSL_SDHC: 0 PCIe1: Root Complex, x1 gen2, regs @ 0xfe240000 01:00.0 - 10ee:7028 - Memory controller PCIe1: Bus 00 - 01 PCIe3: Root Complex, no link, regs @ 0xfe260000 PCIe3: Bus 02 - 02 PCIe4: Root Complex, no link, regs @ 0xfe270000 PCIe4: Bus 03 - 03 In: serial Out: serial Err: serial Net: Fman1: Uploading microcode version 108.4.5 Could not get PHY for FM_TGEC_MDIO: addr 16 Failed to connect Fman2: Uploading microcode version 108.4.5 Could not get PHY for FSL_MDIO0: addr 1 Failed to connect Could not get PHY for FSL_MDIO0: addr 2 Failed to connect PHY reset timed out FM1@TGEC1, FM2@DTSEC1, FM2@DTSEC2, FM2@DTSEC5 [PRIME], FM2@DTSEC6, FM2@DTSEC9, FM2@DTSEC10 POST Start: POST Complete U-boot Software Version : 1.0 tap env addr 0 Updating Boot Status Successful Hit any key to stop autoboot: 0 Q4241_uboot>

lspci output prints

root@t4240rdb:~# ./lspci -vvv -d 10ee:7028 0000:01:00.0 Class 0580: Device 10ee:7028 Subsystem: Device 10ee:0007 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 40 IOMMU group: 60 Region 0: Memory at c00000000 (32-bit, non-prefetchable) [size=2K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x1 (downgraded) TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [9c] MSI-X: Enable- Count=1 Masked- Vector table: BAR=0 offset=00000000 PBA: BAR=0 offset=00000000 Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

Regards,

Balaji.G

Re: T4241, PCIe enumeration issue

ufedor — Wed, 05 Jan 2022 18:44:40 GMT

Please provide the processor connection schematics as searchable PDF.

Re: T4241, PCIe enumeration issue

immbalu — Thu, 06 Jan 2022 12:35:07 GMT

Hello @ufedor ,

Thanks for the reply.,

i'm sharing the schematic connections between processor and fpga. please find the attached pdf.

Thanks,

Balaji.G

Re: T4241, PCIe enumeration issue

ufedor — Fri, 07 Jan 2022 10:24:36 GMT

In the provided schematics T4241 PCIe lanes are connected to the FPGA in arbitrary order.

Please consider that order of lanes is very important.

For the x8 configuration it could be either (T4241 <-> FPGA):

0 <-> 0

1 <-> 1

6 <-> 6

7 <-> 7

0 <-> 7

1 <-> 6

...

6 <-> 1

7 <-> 0

As result only T4241 lane 7 connected to the FPGA lane 0 works.

Re: T4241, PCIe enumeration issue

immbalu — Mon, 10 Jan 2022 07:00:33 GMT

Hello @ufedor ,

In the schismatics,

lane 0 is connected as,

SD3_TX0 is connected to MGTXRXP3_116 and SD3_TX0_B is connected to MGTXRXN3_116,

MGTXTXN3_116 is connected to SD3_RX0_B and MGTXTXP3_116 is connected to SD3_RX0

lane 1 is connected as,

SD3_TX1 is connected to MGTXRXP2_116 and SD3_TX1_B is connected to MGTXRXN2_116,

MGTXTXN2_116 is connected to SD3_RX1_B and MGTXTXP2_116 is connected to SD3_RX1

and same as other lanes which is in lane reversal order. This can be confirmed by referring to https://www.xilinx.com/support/documentation/boards_and_kits/kc705_Schematic_xtp132_rev1_1.pdf schismatics page no 9.

Also, some times pcie is detecting as x2 gen 1 and sometimes its detecting as x1 gen 2.

Thanks,

Balaji.G.

Re: T4241, PCIe enumeration issue

ufedor — Mon, 10 Jan 2022 08:52:05 GMT

How the PCIe lanes are implemented and what are the lanes lengths?

Did you simulate the PCIe lanes in HyperLynx?

Have you reported the issue to Xilinx and what is their response?

Re: T4241, PCIe enumeration issue

immbalu — Mon, 21 Feb 2022 04:49:13 GMT

Found the issue with SI analysis

Thanks,

Balaji.G

T-SeriesのトピックRe: T4241, PCIe enumeration issue

T4241, PCIe enumeration issue

Re: T4241, PCIe enumeration issue

Re: T4241, PCIe enumeration issue

Re: T4241, PCIe enumeration issue

Re: T4241, PCIe enumeration issue

Re: T4241, PCIe enumeration issue

Re: T4241, PCIe enumeration issue