QT4241 PCIe as an Enpoint

[sandhya_bs]

Hi,

PCIe3 is configured as endpoint in a T4241 processor (HOST_AGT_PEX is configured accordingly).

The endpoint is connected to a PCIe switch. The Root Complex is another T4241 processor. Both the boards are communicating over a VPX backplane. During enumeration at Root Complex able to get all other port details of Pcie switch to which Endpoint is connected, but unable to get Endpoint details (CONFIG[CFG_READY] is set)

Is there any other configuration to be taken care of?  

Thanks and regards,

Sandhya

[sandhya_bs]

Hi,

Pcie Switch is configured for Single partition mode (By default Port 0 will be upstream and all other ports will be down stream). 

At Endpoint side Port which is connected RootComplex is configured as upstream and all other ports are configured as downstream.

After the Endpoint configuration in RCW, we are able to get following prints at booting time

PCIe3: Endpoint, Outbound memory range: e0000000:100000000
PCI reg:0 0000000c20000000:00000000e0000000 0000000020000000 00000000
PCI reg:1 0000000ff8010000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000df000007 0000000000fffff9 00000100
PCI reg:3 0000000000000000:0000000000000000 00000000df000007 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000400000000 00000108

We have provided delay while booting, we have taken care so that Enpoint processor will boot first later Rootcomplex will comes up.

After enumeration at RC side getting following output:

at uboot:

PCIe1: Root Complex, no link, regs @ 0xfe240000
PCIe1: Bus 00 - 00
PCIe3: Root Complex, x4 gen2, regs @ 0xfe260000
02:00.0 - 111d:808c - Bridge device
03:02.0 - 111d:808c - Bridge device
03:04.0 - 111d:808c - Bridge device
03:06.0 - 111d:808c - Bridge device
    06:00.0 - 111d:808c - Bridge device
       07:02.0 - 111d:808c - Bridge device
       07:06.0 - 111d:808c - Bridge device
       07:08.0 - 111d:808c - Bridge device
       07:0c.0 - 111d:808c - Bridge device
       07:10.0 - 111d:808c - Bridge device
03:08.0 - 111d:808c - Bridge device
03:0c.0 - 111d:808c - Bridge device
03:10.0 - 111d:808c - Bridge device
PCIe3: Bus 01 - 0f
PCIe4: Root Complex, x2 gen2, regs @ 0xfe270000
11:00.0 - 10ee:7012 - Build before PCI Rev2.0

after linux comes up: 

root@t4240rdb:~# lspci
02:0c.0 Class 0604: 111d:808c
06:08.0 Class 0604: 111d:808c
06:10.0 Class 0604: 111d:808c
01:00.0 Class 0604: 111d:808c
02:06.0 Class 0604: 111d:808c
06:0c.0 Class 0604: 111d:808c
00:00.0 Class 0604: 1957:0441
02:02.0 Class 0604: 111d:808c
05:00.0 Class 0604: 111d:808c
06:06.0 Class 0604: 111d:808c
02:08.0 Class 0604: 111d:808c
02:10.0 Class 0604: 111d:808c
06:02.0 Class 0604: 111d:808c
02:04.0 Class 0604: 111d:808c
root@t4240rdb:~#

During enumeration at RC, it is able to enumerate PCI switch of other processor card but unable to get Endpoint information.

In our case both RC and EP are T4241 processor card which are having common architecture. We have configured one as RC and one as EP.

[ufedor]

Please provide complete block diagram representing the T4241 RC connection to the T4241 EP.

[sandhya_bs]

Please find the attached diagram

[ufedor]

Please provide U-Boot log of the T4241 EP card.

Please consider that T4241 PCIe EP does not initiate link training.

[sandhya_bs]

Please find the following u-boot log for an 

U-Boot 2016.01 (Apr 11 2019 - 13:00:14 +0530)

CPU0: T4241, Version: 2.0, (0x82400020)
Core: e6500, Version: 2.0, (0x80400120)
Clock Configuration:
CPU0:1800 MHz, CPU1:1800 MHz, CPU2:1800 MHz, CPU3:1800 MHz,
CPU4:1800 MHz, CPU5:1800 MHz, CPU6:1800 MHz, CPU7:1800 MHz,
CPU8:1800 MHz, CPU9:1800 MHz, CPU10:1800 MHz, CPU11:1800 MHz,
CCB:600 MHz,
DDR:800 MHz (1600 MT/s data rate) (Asynchronous), IFC:150 MHz
FMAN1: 700 MHz
FMAN2: 600 MHz
QMAN: 300 MHz
PME: 466.667 MHz
L1: D-cache 32 KiB enabled
I-cache 32 KiB enabled
I2C: ready
Board: T4241, SERDES Reference Clocks:
SERDES1=125MHz SERDES2=125MHz
SERDES3=100MHz SERDES4=100MHz
SPI: ready
DRAM: Initializing DDR
10 GiB left unmapped
12 GiB (DDR3, 64-bit, CL=11, ECC on)

Value From Fuse Register = 10250

Corresponding value to be Written = 42

VID Write Successfull
Flash: 256 MiB
L2: 2 MiB enabled
enable l2 for cluster 1 fec60000
enable l2 for cluster 2 feca0000
Corenet Platform Cache: 1.5 MiB enabled
A007186 Serdes PLL locked
A007186 Serdes PLL locked
NAND: 8192 MiB
MMC: FSL_SDHC: 0

PCIe3: Endpoint, Outbound memory range: e0000000:100000000

PCI reg:0 0000000c20000000:00000000e0000000 0000000020000000 00000000
PCI reg:1 0000000ff8010000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000df000007 0000000000fffff9 00000100
PCI reg:3 0000000000000000:0000000000000000 00000000df000007 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000400000000 00000108
x4 gen2, regs @ 0xfe260000
PCIe3: Bus 00 - 00
In: serial
Out: serial
Err: serial
Net:
Fman1: Uploading microcode version 108.4.5
Updating Boot Status Successful
Hit any key to stop autoboot: 2 0
Q4241_uboot>
Q4241_uboot>m md 0xfe260000
fe260000: 8000f800 57194104 00000000 0013ffff ....W.A.........
fe260010: 0400ffff 00080029 00008000 00000000 .......)........
fe260020: 00000280 00000000 000cffff 00000000 ................
fe260030: 00000000 00000000 00000000 00000000 ................
fe260040: 00000134 00000000 00000000 00000000 ...4............
fe260050: 00000000 00000000 00000000 00000000 ................

can you please tell me is there any way to find out link training is successful at T4241 PCIe EP.

And one more observation, When we tried modify Endpoint configuration registers(like tried modify command register) the values are getting reset once the enumeration happens at Root complex side.

Is there any configuration to be modified or to be taken care at Endpoint side? 

[ufedor]

> can you please tell me is there any way to find out link training is successful at T4241 PCIe EP.

It is required to read the LTSSM_SC value in the PEX_CSR0 - refer to the QorIQ T4240 Reference Manual, 19.4.41 PEX control/status register 0 (CSR0). The value should be 010001b - L0.

[vinothkumars]

Sandhya,

Please provide the log and schematic and also check the provided Reset Configuration Word (RCW) value is proper or not.