T1024 PCIe RC3 device enumeration

[eduardkromskoy]

Hi,

We have board based on T1024RDB reference design. Sometimes, especially after hot reboot, u-boot fails to detect device (PCIe switch) connected to RC3. I don't see any problems with device (network controller) connected to RC2. I noticed that u-boot after initializing Root Complex registers just tries to read config space for all possible bus-dev-func combinations to find devices. So, sometimes all these functions return 0xffffffff as no devices are connected to RC3; however, from u-boot command line I can see PCIe switch (obviously not configured) and can modify its config space. On a power cycle (cold boot) we don't see this problem; all PCIe devices are found. Does anyone have any idea where to look to?

Thank you,

Ed 

[eduardkromskoy]

Table 24 has Note 1 of T1024 Data sheet states  "PORESET_B must be driven asserted before the core and platform power supplies are powered up". 

Can anyone clarify is it bad language or T1024 indeed must not be reset by asserting active PORESET_B signal without interrupting core voltages. 

[eduardkromskoy]

Just to clarify: it is same reset mechanism and same code running on cold and hot boot. Just in one case power was off and then T1024 CPU starting; and in other case it was running but then received reset pulse triggered by reset button (or watchdog, etc).

[ufedor]

Please consider: when PORESET_B and other involved signals are applied in accordance with the processor's Data Sheet requirements (Table 24. RESET Initialization timing specifications) - there should be no difference in the processor's behaviour for "cold" and "hot" boots.

[eduardkromskoy]

Table 24 has Note 1 which states  "PORESET_B must be driven asserted before the core and platform power supplies are powered up". Does it mean there is no such thing as hot boot.in PPC land? According to that note we must switch off CPU core voltages before asserting PORESET_B and starting boot process again. Is my understanding correct? Doesn't make much sense except to hide some HW bugs.

[eduardkromskoy]

Short answer: Reset button form reset pulse to main CPU, nothing else; then main CPU initialize all peripheral devices.

Long answer:

Cold re-boot involves power cycle, i.e.interruption for power voltages. Technically it is possible to send command to power management unit (and use corresponding input signal of PM chip for "reset" button); and this how I see PPC hardware often implemented. Our HW guys have reasons not to interrupt power supplies, so we don't use this functionality of power management chip. Instead we have FPGA that handles "reset" button and corresponding command from CPU (for reboot from command line) and it forms reset pulse to main T1024 CPU. It boots from IFC (MMC flash) and use GPIO to reset and init other hardware. Normally it loads bit file to FPGA over SPI and form reset pulses to FPGA and PCIe switch. I already tried to take FPGA out of this and didn't see any change; checked all pulses and signals. I don't see anything wrong with just host CPU being reset and doing system boot up sequence by itself. However I noticed that PowerPC doesn't keep cold/hot boot status bit (as some TI DSPs do) but there are some distinctions in boot procedure. I am afraid this is where problem may lay

.

[eduardkromskoy]

One more thing I noticed.

If it is hot boot and PCIe switch on RC3 not detected, if I do delay, say, 1 second and re-scan bus (changes to bus scan in pci.c to rescan bus if nothing found) I don't see device on second bus scan too. However, there is access from u-boot command prompt.

If I change fsl_pci_init.c to similar logic: if no devices found, delay and call fsl_pcie_init_ctrl() again then it detects PCIe switch on second run. Delay should be 300ms or more. That's why I am suspicious to Root Complex initialization.  

Unfortunatelly, we don't have PCIe bus analyzer and renting one would take time. 

[ufedor]

How exactly hot re-boot is implemented?

Why GPIOs are needed to reset external devices? - i.e. why it is not possible to use the same reset mechanism as for the cold boot?

[eduardkromskoy]

This is first what we did. We checked all reset pulses are wide enough and their timing (offsets) are correct. Long enough delay is already there; as well as we tried to increase it to seconds with same results. Please note this GPIO section to reset external hardware is executed every time with no regard cold or hot boot. First we suspected that PCIe switch is behaving differently when it's power cycled or just reset however it doesn't seem to be the case. It doesn't reply to read config space access on hot boot, but recovers somehow later. So we suspect it either doesn't receive these commands or they are broken (not sent at all) or indeed PCIe switch by some reason discards them. Currently we are working to rent bus analyzer and nail it down; meanwhile I am looking if there is anything in CPU/MPC/RCW that may affect PCIe RC initialization (hot vs cold) and reviewing u-boot code to find why access config space functions might be deferred somehow.  

[ufedor]

I agree that the PCIe bus analyzer could help to clarify at which side of the link the issue cause resides.

[eduardkromskoy]

I attached full logs.

Here is relevant part on cold boot:

PCIE Reset Done PCIe1: Root Complex, Outbound memory range: e0000000:f0000000
PCICSRBAR @ 0xff000000
R0 bus_start: 0 phys_start: 0 size: 80000000
R64 bus_start: 1000000000 phys_start: 0 size: 80000000
PCI reg:0 0000000c00000000:00000000e0000000 0000000010000000 00000000
PCI reg:1 0000000ff8000000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000ff000000 0000000001000000 00000100
PCI reg:3 0000000000000000:0000000000000000 0000000080000000 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000080000000 00000108
PCI Autoconfig: Bus Memory region: [e0000000-efffffff],
Physical Memory [c00000000-c0fffffffx]
PCI Autoconfig: Bus I/O region: [0-ffff],
Physical Memory [ff8000000-ff800ffffx]
no link, regs @ 0xfe240000
PCIe1: Bus 00 - 00
PCIe2: Root Complex, with errors. Clearing. Now 0x00000000Outbound memory range: e0000000:f0000000
PCICSRBAR @ 0xff000000
R0 bus_start: 0 phys_start: 0 size: 80000000
R64 bus_start: 1000000000 phys_start: 0 size: 80000000
PCI reg:0 0000000c10000000:00000000e0000000 0000000010000000 00000000
PCI reg:1 0000000ff8010000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000ff000000 0000000001000000 00000100
PCI reg:3 0000000000000000:0000000000000000 0000000080000000 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000080000000 00000108
PCI Autoconfig: Bus Memory region: [e0000000-efffffff],
Physical Memory [c10000000-c1fffffffx]
PCI Autoconfig: Bus I/O region: [0-ffff],
Physical Memory [ff8010000-ff801ffffx]
x1 gen2, regs @ 0xfe250000
Scanning PCI bus 02
PCI Scan: Found Bus 2, Device 0, Function 0
02:00.0 - 11ab:f400 - Network controller
PCI Autoconfig: BAR 0, Mem, size=0x100000, address=0xe0000000 bus_lower=0xe0100000
PCI Autoconfig: BAR 1, Mem, size=0x4000000, address=0xe4000000 bus_lower=0xe8000000
PCI Autoconfig: BAR 2, Mem, size=0x800000, address=0xe8000000 bus_lower=0xe8800000
PCIe2: Bus 01 - 02
PCIe3: Root Complex, with errors. Clearing. Now 0x00000000Outbound memory range: d0000000:e0000000
PCICSRBAR @ 0xff000000
R0 bus_start: 0 phys_start: 0 size: 80000000
R64 bus_start: 1000000000 phys_start: 0 size: 80000000
PCI reg:0 0000000c20000000:00000000d0000000 0000000010000000 00000000
PCI reg:1 0000000ff8020000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000ff000000 0000000001000000 00000100
PCI reg:3 0000000000000000:0000000000000000 0000000080000000 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000080000000 00000108
PCI Autoconfig: Bus Memory region: [d0000000-dfffffff],
Physical Memory [c20000000-c2fffffffx]
PCI Autoconfig: Bus I/O region: [0-ffff],
Physical Memory [ff8020000-ff802ffffx]
x1 gen2, regs @ 0xfe260000
Scanning PCI bus 04
PCI Scan: Found Bus 4, Device 0, Function 0
04:00.0 - 12d8:2608 - Bridge device
PCI Autoconfig: Found P2P bridge, device 0
PCI Scan: Found Bus 5, Device 1, Function 0
05:01.0 - 12d8:2608 - Bridge device
PCI Autoconfig: Found P2P bridge, device 1
PCI Scan: Found Bus 5, Device 2, Function 0
05:02.0 - 12d8:2608 - Bridge device
PCI Autoconfig: Found P2P bridge, device 2
PCI Scan: Found Bus 5, Device 3, Function 0
05:03.0 - 12d8:2608 - Bridge device
PCI Autoconfig: Found P2P bridge, device 3
PCI Scan: Found Bus 5, Device 4, Function 0
05:04.0 - 12d8:2608 - Bridge device
PCI Autoconfig: Found P2P bridge, device 4
PCI Scan: Found Bus 5, Device 5, Function 0
05:05.0 - 12d8:2608 - Bridge device
PCI Autoconfig: Found P2P bridge, device 5
PCI Scan: Found Bus 10, Device 0, Function 0
0a:00.0 - aaaa:0001 - Base system peripheral
PCI Autoconfig: BAR 0, Mem, size=0x100000, address=0xd0000000 bus_lower=0xd0100000
PCIe3: Bus 03 - 0a

On hot boot (host CPU Reset with no power interruption and at the beginning of u-boot we have code that use GPIO to generate reset pulses to rest of hardware including PCIe switch):

PCIE Reset Done PCIe1: Root Complex, Outbound memory range: e0000000:f0000000
PCICSRBAR @ 0xff000000
R0 bus_start: 0 phys_start: 0 size: 80000000
R64 bus_start: 1000000000 phys_start: 0 size: 80000000
PCI reg:0 0000000c00000000:00000000e0000000 0000000010000000 00000000
PCI reg:1 0000000ff8000000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000ff000000 0000000001000000 00000100
PCI reg:3 0000000000000000:0000000000000000 0000000080000000 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000080000000 00000108
PCI Autoconfig: Bus Memory region: [e0000000-efffffff],
Physical Memory [c00000000-c0fffffffx]
PCI Autoconfig: Bus I/O region: [0-ffff],
Physical Memory [ff8000000-ff800ffffx]
no link, regs @ 0xfe240000
PCIe1: Bus 00 - 00
PCIe2: Root Complex, with errors. Clearing. Now 0x00000000Outbound memory range: e0000000:f0000000
PCICSRBAR @ 0xff000000
R0 bus_start: 0 phys_start: 0 size: 80000000
R64 bus_start: 1000000000 phys_start: 0 size: 80000000
PCI reg:0 0000000c10000000:00000000e0000000 0000000010000000 00000000
PCI reg:1 0000000ff8010000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000ff000000 0000000001000000 00000100
PCI reg:3 0000000000000000:0000000000000000 0000000080000000 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000080000000 00000108
PCI Autoconfig: Bus Memory region: [e0000000-efffffff],
Physical Memory [c10000000-c1fffffffx]
PCI Autoconfig: Bus I/O region: [0-ffff],
Physical Memory [ff8010000-ff801ffffx]
x1 gen2, regs @ 0xfe250000
Scanning PCI bus 02
PCI Scan: Found Bus 2, Device 0, Function 0
02:00.0 - 11ab:f400 - Network controller
PCI Autoconfig: BAR 0, Mem, size=0x100000, address=0xe0000000 bus_lower=0xe0100000
PCI Autoconfig: BAR 1, Mem, size=0x4000000, address=0xe4000000 bus_lower=0xe8000000
PCI Autoconfig: BAR 2, Mem, size=0x800000, address=0xe8000000 bus_lower=0xe8800000
PCIe2: Bus 01 - 02
PCIe3: Root Complex, with errors. Clearing. Now 0x00000000Outbound memory range: d0000000:e0000000
PCICSRBAR @ 0xff000000
R0 bus_start: 0 phys_start: 0 size: 80000000
R64 bus_start: 1000000000 phys_start: 0 size: 80000000
PCI reg:0 0000000c20000000:00000000d0000000 0000000010000000 00000000
PCI reg:1 0000000ff8020000:0000000000000000 0000000000010000 00000001
PCI reg:2 0000000ffe000000:00000000ff000000 0000000001000000 00000100
PCI reg:3 0000000000000000:0000000000000000 0000000080000000 00000108
PCI reg:4 0000000000000000:0000001000000000 0000000080000000 00000108
PCI Autoconfig: Bus Memory region: [d0000000-dfffffff],
Physical Memory [c20000000-c2fffffffx]
PCI Autoconfig: Bus I/O region: [0-ffff],
Physical Memory [ff8020000-ff802ffffx]
x1 gen2, regs @ 0xfe260000
Scanning PCI bus 04
PCIe3: Bus 03 - 04

Technically I don't see difference or hints in logs. If I add debug print into drivers/pci/pci.c function

int pci_hose_scan_bus(struct pci_controller *hose, int bus)

I can see that on hot boot pci_hose_read_config_word(hose, dev, PCI_VENDOR_ID, &vendor); returned vendor is 0xffff for entire bus 4. However, later I can access device on bus 4 from command prompt.

[ufedor]

> at the beginning of u-boot we have code that use GPIO to generate reset pulses to rest of hardware including PCIe switch)

Please insert a delay after this code to ensure that all devices will be able to complete their reset sequences.

[ufedor]

Please provide U-Boot logs corresponding to both cases (hot and cold).