LX2160A RDB rev2 board. PCIe devices not enumerated behind PCIe bridge

[kikote]

Hello all,

Enumeration of PCIe devices behind a PCIe bridge on LX2160ARDBrev2 fails, tested in LSDK 21.08 or LSDK 20.12.

Tested using two type of PCIe bridges - Pericom [12d8:2404] 1/3 and PLX8714 1/4

In both cases I see similar message in kernel log:
pci_bus 0001:03: busn_res: can not insert [bus 03] under [bus 02] (conflicts with (null) [bus 02])

Prebuilt LX2160 SD image used, downloaded from https://lsdk.github.io/components.html

https://www.nxp.com/lgfiles/sdk/lsdk2108/firmware_lx2160ardb_rev2_sdboot.img

Booting RDB, 3 PCIe cards are inserted @ Pericom switch (usb3.0 host + SATA hpost + dual eth Intel card),
but no card is visible. All 3 cards are correctly detected when plugged into RDB pcie slot.

...
[ 2.498146] layerscape-pcie 3600000.pcie: host bridge /soc/pcie@3600000 ranges:
[ 2.505463] layerscape-pcie 3600000.pcie: MEM 0x9040000000..0x907fffffff -> 0x0040000000
[ 2.514050] layerscape-pcie 3600000.pcie: PCI host bridge to bus 0000:00
[ 2.520747] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.526226] pci_bus 0000:00: root bus resource [mem 0x9040000000-0x907fffffff] (bus address [0x40000000-0x7fffffff])
[ 2.536761] pci 0000:00:00.0: [1957:8d80] type 01 class 0x060400
[ 2.542814] pci 0000:00:00.0: supports D1 D2
[ 2.547076] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 2.554584] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 2.559918] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[ 2.567235] layerscape-pcie 3800000.pcie: MEM 0xa040000000..0xa07fffffff -> 0x0040000000
[ 2.575822] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0001:00
[ 2.582517] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 2.587997] pci_bus 0001:00: root bus resource [mem 0xa040000000-0xa07fffffff] (bus address [0x40000000-0x7fffffff])
[ 2.598526] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400
[ 2.604588] pci 0001:00:00.0: supports D1 D2
[ 2.608851] pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 2.615875] pci 0001:01:00.0: [12d8:2404] type 01 class 0x060400
[ 2.623192] pci 0001:01:00.0: supports D1 D2
[ 2.627455] pci 0001:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.635989] pci 0001:02:01.0: [12d8:2404] type 01 class 0x060400
[ 2.643316] pci 0001:02:01.0: supports D1 D2
[ 2.647578] pci 0001:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.655036] pci 0001:02:02.0: [12d8:2404] type 01 class 0x060400
[ 2.662363] pci 0001:02:02.0: supports D1 D2
[ 2.666625] pci 0001:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.674093] pci 0001:02:03.0: [12d8:2404] type 01 class 0x060400
[ 2.681425] pci 0001:02:03.0: supports D1 D2
[ 2.685687] pci 0001:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.694199] pci_bus 0001:03: busn_res: can not insert [bus 03] under [bus 02] (conflicts with (null) [bus 02])
[ 2.720698] pci 0001:02:01.0: devices behind bridge are unusable because [bus 03] cannot be assigned for them
[ 2.730954] pci_bus 0001:04: busn_res: can not insert [bus 04] under [bus 02] (conflicts with (null) [bus 02])
[ 2.757448] pci 0001:02:02.0: devices behind bridge are unusable because [bus 04] cannot be assigned for them
[ 2.767706] pci_bus 0001:05: busn_res: can not insert [bus 05] under [bus 02] (conflicts with (null) [bus 02])
[ 2.794197] pci 0001:02:03.0: devices behind bridge are unusable because [bus 05] cannot be assigned for them
[ 2.804269] pci 0001:01:00.0: bridge has subordinate 02 but max busn 05
[ 2.810954] pci 0001:02:01.0: PCI bridge to [bus 03]
[ 2.816021] pci 0001:02:02.0: PCI bridge to [bus 04]
[ 2.821086] pci 0001:02:03.0: PCI bridge to [bus 05]
[ 2.826151] pci 0001:01:00.0: PCI bridge to [bus 02]
[ 2.831217] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
...

# lspci -ntv
-+-[0001:00]---00.0-[01-ff]----00.0-[02]--+-01.0-[03]--
|                                                             +-02.0-[04]--
|                                                              \-03.0-[05]--
\-[0000:00]---00.0-[01-ff]--

# lspci -nv
0000:00:00.0 0604: 1957:8d80 (rev 20) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 381, IOMMU group 10
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [158] Secondary PCI Express
Kernel driver in use: pcieport

0001:00:00.0 0604: 1957:8d80 (rev 20) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 382, IOMMU group 11
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [158] Secondary PCI Express
Kernel driver in use: pcieport

0001:01:00.0 0604: 12d8:2404 (rev 05) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IOMMU group 11
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [5c] Vital Product Data
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: 0000:0000
Capabilities: [c0] Express Upstream Port, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [20c] Power Budgeting <?>
Capabilities: [230] Latency Tolerance Reporting

0001:02:01.0 0604: 12d8:2404 (rev 05) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 383, IOMMU group 11
Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: 0000:0000
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [20c] Power Budgeting <?>
Capabilities: [220] Access Control Services
Kernel driver in use: pcieport

0001:02:02.0 0604: 12d8:2404 (rev 05) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 384, IOMMU group 11
Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: 0000:0000
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [20c] Power Budgeting <?>
Capabilities: [220] Access Control Services
Kernel driver in use: pcieport

0001:02:03.0 0604: 12d8:2404 (rev 05) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 385, IOMMU group 11
Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: 0000:0000
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [20c] Power Budgeting <?>
Capabilities: [220] Access Control Services
Kernel driver in use: pcieport

I'll be very grateful if somebody can insert a PCIe bridge card in LX2160ARDB board, and check that
devices after the bridge are really enumerated and visible in kernel, or at least in u-boot.

Cheers,

[kikote]

Hello Zhao,

many thanks for the hint, I compared u-boot LSDK 21.08 against mainline U-Boot git tree,
and patched 4 files in U-Boot + one in kernel, see attached archive.

No changes were required in LX2160ARDB device tree or
default config file configs/lx2160ardb_tfa_defconfig

All PCIe devices were correctly detected in kernel, up to 14 devices.

Have a nice day,

[Zhao_Zheng · Accepted solution]

This issue had been fixed in latest code of u-boot upstream.

https://source.denx.de/u-boot/u-boot/-/commit/19e1b8d9b283514eb6ab8d7253f4867068007938