IMX8MP hang during PCI init

[timharvey]

Greetings,

I'm working with a custom IMX8M Plus board that uses the IMX8M Plus SoC with a Diodes Incorporated PI7C9X2G608G Gen2 switch. We use an external PCIe clock to the imx8mp and switch provided by a  AB55703HCHCF dual output 100Mhz LPHSCL clock generator and have the device-tree configured for the imx8mp to use a clock input. When leaving fsl,max-link-speed=3 (default for imx8mp.dtsi) I find that 60% of boots result in a hang during pci init and 30% of the boots successfully link at gen2. If I set fsl,max-link-speed=2 I see the same behavior but if I set fsl,max-link-speed=1 it links at Gen1 without hang every time but of course this limits PCIe performance quite a bit.

I have re-created this issue on every kernel version I have tried including 6.1, 6.5-rc6, and NXP's lf-6.1.22-2.0.0.

A successful link at gen2 using Linux 6.5-rc6 looks like this:
...
[ 0.324855] Asymmetric key parser 'x509' registered
[ 0.335035] clk: failed to reparent gic to sys_pll2_500m: -16
[ 0.342998] SoC: i.MX8MP revision 1.1
[ 0.347029] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.352678] 30890000.serial: ttymxc1 at MMIO 0x30890000 (irq = 15, base_baud = 1500000) is a IMX
[ 0.359364] printk: console [ttymxc1] enabled
[ 0.359364] printk: console [ttymxc1] enabled
[ 0.368034] printk: bootconsole [ec_imx6q0] disabled
[ 0.368034] printk: bootconsole [ec_imx6q0] disabled
[ 0.380670] i2c_dev: i2c /dev entries driver
[ 0.386079] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.392117] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[ 0.399731] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[ 0.412372] Loading compiled-in X.509 certificates
[ 0.431454] gpio gpiochip0: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.441555] gpio gpiochip1: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.451671] gpio gpiochip2: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.461744] gpio gpiochip3: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.471859] gpio gpiochip4: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.495753] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
[ 0.496382] clk: Not disabling unused clocks
[ 0.503042] imx6q-pcie 33800000.pcie: IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
[ 0.515505] imx6q-pcie 33800000.pcie: MEM 0x0018000000..0x001fefffff -> 0x0018000000
[ 0.739498] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
[ 0.847519] imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
[ 0.953163] imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
[ 0.958773] imx6q-pcie 33800000.pcie: Link up, Gen2
[ 0.963663] imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
[ 0.969553] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
[ 0.975928] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.981426] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.987618] pci_bus 0000:00: root bus resource [mem 0x18000000-0x1fefffff]
[ 0.994530] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[ 1.000563] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[ 1.006847] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[ 1.013600] pci 0000:00:00.0: supports D1
[ 1.017620] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[ 1.025843] pci 0000:01:00.0: [12d8:2608] type 01 class 0x060400
[ 1.032244] pci 0000:01:00.0: supports D1 D2
[ 1.036529] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.043510] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.051961] pci 0000:02:01.0: [12d8:2608] type 01 class 0x060400
[ 1.058401] pci 0000:02:01.0: supports D1 D2
[ 1.062695] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.069746] pci 0000:02:02.0: [12d8:2608] type 01 class 0x060400
[ 1.076151] pci 0000:02:02.0: supports D1 D2
[ 1.080435] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.087447] pci 0000:02:03.0: [12d8:2608] type 01 class 0x060400
[ 1.093834] pci 0000:02:03.0: supports D1 D2
[ 1.098111] pci 0000:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.105110] pci 0000:02:04.0: [12d8:2608] type 01 class 0x060400
[ 1.111504] pci 0000:02:04.0: supports D1 D2
[ 1.115781] pci 0000:02:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.124178] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.132218] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.140251] pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.148282] pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.156651] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 41
[ 1.163569] pci_bus 0000:42: busn_res: [bus 42-ff] end is updated to 80
[ 1.170472] pci_bus 0000:81: busn_res: [bus 81-ff] end is updated to bf
[ 1.177356] pci_bus 0000:c0: busn_res: [bus c0-ff] end is updated to fe
[ 1.183999] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to fe
[ 1.190666] pci 0000:00:00.0: BAR 0: assigned [mem 0x18000000-0x180fffff]
[ 1.197470] pci 0000:00:00.0: BAR 8: assigned [mem 0x18100000-0x188fffff]
[ 1.204277] pci 0000:00:00.0: BAR 9: assigned [mem 0x18900000-0x190fffff pref]
[ 1.211513] pci 0000:00:00.0: BAR 6: assigned [mem 0x19100000-0x1910ffff pref]
[ 1.218748] pci 0000:00:00.0: BAR 7: assigned [io 0x1000-0x4fff]
[ 1.224857] pci 0000:01:00.0: BAR 8: assigned [mem 0x18100000-0x188fffff]
[ 1.231657] pci 0000:01:00.0: BAR 9: assigned [mem 0x18900000-0x190fffff 64bit pref]
[ 1.239410] pci 0000:01:00.0: BAR 7: assigned [io 0x1000-0x4fff]
[ 1.245524] pci 0000:02:01.0: BAR 8: assigned [mem 0x18100000-0x182fffff]
[ 1.252322] pci 0000:02:01.0: BAR 9: assigned [mem 0x18900000-0x18afffff 64bit pref]
[ 1.260077] pci 0000:02:02.0: BAR 8: assigned [mem 0x18300000-0x184fffff]
[ 1.266874] pci 0000:02:02.0: BAR 9: assigned [mem 0x18b00000-0x18cfffff 64bit pref]
[ 1.274632] pci 0000:02:03.0: BAR 8: assigned [mem 0x18500000-0x186fffff]
[ 1.281444] pci 0000:02:03.0: BAR 9: assigned [mem 0x18d00000-0x18efffff 64bit pref]
[ 1.289205] pci 0000:02:04.0: BAR 8: assigned [mem 0x18700000-0x188fffff]
[ 1.296003] pci 0000:02:04.0: BAR 9: assigned [mem 0x18f00000-0x190fffff 64bit pref]
[ 1.303757] pci 0000:02:01.0: BAR 7: assigned [io 0x1000-0x1fff]
[ 1.309859] pci 0000:02:02.0: BAR 7: assigned [io 0x2000-0x2fff]
[ 1.315962] pci 0000:02:03.0: BAR 7: assigned [io 0x3000-0x3fff]
[ 1.322070] pci 0000:02:04.0: BAR 7: assigned [io 0x4000-0x4fff]
[ 1.328183] pci 0000:02:01.0: PCI bridge to [bus 03-41]
[ 1.333426] pci 0000:02:01.0: bridge window [io 0x1000-0x1fff]
[ 1.339543] pci 0000:02:01.0: bridge window [mem 0x18100000-0x182fffff]
[ 1.346347] pci 0000:02:01.0: bridge window [mem 0x18900000-0x18afffff 64bit pref]
[ 1.354114] pci 0000:02:02.0: PCI bridge to [bus 42-80]
[ 1.359352] pci 0000:02:02.0: bridge window [io 0x2000-0x2fff]
[ 1.365464] pci 0000:02:02.0: bridge window [mem 0x18300000-0x184fffff]
[ 1.372267] pci 0000:02:02.0: bridge window [mem 0x18b00000-0x18cfffff 64bit pref]
[ 1.380039] pci 0000:02:03.0: PCI bridge to [bus 81-bf]
[ 1.385282] pci 0000:02:03.0: bridge window [io 0x3000-0x3fff]
[ 1.391407] pci 0000:02:03.0: bridge window [mem 0x18500000-0x186fffff]
[ 1.398215] pci 0000:02:03.0: bridge window [mem 0x18d00000-0x18efffff 64bit pref]
[ 1.405988] pci 0000:02:04.0: PCI bridge to [bus c0-fe]
[ 1.411227] pci 0000:02:04.0: bridge window [io 0x4000-0x4fff]
[ 1.417341] pci 0000:02:04.0: bridge window [mem 0x18700000-0x188fffff]
[ 1.424144] pci 0000:02:04.0: bridge window [mem 0x18f00000-0x190fffff 64bit pref]
...

A hang looks like this:
...
[ 0.319551] Asymmetric key parser 'x509' registered
[ 0.329739] clk: failed to reparent gic to sys_pll2_500m: -16
[ 0.337699] SoC: i.MX8MP revision 1.1
[ 0.341761] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.347374] 30890000.serial: ttymxc1 at MMIO 0x30890000 (irq = 15, base_baud = 1500000) is a IMX
[ 0.354071] printk: console [ttymxc1] enabled
[ 0.354071] printk: console [ttymxc1] enabled
[ 0.362759] printk: bootconsole [ec_imx6q0] disabled
[ 0.362759] printk: bootconsole [ec_imx6q0] disabled
[ 0.384699] i2c_dev: i2c /dev entries driver
[ 0.390010] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.396043] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[ 0.403648] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[ 0.416370] Loading compiled-in X.509 certificates
[ 0.435397] gpio gpiochip0: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.445488] gpio gpiochip1: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.455627] gpio gpiochip2: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.465687] gpio gpiochip3: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.475771] gpio gpiochip4: Static allocation of GPIO base is deprecated, use dynamic allocation.
[ 0.499660] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
[ 0.500276] clk: Not disabling unused clocks
[ 0.506960] imx6q-pcie 33800000.pcie: IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
[ 0.519401] imx6q-pcie 33800000.pcie: MEM 0x0018000000..0x001fefffff -> 0x0018000000
[ 0.743554] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
[ 0.851578] imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up

^^^ hang at this point until watchdog resets

Note the 'clk: failed to reparent gic to sys_pll2_500m: -16' message which I do not get on the same board with an imx8mm SoC. I'm not sure if that is pointing to an issue at this point or not but I see it on the imx8mp even when PCI init behaves normally. Also note above I have kept the kernel from disabling unused clocks as a precaution.

I have verified with a scope that PERST# is low until the imx6 PCIe driver drives it high and that the external PCI clock going to the imx8mp as well as the switch.

I can not find a section in the IMX8MPRM that details the steps required for PCI init (there was a thorough list in the IMX6DQRM) to verify that the driver is performing the steps properly.

Does anyone have any ideas what can be going on here?

Best regards,

Tim

[timharvey]

Hi Zhiming,

We did find that the problem is related to the PI7C9X2G608G PCIe Gen2 switch which has an errata related to a rate change from 2.5Gbps to 5Gbps which can cause a link-down and register reset. We believe that this then causes the Synopsis core to crash which is what causes the CPU hang.

If I modify the Linux driver to link at max link first we do not encounter this.

I'm still trying to determine if the PCIe spec specifies if you should be linking at the max rate first vs linking at gen1 and working up (like the IMX6+ PCI driver is currently doing).

Best regards,

Tim

[Zhiming_Liu]

Hi @timharvey 

From the driver's point of view, the pci-imx6 driver will call dw_pcie_wait_for_link(drivers/pci/controller/dwc/pcie-designware.c) to wait link up. These pci driver functions are very stable. So I think it might still be a hardware thing. Maybe the timing is out of sync causing the pci core to go down when this issue happen. You need veify the stability and synchronization of clocks when this issue occurs. There must be something unstable from hardware and then casue core reset.

Best Regards

Zhiming