LS1021A PCIe driver crashes

[renaud]

We have a LS1021A based design. I am doing the board bring-up. We have our own boot laoder but also can run U-boot. When booting our Linux 5.17 kernel the system boots to the prompt but fails to get the correct vendor/device ID for a device connected to LANE 0. Though on the Ls1021A side everything seems good:

0.174370] layerscape-pcie 3400000.pcie: host bridge /soc/pcie@3400000 ranges:
[ 0.174444] layerscape-pcie 3400000.pcie: IO 0x4000010000..0x400001ffff -> 0x0000000000
[ 0.174483] layerscape-pcie 3400000.pcie: MEM 0x4040000000..0x407fffffff -> 0x0040000000
[ 0.174520] layerscape-pcie 3400000.pcie: non-prefetchable memory resource required
[ 0.174554] layerscape-pcie 3400000.pcie: iATU unroll: disabled
[ 0.174567] layerscape-pcie 3400000.pcie: Detected iATU regions: 6 outbound, 6 inbound
[ 0.174597] layerscape-pcie 3400000.pcie: Link up
[ 0.174722] layerscape-pcie 3400000.pcie: PCI host bridge to bus 0000:00
[ 0.174742] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.174761] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.174809] pci 0000:00:00.0: [1957:0e09] type 01 class 0x060400 ==> NXP OK
[ 0.174840] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[ 0.174861] pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
[ 0.174880] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[ 0.174941] pci 0000:00:00.0: supports D1 D2
[ 0.174954] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 0.176805] PCI: bus0: Fast back to back transfers disabled
[ 0.176942] pci 0000:01:00.0: [f018:e59f] type 1f class 0xe59ff0  ==> DEVICE NOT OK
[ 0.176975] pci 0000:01:00.0: unknown header type 1f, ignoring device
[ 0.178508] PCI: bus1: Fast back to back transfers enabled
[ 0.178547] pci 0000:00:00.0: BAR 1: no space for [mem size 0x04000000]
[ 0.178566] pci 0000:00:00.0: BAR 1: failed to assign [mem size 0x04000000]
[ 0.178582] pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000]
[ 0.178597] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
[ 0.178612] pci 0000:00:00.0: BAR 6: no space for [mem size 0x01000000 pref]
[ 0.178627] pci 0000:00:00.0: BAR 6: failed to assign [mem size 0x01000000 pref]
[ 0.178645] pci 0000:00:00.0: PCI bridge to [bus 01-ff]

When booting U-boot, I can see the correct ID:

=> pci heade 1.0.0
PCIe1: pcie@3400000 Root Complex: x1 gen1
vendor ID = 0x14e4
device ID = 0xb861

However, booting the NXP kernel 5.10:

[ 5.251637] layerscape-pcie 3400000.pcie: host bridge /soc/pcie@3400000 ranges:
[ 5.339741] layerscape-pcie 3400000.pcie: IO 0x4000010000..0x400001ffff -> 0x00000000
[ 5.437213] layerscape-pcie 3400000.pcie: MEM 0x4040000000..0x407fffffff -> 0x40000000
[ 5.534689] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[ 5.632140] pgd = (ptrval)
[ 5.664637] [00000000] *pgd=00000000
[ 5.707615] Internal error: Oops: 5 [#1] SMP ARM
[ 5.763147] Modules linked in:
[ 5.799823] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.68-00019-ga5c3ba83fc17-dirty #6
[ 5.899349] Hardware name: Freescale LS1021A
[ 5.950699] PC is at dw_pcie_host_init+0x20c/0x540
[ 6.008322] LR is at devm_pci_remap_iospace+0x58/0x78
[ 6.069083] pc : [<c04999c4>] lr : [<c047a12c>] psr: a0000053
[ 6.144514] sp : d708bde8 ip : 00000000 fp : d7fd710c
[ 6.207376] r10: d7216fb0 r9 : d710d010 r8 : d7216e00
[ 6.270238] r7 : d7216fb0 r6 : d7216fb0 r5 : 00000100 r4 : d7216c58
[ 6.348811] r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : 00000000
[ 6.427388] Flags: NzCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment none
[ 6.514341] Control: 10c5387d Table: 8400406a DAC: 00000051
[ 6.583488] Process swapper/0 (pid: 1, stack limit = 0x(ptrval))
[ 6.655777] Stack: (0xd708bde8 to 0xd708c000)
[ 6.708167] bde0: d7216c80 dc910000 d710d010 c0c8d5d8 c0cab170 c0cc0cf8
[ 6.806652] be00: 00010000 d7270040 d710d010 d710d000 d710d010 00000000 c0f4f564 00000000
[ 6.905137] be20: c0e0049c c0e31524 00000000 d710d010 c0f4f564 c0fe9bbc 00000000 c05410b0
[ 7.003621] be40: d710d010 c0fe9bb8 00000000 c053f38c 00000000 d710d010 c0f4f564 d710d044
[ 7.102107] be60: c0f68e78 c0fac380 c0ea1aec c0e87854 c0e0049c c053f648 c053f750 c0f68e78
[ 7.200591] be80: d710d010 c0f4f564 d710d044 c0f68e78 c0fac380 c053f864 00000000 c0f4f564
[ 7.299076] bea0: c053f750 c053d20c d70e9058 d7100834 c0f4f564 d7265f80 00000000 c053e5f0
[ 7.397561] bec0: c0cc0dfc d7265f00 00000000 c0f4f564 00000000 c0f4f564 00000000 c05402c0
[ 7.496046] bee0: c0f4f550 c0e31428 c0f4f564 c054119c c0fa17c0 c0e31428 ffffe000 00000000
[ 7.594531] bf00: c0fac380 c0102c78 000005eb 00000000 d7fffd00 d7fffda3 d7fffd9d c0142020
[ 7.693016] bf20: d708bf20 c0d190f8 0000012f 00000000 00000006 00000006 d7fffdae d7fffdb4
[ 7.791501] bf40: 00000000 0000012f c0fac380 c0fa17c0 c0e87830 0000012f 00000007 c0fa17c0
[ 7.889986] bf60: c0e87834 c0e01000 00000006 00000006 00000000 c0e0049c d708a000 00000000
[ 7.988470] bf80: 00000000 00000000 c0a465d0 00000000 00000000 00000000 00000000 00000000
[ 8.086955] bfa0: 00000000 c0a465d8 00000000 c01010e8 00000000 00000000 00000000 00000000
[ 8.185439] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 8.283925] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 8.382426] [<c04999c4>] (dw_pcie_host_init) from [<c0e31524>] (ls_pcie_probe+0xe4/0x10c)
[ 8.480913] [<c0e31524>] (ls_pcie_probe) from [<c05410b0>] (platform_drv_probe+0x48/0x98)
[ 8.579396] [<c05410b0>] (platform_drv_probe) from [<c053f38c>] (really_probe+0x2b0/0x3c0)
[ 8.678927] [<c053f38c>] (really_probe) from [<c053f648>] (driver_probe_device+0x60/0x168)
[ 8.778459] [<c053f648>] (driver_probe_device) from [<c053f864>] (__driver_attach+0x114/0x134)
[ 8.882183] [<c053f864>] (__driver_attach) from [<c053d20c>] (bus_for_each_dev+0x64/0x90)
[ 8.980668] [<c053d20c>] (bus_for_each_dev) from [<c053e5f0>] (bus_add_driver+0x184/0x208)
[ 9.080198] [<c053e5f0>] (bus_add_driver) from [<c05402c0>] (driver_register+0x88/0x118)
[ 9.177637] [<c05402c0>] (driver_register) from [<c054119c>] (__platform_driver_probe+0x64/0x108)
[ 9.284505] [<c054119c>] (__platform_driver_probe) from [<c0102c78>] (do_one_initcall+0x44/0x1e4)
[ 9.391373] [<c0102c78>] (do_one_initcall) from [<c0e01000>] (kernel_init_freeable+0x26c/0x310)
[ 9.496150] [<c0e01000>] (kernel_init_freeable) from [<c0a465d8>] (kernel_init+0x8/0x10c)
[ 9.594637] [<c0a465d8>] (kernel_init) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[ 9.685781] Exception stack(0xd708bfb0 to 0xd708bff8)
[ 9.746549] bfa0: 00000000 00000000 00000000 00000000
[ 9.845030] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 9.943514] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 10.023139] Code: e5143014 e3530000 0a00009f e5943050 (e5933000)
[ 10.096536] ---[ end trace 8a908bbfd8a662b3 ]---
[ 10.152112] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 10.152112]

So not sure if I have a hardware problem or software problem.

If you could point to things I may have not looked at or any suggestion, that could help.

Cheers,

Renaud

[renaud]

Missing LPAE config in Linux if I remember correctly

[yipingwang]

For the error case, are you using linux kernel from LSDK? Seems this linux kernel is not released by NXP:

[ 5.251637] layerscape-pcie 3400000.pcie: host bridge /soc/pcie@3400000 ranges:
[ 5.339741] layerscape-pcie 3400000.pcie: IO 0x4000010000..0x400001ffff -> 0x00000000
[ 5.437213] layerscape-pcie 3400000.pcie: MEM 0x4040000000..0x407fffffff -> 0x40000000
[ 5.534689] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[ 5.632140] pgd = (ptrval)
[ 5.664637] [00000000] *pgd=00000000
[ 5.707615] Internal error: Oops: 5 [#1] SMP ARM

In your log, seems non-prefetchable memory resource not configured. In NXP released linux, it should print 36-bits address, this code is from our linux kernel:
dev_dbg(dev, "Parsing ranges property...\n");
for_each_of_pci_range(&parser, &range) {
/* Read next ranges element */
if ((range.flags & IORESOURCE_TYPE_BITS) == IORESOURCE_IO)
range_type = "IO";
else if ((range.flags & IORESOURCE_TYPE_BITS) == IORESOURCE_MEM)
range_type = "MEM";
else
range_type = "err";
dev_info(dev, " %6s %#012llx..%#012llx -> %#012llx\n",
range_type, range.cpu_addr,
range.cpu_addr + range.size - 1, range.pci_addr);