64 bit PCI BAR address

williamanderson — Tue, 17 Jan 2017 18:39:09 GMT

I am attempting to connect to a PCI edge device that uses large addresse ( > 32bit).

I am using the LS10121A with 4.1.8 kernel

I get the following errors at boot.

Does anyone know if there is a kernel configuration option that will allow the LS1021 to accept large BAR values.

Any pointer appreciated.

Thanks

Bill A

[ 0.165002] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.165012] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.165025] pci_bus 0000:00: root bus resource [mem 0x4040000000-0x407fffffff] (bus address [0x40000000-0x7fffffff])
[ 0.165438] PCI: bus0: Fast back to back transfers disabled
[ 0.165869] pci 0000:01:00.0: reg 0x20: can't handle BAR above 4GB (bus address 0x1000c0000000)    <-----------
[ 0.166336] PCI: bus1: Fast back to back transfers disabled
[ 0.166417] pci 0000:00:00.0: BAR 9: no space for [mem size 0x60000000 pref]
[ 0.166428] pci 0000:00:00.0: BAR 9: failed to assign [mem size 0x60000000 pref]
[ 0.166441] pci 0000:00:00.0: BAR 1: assigned [mem 0x4040000000-0x4043ffffff]
[ 0.166455] pci 0000:00:00.0: BAR 0: assigned [mem 0x4044000000-0x4044ffffff]
[ 0.166468] pci 0000:00:00.0: BAR 6: assigned [mem 0x4045000000-0x4045ffffff pref]
[ 0.166482] pci 0000:01:00.0: BAR 4: no space for [mem size 0x40000000 64bit pref]                              <-----------
[ 0.166493] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x40000000 64bit pref]                          <-----------
[ 0.166504] pci 0000:01:00.0: BAR 2: no space for [mem size 0x04000000 64bit pref]                              <-----------
[ 0.166514] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit pref]                           <-----------
[ 0.166525] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]                              <-----------
[ 0.166535] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]                           <-----------
[ 0.166545] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 0.166851] PCI host bridge /soc/pcie@3500000 ranges:
[ 0.166871] IO 0x4800010000..0x480001ffff -> 0x00000000
[ 0.166884] MEM 0x4840000000..0x487fffffff -> 0x40000000
[ 0.167074] layerscape-pcie 3500000.pcie: PCI host bridge to bus 0001:00
[ 0.167087] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 0.167099] pci_bus 0001:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
[ 0.167112] pci_bus 0001:00: root bus resource [mem 0x4840000000-0x487fffffff] (bus address [0x40000000-0x7fffffff])

LSPCI reports

root@ls1021aiot:~# lspci -v
0000:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0e0a (rev 20) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 54
Memory at 4044000000 (32-bit, non-prefetchable) [size=16M]
Memory at 4040000000 (32-bit, non-prefetchable) [size=64M]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Expansion ROM at 4045000000 [disabled] [size=16M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error ReportingThanks.
Capabilities: [148] #19

0000:01:00.0 Ethernet controller: Marvell Technology Group Ltd. Device be00
Subsystem: Marvell Technology Group Ltd. Device 11ab
Flags: bus master, fast devsel, latency 0, IRQ 54
Memory at <ignored> (64-bit, prefetchable)
Memory at <ignored> (64-bit, prefetchable)
Memory at <ignored> (64-bit, prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [60] Express Legacy Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting

0001:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0e0a (rev 20) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 55
Memory at 4844000000 (32-bit, non-prefetchable) [size=16M]
Memory at 4840000000 (32-bit, non-prefetchable) [size=64M]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
[virtual] Expansion ROM at 4845000000 [disabled] [size=16M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19

root@ls1021aiot:~# lspci -x
0000:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0e0a (rev 20)
00: 57 19 0a 0e 47 01 10 00 20 00 04 06 10 00 01 00
10: 00 00 00 44 00 00 00 40 00 01 01 00 f0 00 00 00
20: f0 ff 00 00 f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 08 36 01 01 00

0000:01:00.0 Ethernet controller: Marvell Technology Group Ltd. Device be00
00: ab 11 00 be 47 01 10 00 00 00 00 02 10 00 00 00
10: 0c 00 00 09 00 00 00 00 0c 00 00 0c 00 00 00 00
20: 0c 00 00 c0 00 10 00 00 00 00 00 00 ab 11 ab 11 <----------- Offending BAR
30: 00 00 00 00 40 00 00 00 00 00 00 00 36 01 00 00

0001:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0e0a (rev 20)
00: 57 19 0a 0e 47 01 10 00 20 00 04 06 10 00 01 00
10: 00 00 00 44 00 00 00 40 00 01 01 00 f0 00 00 00
20: f0 ff 00 00 f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 37 01 01 00

Re: 64 bit PCI BAR address

williamanderson — Tue, 24 Jan 2017 21:25:29 GMT

Answer

I have confirmed with NXP that LS1021a only support PCI End point BARs of less than or equal to 512 MB . If the requested size goes above 512 MB the PCI address assigned is above 4GB and cant be accessed by the PCI controller.

I confirmed that a 512 MB size BAR does work.

topic Re: 64 bit PCI BAR address in Layerscape

64 bit PCI BAR address

Re: 64 bit PCI BAR address