[ERROR] PCI memory allocation (BAR Registers) on T1042D4RDB-64B

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

[ERROR] PCI memory allocation (BAR Registers) on T1042D4RDB-64B

Jump to solution
19,896 Views
yusufalti333
Contributor IV
Hello,
My problem is briefly, PCI is not allocating memory on t1042d4-64b demo board.
Below is the 'dmesg' messages and U-boot messages.
I will be grateful for your helps.
'dmesg' messages from Linux
[    0.464934] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.473580] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.477426] pci 0000:00:00.0:   bridge window [io  0x8000080000010000-0x8000080000010fff]
[    0.477431] pci 0000:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.477438] pci 0000:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.477451] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.477456] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01
[    0.477584] fsl-pci ffe250000.pcie: PCI host bridge to bus 0001:00
[    0.482383] pci_bus 0001:00: root bus resource [io  0x8000080000021000-0x8000080000030fff] (bus address [0x0000-0xffff])
[    0.491940] pci_bus 0001:00: root bus resource [mem 0xc10000000-0xc1fffffff] (bus address [0xe0000000-0xefffffff])
[    0.500980] pci_bus 0001:00: root bus resource [bus 00-01]
[    0.505153] pci_bus 0001:00: busn_res: [bus 00-01] end is updated to ff
[    0.505172] pci 0001:00:00.0: [1957:0824] type 01 class 0x0b2000
[    0.505191] pci 0001:00:00.0: reg 0x10: [mem 0xff000000-0xffffffff]
[    0.505236] pci 0001:00:00.0: supports D1 D2
[    0.505239] pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.505346] iommu: Removing device ffe250000.pcie from group 20
[    0.509901] iommu: Adding device 0001:00:00.0 to group 19
[    0.516106] pci 0001:01:00.0: [1002:6987] type 00 class 0x030000
[    0.516160] pci 0001:01:00.0: reg 0x10: [mem 0xc10000000-0xc1fffffff 64bit pref]
[    0.516175] pci 0001:01:00.0: reg 0x18: [mem 0x1000ffe00000-0x1000ffffffff 64bit pref]
[    0.516186] pci 0001:01:00.0: reg 0x20: [io  0x8000080000022100-0x80000800000221ff]
[    0.516196] pci 0001:01:00.0: reg 0x24: [mem 0xfffc0000-0xffffffff]
[    0.516207] pci 0001:01:00.0: reg 0x30: [mem 0xfffe0000-0xffffffff pref]
[    0.516219] pci 0001:01:00.0: enabling Extended Tags
[    0.519920] pci 0001:01:00.0: supports D1 D2
[    0.519923] pci 0001:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.519958] pci 0001:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0001:00:00.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
[    0.532404] iommu: Adding device 0001:01:00.0 to group 19
[    0.536460] pci 0001:01:00.1: [1002:aae0] type 00 class 0x040300
[    0.536500] pci 0001:01:00.1: reg 0x10: [mem 0x1200ffffc000-0x1200ffffffff 64bit]
[    0.536553] pci 0001:01:00.1: enabling Extended Tags
[    0.540200] pci 0001:01:00.1: supports D1 D2
[    0.540291] iommu: Adding device 0001:01:00.1 to group 19
[    0.546301] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    0.550147] pci 0001:00:00.0:   bridge window [io  0x8000080000022000-0x8000080000022fff]
[    0.550152] pci 0001:00:00.0:   bridge window [mem 0xc10000000-0xc1fffffff]
[    0.550235] pci_bus 0001:01: busn_res: [bus 01-ff] end is updated to 01
[    0.550242] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to 01
[    0.550376] fsl-pci ffe260000.pcie: PCI host bridge to bus 0002:00
[    0.555177] pci_bus 0002:00: root bus resource [io  0x8000080000032000-0x8000080000041fff] (bus address [0x0000-0xffff])
[    0.564732] pci_bus 0002:00: root bus resource [mem 0xc20000000-0xc2fffffff] (bus address [0xe0000000-0xefffffff])
[    0.573773] pci_bus 0002:00: root bus resource [bus 00]
[    0.577680] pci_bus 0002:00: busn_res: [bus 00] end is updated to ff
[    0.577698] pci 0002:00:00.0: [1957:0824] type 01 class 0x0b2000
[    0.577718] pci 0002:00:00.0: reg 0x10: [mem 0xff000000-0xffffffff]
[    0.577764] pci 0002:00:00.0: supports D1 D2
[    0.577767] pci 0002:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.577880] iommu: Removing device ffe260000.pcie from group 21
[    0.582434] iommu: Adding device 0002:00:00.0 to group 20
[    0.588545] pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.597194] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    0.601040] pci 0002:00:00.0:   bridge window [io  0x8000080000032000-0x8000080000032fff]
[    0.601045] pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.601052] pci 0002:00:00.0:   bridge window
[    0.601064] pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
[    0.601070] pci_bus 0002:00: busn_res: [bus 00-ff] end is updated to 01
[    0.601197] fsl-pci ffe270000.pcie: PCI host bridge to bus 0003:00
[    0.605995] pci_bus 0003:00: root bus resource [io  0x8000080000043000-0x8000080000052fff] (bus address [0x0000-0xffff])
[    0.615553] pci_bus 0003:00: root bus resource [mem 0xc30000000-0xc3fffffff] (bus address [0xe0000000-0xefffffff])
[    0.624593] pci_bus 0003:00: root bus resource [bus 00]
[    0.601064] pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
[    0.601070] pci_bus 0002:00: busn_res: [bus 00-ff] end is updated to 01
[    0.601197] fsl-pci ffe270000.pcie: PCI host bridge to bus 0003:00
[    0.605995] pci_bus 0003:00: root bus resource [io  0x8000080000043000-0x8000080000052fff] (bus address [0x0000-0xffff])
[    0.615553] pci_bus 0003:00: root bus resource [mem 0xc30000000-0xc3fffffff] (bus address [0xe0000000-0xefffffff])
[    0.624593] pci_bus 0003:00: root bus resource [bus 00]
[    0.628501] pci_bus 0003:00: busn_res: [bus 00] end is updated to ff
[    0.628519] pci 0003:00:00.0: [1957:0824] type 01 class 0x0b2000
[    0.628538] pci 0003:00:00.0: reg 0x10: [mem 0xff000000-0xffffffff]
[    0.628584] pci 0003:00:00.0: supports D1 D2
[    0.628588] pci 0003:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.628703] iommu: Removing device ffe270000.pcie from group 22
[    0.633257] iommu: Adding device 0003:00:00.0 to group 21
[    0.639362] pci 0003:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.651912] pci 0003:00:00.0:   bridge window [io  0x8000080000043000-0x8000080000043fff]
[    0.651917] pci 0003:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.651923] pci 0003:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.628501] pci_bus 0003:00: busn_res: [bus 00] end is updated to ff
[    0.628519] pci 0003:00:00.0: [1957:0824] type 01 class 0x0b2000
[    0.628538] pci 0003:00:00.0: reg 0x10: [mem 0xff000000-0xffffffff]
[    0.628584] pci 0003:00:00.0: supports D1 D2
[    0.628588] pci 0003:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.628703] iommu: Removing device ffe270000.pcie from group 22
[    0.633257] iommu: Adding device 0003:00:00.0 to group 21
[    0.639362] pci 0003:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.651912] pci 0003:00:00.0:   bridge window [io  0x8000080000043000-0x8000080000043fff]
[    0.651917] pci 0003:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.651923] pci 0003:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.651936] pci_bus 0003:01: busn_res: [bus 01-ff] end is updated to 01
[    0.651942] pci_bus 0003:00: busn_res: [bus 00-ff] end is updated to 01
[    0.651988] PCI: Cannot allocate resource region 0 of device 0000:00:00.0, will remap
[    0.658435] PCI: Cannot allocate resource region 0 of device 0001:00:00.0, will remap
[    0.664953] PCI: Cannot allocate resource region 2 of device 0001:01:00.0, will remap
[    0.671469] PCI: Cannot allocate resource region 5 of device 0001:01:00.0, will remap
[    0.677989] PCI: Cannot allocate resource region 6 of device 0001:01:00.0, will remap
[    0.684514] PCI: Cannot allocate resource region 0 of device 0001:01:00.1, will remap
[    0.691033] PCI: Cannot allocate resource region 0 of device 0002:00:00.0, will remap
[    0.697552] PCI: Cannot allocate resource region 0 of device 0003:00:00.0, will remap
[    0.704083] PCI 0000:00 Cannot reserve Legacy IO [io  0x8000080000010000-0x10fff]
[    0.704086] PCI 0001:00 Cannot reserve Legacy IO [io  0x8000080000021000-0x21fff]
[    0.704089] PCI 0002:00 Cannot reserve Legacy IO [io  0x8000080000032000-0x32fff]
[    0.704092] PCI 0003:00 Cannot reserve Legacy IO [io  0x8000080000043000-0x43fff]
[    0.704107] pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000]
[    0.709375] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
[    0.715026] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.718673] pci 0000:00:00.0:   bridge window [io  0x8000080000010000-0x800008000001ffff]
[    0.725545] pci 0000:00:00.0:   bridge window [mem 0xc00000000-0xc0fffffff]
[    0.731196] pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.738932] pci_bus 0000:00: resource 4 [io  0x8000080000010000-0x800008000001ffff]
[    0.738935] pci_bus 0000:00: resource 5 [mem 0xc00000000-0xc0fffffff]
[    0.738939] pci_bus 0000:01: resource 0 [io  0x8000080000010000-0x800008000001ffff]
[    0.738942] pci_bus 0000:01: resource 1 [mem 0xc00000000-0xc0fffffff]
[    0.738953] pci 0001:00:00.0: BAR 0: no space for [mem size 0x01000000]
[    0.744235] pci 0001:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
[    0.749887] pci 0001:00:00.0: BAR 9: no space for [mem size 0x00200000 64bit pref]
[    0.756144] pci 0001:00:00.0: BAR 9: failed to assign [mem size 0x00200000 64bit pref]
[    0.762754] pci 0001:01:00.0: BAR 2: no space for [mem size 0x00200000 64bit pref]
[    0.769010] pci 0001:01:00.0: BAR 2: failed to assign [mem size 0x00200000 64bit pref]
[    0.775617] pci 0001:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.780920] pci 0001:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.786571] pci 0001:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.792308] pci 0001:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.798394] pci 0001:01:00.1: BAR 0: no space for [mem size 0x00004000 64bit]
[    0.804218] pci 0001:01:00.1: BAR 0: failed to assign [mem size 0x00004000 64bit]
[    0.810391] pci 0001:00:00.0: PCI bridge to [bus 01]
[    0.814038] pci 0001:00:00.0:   bridge window [io  0x8000080000021000-0x8000080000030fff]
[    0.820910] pci 0001:00:00.0:   bridge window [mem 0xc10000000-0xc1fffffff]
[    0.826561] pci_bus 0001:00: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.834298] pci_bus 0001:00: resource 4 [io  0x8000080000021000-0x8000080000030fff]
[    0.834301] pci_bus 0001:00: resource 5 [mem 0xc10000000-0xc1fffffff]
[    0.834305] pci_bus 0001:01: resource 0 [io  0x8000080000021000-0x8000080000030fff]
[    0.834308] pci_bus 0001:01: resource 1 [mem 0xc10000000-0xc1fffffff]
[    0.834318] pci 0002:00:00.0: BAR 0: no space for [mem size 0x01000000]
[    0.839600] pci 0002:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
[    0.845251] pci 0002:00:00.0: PCI bridge to [bus 01]
[    0.848898] pci 0002:00:00.0:   bridge window [io  0x8000080000032000-0x8000080000041fff]
[    0.855780] pci 0002:00:00.0:   bridge window [mem 0xc20000000-0xc2fffffff]
[    0.861422] pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.869158] pci_bus 0002:00: resource 4 [io  0x8000080000032000-0x8000080000041fff]
[    0.869161] pci_bus 0002:00: resource 5 [mem 0xc20000000-0xc2fffffff]
[    0.869165] pci_bus 0002:01: resource 0 [io  0x8000080000032000-0x8000080000041fff]
[    0.869168] pci_bus 0002:01: resource 1 [mem 0xc20000000-0xc2fffffff]
[    0.869178] pci 0003:00:00.0: BAR 0: no space for [mem size 0x01000000]
[    0.874468] pci 0003:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
[    0.880112] pci 0003:00:00.0: PCI bridge to [bus 01]
[    0.883759] pci 0003:00:00.0:   bridge window [io  0x8000080000043000-0x8000080000052fff]
[    0.890632] pci 0003:00:00.0:   bridge window [mem 0xc30000000-0xc3fffffff]
[    0.896284] pci_bus 0003:00: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.904018] pci_bus 0003:00: resource 4 [io  0x8000080000043000-0x8000080000052fff]
[    0.904022] pci_bus 0003:00: resource 5 [mem 0xc30000000-0xc3fffffff]
[    0.904025] pci_bus 0003:01: resource 0 [io  0x8000080000043000-0x8000080000052fff]
[    0.904028] pci_bus 0003:01: resource 1 [mem 0xc30000000-0xc3fffffff]
[    0.904030] PCI: Probing PCI hardware done
PCI console messages from U-BOOT
PCI: Probing PCI hardware
fsl-pci ffe240000.pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io  0x8000080000010000-0x800008000001ffff] (bus address [0x0000-0xffff])
pci_bus 0000:00: root bus resource [mem 0xc00000000-0xc0fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0000:00: root bus resource [bus 00]
iommu: Removing device ffe240000.pcie from group 19
iommu: Adding device 0000:00:00.0 to group 24
pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:00:00.0: PCI bridge to [bus 01-ff]
fsl-pci ffe250000.pcie: PCI host bridge to bus 0001:00
pci_bus 0001:00: root bus resource [io  0x8000080000021000-0x8000080000030fff] (bus address [0x0000-0xffff])
pci_bus 0001:00: root bus resource [mem 0xc10000000-0xc1fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0001:00: root bus resource [bus 00-01]
iommu: Removing device ffe250000.pcie from group 20
iommu: Adding device 0001:00:00.0 to group 19
pci 0001:01:00.0: enabling Extended Tags
pci 0001:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0001:00:00.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
iommu: Adding device 0001:01:00.0 to group 19
pci 0001:01:00.1: enabling Extended Tags
iommu: Adding device 0001:01:00.1 to group 19
pci 0001:00:00.0: PCI bridge to [bus 01-ff]
fsl-pci ffe260000.pcie: PCI host bridge to bus 0002:00
pci_bus 0002:00: root bus resource [io  0x8000080000032000-0x8000080000041fff] (bus address [0x0000-0xffff])
pci_bus 0002:00: root bus resource [mem 0xc20000000-0xc2fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0002:00: root bus resource [bus 00]
iommu: Removing device ffe260000.pcie from group 21
iommu: Adding device 0002:00:00.0 to group 20
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
fsl-pci ffe270000.pcie: PCI host bridge to bus 0003:00
pci_bus 0003:00: root bus resource [io  0x8000080000043000-0x8000080000052fff] (bus address [0x0000-0xffff])
pci_bus 0003:00: root bus resource [mem 0xc30000000-0xc3fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0003:00: root bus resource [bus 00]
iommu: Removing device ffe270000.pcie from group 22
iommu: Adding device 0003:00:00.0 to group 21
pci 0003:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0003:00:00.0: PCI bridge to [bus 01-ff]
PCI: Cannot allocate resource region 0 of device 0000:00:00.0, will remap
PCI: Cannot allocate resource region 0 of device 0001:00:00.0, will remap
PCI: Cannot allocate resource region 2 of device 0001:01:00.0, will remap
PCI: Cannot allocate resource region 5 of device 0001:01:00.0, will remap
PCI: Cannot allocate resource region 6 of device 0001:01:00.0, will remap
PCI: Cannot allocate resource region 0 of device 0001:01:00.1, will remap
PCI: Cannot allocate resource region 0 of device 0002:00:00.0, will remap
PCI: Cannot allocate resource region 0 of device 0003:00:00.0, will remap
pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0:   bridge window [io  0x8000080000010000-0x800008000001ffff]
pci 0000:00:00.0:   bridge window [mem 0xc00000000-0xc0fffffff]
pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci 0001:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0001:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0001:00:00.0: BAR 9: no space for [mem size 0x00200000 64bit pref]
pci 0001:00:00.0: BAR 9: failed to assign [mem size 0x00200000 64bit pref]
pci 0001:01:00.0: BAR 2: no space for [mem size 0x00200000 64bit pref]
pci 0001:01:00.0: BAR 2: failed to assign [mem size 0x00200000 64bit pref]
pci 0001:01:00.0: BAR 5: no space for [mem size 0x00040000]
pci 0001:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
pci 0001:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
pci 0001:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
pci 0001:01:00.1: BAR 0: no space for [mem size 0x00004000 64bit]
pci 0001:01:00.1: BAR 0: failed to assign [mem size 0x00004000 64bit]
pci 0001:00:00.0: PCI bridge to [bus 01]
pci 0001:00:00.0:   bridge window [io  0x8000080000021000-0x8000080000030fff]
pci 0001:00:00.0:   bridge window [mem 0xc10000000-0xc1fffffff]
pci_bus 0001:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci 0002:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0002:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0:   bridge window [io  0x8000080000032000-0x8000080000041fff]
pci 0002:00:00.0:   bridge window [mem 0xc20000000-0xc2fffffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci 0003:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0003:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0003:00:00.0: PCI bridge to [bus 01]
pci 0003:00:00.0:   bridge window [io  0x8000080000043000-0x8000080000052fff]
pci 0003:00:00.0:   bridge window [mem 0xc30000000-0xc3fffffff]
pci_bus 0003:00: Some PCI device resources are unassigned, try booting with pci=realloc
0 Kudos
1 Solution
17,569 Views
yusufalti333
Contributor IV

Hello yipingwang‌,

I solved the problem by using a gap area between PCIe2 and PCIe3. My drivers seems need 2 MB more space after the end of 256 MB offset.

But there are still points I do not understand. Even if I arrange 4 GB or more for PCIe2, kernel is only taking 256 MB + 2MB and not more and the reason behind it seems BUS_ADRESS which is 0xE0000000. I tried to change this to 0x0 but system crashed with machine check. Is it impossible to change this value? Why it's acting like 32 bit linux kernel ?

from dmesg:

Device tree node: /sys/firmware/devicetree/base/pcie@ffe250000/pcie@0
Flags: bus master, fast devsel, latency 0, IRQ 21
Memory at <ignored> (32-bit, non-prefetchable)
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-0000ffff [size=64K]
Memory behind bridge: e0000000-ffffffff [size=512M] <--------------------

Here, base adress is always beginning with e0000000 and acting like 32 bit.

and I also realized that changing only the linux kernel (.dtsi) file is enough to make driver run. PCIe2 - 3 and 4 does not even have a TLB entry. I checked tlb.c file in u-boot.

This is Device Tree Source successfully worked:

pci0: pcie@ffe240000 {
        reg = <0xf 0xfe240000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0x2 0x0 0x0 0x10000000
              0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

    pci1: pcie@ffe250000 {
        reg = <0xf 0xfe250000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0x2 0x10000000 0x1 0x00000000
              0x01000000 0 0 0xf 0xf8010000 0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0x1 0x00000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

    pci2: pcie@ffe260000 {
        reg = <0xf 0xfe260000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0x3 0x20000000 0 0x10000000
              0x01000000 0 0x00000000 0xf 0xf8020000 0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

    pci3: pcie@ffe270000 {
        reg = <0xf 0xfe270000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0x3 0x30000000 0 0x10000000
              0x01000000 0 0x00000000 0xf 0xf8030000 0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

View solution in original post

0 Kudos
22 Replies
14,841 Views
yusufalti333
Contributor IV

Hello yipingwang is there any update on SDK or Uboot mainline to use U-boot pcie bridge for devices that uses 4 G memory like GPU's ?

0 Kudos
14,875 Views
yipingwang
NXP TechSupport
NXP TechSupport

Please run "lspci -vv" to list PCI devices. 

The problem is caused by the LAW and MMU memory allocations for PCIe in

u-boot and Linux Kernel are not enough for your PCIe devices.
Please modify the memory map in u-boot and dts.

u-boot:

#ifdef CONFIG_PCIE1
#define CONFIG_SYS_PCIE1_MEM_VIRT       0x80000000
#define CONFIG_SYS_PCIE1_MEM_BUS        0xe0000000
#define CONFIG_SYS_PCIE1_MEM_PHYS       0xc00000000ull
#define CONFIG_SYS_PCIE1_MEM_SIZE       0x10000000      /* 256M */
#define CONFIG_SYS_PCIE1_IO_VIRT        0xf8000000
#define CONFIG_SYS_PCIE1_IO_BUS         0x00000000
#define CONFIG_SYS_PCIE1_IO_PHYS        0xff8000000ull
#define CONFIG_SYS_PCIE1_IO_SIZE        0x00010000      /* 64k */
#endif

In dts:

       pci0: pcie@ffe240000 {
                reg = <0xf 0xfe240000 0 0x10000>;
                ranges = <0x02000000 0 0xe0000000 0xc 0x0 0x0 0x10000000
                          0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00010000>;
                pcie@0 {
                        ranges = <0x02000000 0 0xe0000000
                                  0x02000000 0 0xe0000000
                                  0 0x10000000

                                  0x01000000 0 0x00000000
                                  0x01000000 0 0x00000000
                                  0 0x00010000>;
                };
        };

0 Kudos
14,875 Views
yusufalti333
Contributor IV

Hello Yiping,

This is how my t104xd4rdb.dtsi file looks;

I found no difference with the one you wrote. Am I missing something else ?

pci0: pcie@ffe240000 {
        reg = <0xf 0xfe240000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0xc 0x0 0x0 0x10000000
              0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

    pci1: pcie@ffe250000 {
        reg = <0xf 0xfe250000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0xc 0x10000000 0 0x10000000
              0x01000000 0 0 0xf 0xf8010000 0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

    pci2: pcie@ffe260000 {
        reg = <0xf 0xfe260000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0xc 0x20000000 0 0x10000000
              0x01000000 0 0x00000000 0xf 0xf8020000 0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

    pci3: pcie@ffe270000 {
        reg = <0xf 0xfe270000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0xc 0x30000000 0 0x10000000
              0x01000000 0 0x00000000 0xf 0xf8030000 0 0x00010000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

By the way, I am booting kernel with ramdisk using these parameteres,

tftp 2000000 uImage--4.19-r0-t1042d4rdb-64b-20191125053146.bin
tftp 4000000 t1042d4rdb--4.19-r0-t1042d4rdb-64b-20191125053146.dtb
tftp 12000000 asd.rootfs.ext2.gz.u-boot
bootm 2000000 12000000 4000000

Could this be a problem ?

0 Kudos
14,875 Views
yipingwang
NXP TechSupport
NXP TechSupport

Hello Yusuf Altıparmak,

What PCIe EP devices did you plugin on T1042D4RDB?

Would you please send your dtb file to me to do more investigation?

Thanks,

Yiping

0 Kudos
14,875 Views
yusufalti333
Contributor IV

Hello Yiping,

I added all the dts,dtsi files about t140x in my linux kernel and .dtb file created from bitbake fsl-image-networking-full. I also addeed my t1042d4rdb-64b.conf file.

I am trying to use kernel version 4.19 obtained from Yocto 2.7 Warrior (readme - qoriq-components/yocto-sdk - Repo manifest for QorIQ Yocto SDK ). All the bitbake operations were successfull. But my U-boot is generated from Yocto 2.0 (Generated from SDK 2.0 - 1703). It means U-boot and Linux kernel have different version of PCI drivers. Both of them is throwing same error.

I am trying to connect a GPU via PCIe but GPU driver is throwing error with exit code -12 which is 'OUT OF MEMORY' error. I read the documentation. Each PCI channel is has 256 MB for memory and 64 KB for I/O. Still didn't understand. Should I decrease the size of memory in .dts file ?  Like 0x10000000 -> 0x01000000 ?

Thanks.

0 Kudos
14,875 Views
yipingwang
NXP TechSupport
NXP TechSupport

Hello Yusuf,

Please increase memory map for PCIE1.

#ifdef CONFIG_PCIE1
#define CONFIG_SYS_PCIE1_MEM_VIRT       0x80000000
#define CONFIG_SYS_PCIE1_MEM_BUS        0xe0000000
#define CONFIG_SYS_PCIE1_MEM_PHYS       0xc00000000ull
#define CONFIG_SYS_PCIE1_MEM_SIZE       0x20000000      /* 256M */
#define CONFIG_SYS_PCIE1_IO_VIRT        0xf8000000
#define CONFIG_SYS_PCIE1_IO_BUS         0x00000000
#define CONFIG_SYS_PCIE1_IO_PHYS        0xff8000000ull
#define CONFIG_SYS_PCIE1_IO_SIZE        0x00020000      /* 64k */
#endif

In dts:

       pci0: pcie@ffe240000 {
                reg = <0xf 0xfe240000 0 0x10000>;
                ranges = <0x02000000 0 0xe0000000 0xc 0x0 0x0 0x20000000
                          0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00020000>;
                pcie@0 {
                        ranges = <0x02000000 0 0xe0000000
                                  0x02000000 0 0xe0000000
                                  0 0x10000000

                                  0x01000000 0 0x00000000
                                  0x01000000 0 0x00000000
                                  0 0x00010000>;
                };
        };

If your problem remains, please attach the console log to me again.

Thanks,

Yiping

0 Kudos
14,875 Views
yusufalti333
Contributor IV

Hello,

I tried 0x20000000 and 0x40000000 in build_t1042d4rdb-64b/tmp/work/t1042d4rdb_64b-fsl-linux/u-boot-qoriq/2018.09+fslgit-r0/git/include/configs/T104xRDB.h.

Result is same. It's giving the same result whether the GPU is plugged or not in PCIe.

Linux kernel is able to see GPU vendor number and product number. It's also is able to see GPU architecture (POLARIS12).

This is the console log from U-BOOT:

PCI: Probing PCI hardware
fsl-pci ffe240000.pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io 0x8000080000010000-0x800008000001ffff] (bus address [0x0000-0xffff])
pci_bus 0000:00: root bus resource [mem 0xc00000000-0xc0fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0000:00: root bus resource [bus 00]
iommu: Removing device ffe240000.pcie from group 19
iommu: Adding device 0000:00:00.0 to group 24
pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:00:00.0: PCI bridge to [bus 01-ff]
fsl-pci ffe250000.pcie: PCI host bridge to bus 0001:00
pci_bus 0001:00: root bus resource [io 0x8000080000021000-0x8000080000030fff] (bus address [0x0000-0xffff])
pci_bus 0001:00: root bus resource [mem 0xc10000000-0xc1fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0001:00: root bus resource [bus 00]
iommu: Removing device ffe250000.pcie from group 20
iommu: Adding device 0001:00:00.0 to group 19
pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0001:00:00.0: PCI bridge to [bus 01-ff]
fsl-pci ffe260000.pcie: PCI host bridge to bus 0002:00
pci_bus 0002:00: root bus resource [io 0x8000080000032000-0x8000080000041fff] (bus address [0x0000-0xffff])
pci_bus 0002:00: root bus resource [mem 0xc20000000-0xc2fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0002:00: root bus resource [bus 00]
iommu: Removing device ffe260000.pcie from group 21
iommu: Adding device 0002:00:00.0 to group 20
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
fsl-pci ffe270000.pcie: PCI host bridge to bus 0003:00
pci_bus 0003:00: root bus resource [io 0x8000080000043000-0x8000080000052fff] (bus address [0x0000-0xffff])
pci_bus 0003:00: root bus resource [mem 0xc30000000-0xc3fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0003:00: root bus resource [bus 00]
iommu: Removing device ffe270000.pcie from group 22
iommu: Adding device 0003:00:00.0 to group 21
pci 0003:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0003:00:00.0: PCI bridge to [bus 01-ff]
PCI: Cannot allocate resource region 0 of device 0000:00:00.0, will remap
PCI: Cannot allocate resource region 0 of device 0001:00:00.0, will remap
PCI: Cannot allocate resource region 0 of device 0002:00:00.0, will remap
PCI: Cannot allocate resource region 0 of device 0003:00:00.0, will remap
pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0: bridge window [io 0x8000080000010000-0x800008000001ffff]
pci 0000:00:00.0: bridge window [mem 0xc00000000-0xc0fffffff]
pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci 0001:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0001:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0001:00:00.0: PCI bridge to [bus 01]
pci 0001:00:00.0: bridge window [io 0x8000080000021000-0x8000080000030fff]
pci 0001:00:00.0: bridge window [mem 0xc10000000-0xc1fffffff]
pci_bus 0001:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci 0002:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0002:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [io 0x8000080000032000-0x8000080000041fff]
pci 0002:00:00.0: bridge window [mem 0xc20000000-0xc2fffffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci 0003:00:00.0: BAR 0: no space for [mem size 0x01000000]
pci 0003:00:00.0: BAR 0: failed to assign [mem size 0x01000000]
pci 0003:00:00.0: PCI bridge to [bus 01]
pci 0003:00:00.0: bridge window [io 0x8000080000043000-0x8000080000052fff]
pci 0003:00:00.0: bridge window [mem 0xc30000000-0xc3fffffff]
pci_bus 0003:00: Some PCI device resources are unassigned, try booting with pci=realloc

0 Kudos
14,876 Views
yipingwang
NXP TechSupport
NXP TechSupport

Hello Yusuf Altıparmak,

Please also modify the dts file to change the memory map as 512M.

    pci0: pcie@ffe240000 {
                reg = <0xf 0xfe240000 0 0x10000>;
                ranges = <0x02000000 0 0xe0000000 0xc 0x0 0x0 0x20000000
                          0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00020000>;
                pcie@0 {
                        ranges = <0x02000000 0 0xe0000000
                                  0x02000000 0 0xe0000000
                                  0 0x10000000

                                  0x01000000 0 0x00000000
                                  0x01000000 0 0x00000000
                                  0 0x00010000>;
                };
        };

Thanks,

Yiping

0 Kudos
14,876 Views
yusufalti333
Contributor IV

Hello yipingwang‌,

Are these .dtsi and configuration file of U-boot consistent ? If they are, is there any way to reconfigure other hardware memory map according to the hardware we want to change.

For example, After increasing PCIe1 memory 256 mb -> 512 mb, should I also change the start adress of PCIe2 device? If so, can you give me a sample to do this ?

Thanks.

0 Kudos
14,876 Views
yipingwang
NXP TechSupport
NXP TechSupport

Please try to comment PCIE2 and increase PCIe1 memory map from 256M to 512M.

0 Kudos
14,876 Views
yusufalti333
Contributor IV

I got Local Access Window error this time. This is the error log from u-boot console. (I also removed PCIe2 from dts files)

By the way I realized that, GPU is initializing as Root Complex according to U-boot. Is that a normal situation? 

[drm] initializing kernel modesetting (POLARIS12 0x1002:0x6987 0x1787:0x2389 0x80).
[drm] register mmio base: 0x20200000
[drm] register mmio size: 262144
[drm] add ip block number 0 <vi_common>
[drm] add ip block number 1 <gmc_v8_0>
[drm] add ip block number 2 <tonga_ih>
[drm] add ip block number 3 <powerplay>
[drm] add ip block number 4 <dm>
[drm] add ip block number 5 <gfx_v8_0>
[drm] add ip block number 6 <sdma_v3_0>
[drm] add ip block number 7 <uvd_v6_0>
[drm] add ip block number 8 <vce_v3_0>
fsDilsaabblliinngg lloocckk ddeebbuuggggiinngg dduuee ttoo kkeerrnneell ttaaiinntt

x80000001 cecar 0x02140000 cecar2 0x00000007
CaCuasuesde db yb y( f(rfromom M MCSCSR=R=aa00000)0):: LoLoad Error Report
KeKrenrenle lp apnainci c- -n onot t sysyncnciningg: Unrecoverable Machine check
:aClall lT rTarcaec:e

2033 Comm: udevd Tainted: G M 4.19.26+gc0c2141 #1
fsl_corenet_cf ffe018000.corenet-cf: Local Access Window Error
fsl_corenet_cf ffe018000.corenet-cf: address 0xc20203f0c, src id 0x85

0 Kudos
14,876 Views
yipingwang
NXP TechSupport
NXP TechSupport

Would you please provide your modified u-boot header file and Linux dts file?

0 Kudos
14,876 Views
yusufalti333
Contributor IV

Hello yipingwang‌,

I attached linux device tree source (.dtsi) file and U-boot configuration header (T104XRDB.h).

In my U-boot header, this line is;

#define CONFIG_SYS_SDRAM_SIZE    4096    /* for fixed parameter use */

describing like I have 4096 physical memory, but I have 8GB physical memory on my demo board ? Could this be a problem ?

Thanks.

0 Kudos
14,875 Views
yipingwang
NXP TechSupport
NXP TechSupport

#ifdef CONFIG_PCIE1
#define    CONFIG_SYS_PCIE1_MEM_VIRT    0x80000000
#define    CONFIG_SYS_PCIE1_MEM_BUS    0xe0000000
#define    CONFIG_SYS_PCIE1_MEM_PHYS    0xc00000000ull
#define CONFIG_SYS_PCIE1_MEM_SIZE    0x20000000    /* 256M */
#define CONFIG_SYS_PCIE1_IO_VIRT    0xf8000000
#define CONFIG_SYS_PCIE1_IO_BUS        0x00000000
#define CONFIG_SYS_PCIE1_IO_PHYS    0xff8000000ull
#define CONFIG_SYS_PCIE1_IO_SIZE    0x00020000    /* 64k */
#endif

dts:

    pci0: pcie@ffe240000 {
        reg = <0xf 0xfe240000 0 0x10000>;
        ranges = <0x02000000 0 0xe0000000 0xc 0x0 0x0 0x20000000
              0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00020000>;
        pcie@0 {
            ranges = <0x02000000 0 0xe0000000
                  0x02000000 0 0xe0000000
                  0 0x10000000

                  0x01000000 0 0x00000000
                  0x01000000 0 0x00000000
                  0 0x00010000>;
        };
    };

0 Kudos
14,875 Views
yusufalti333
Contributor IV

Hello yipingwang

This is the same settings I did for PCIe2. Do you want me to do the same for PCIe1 too ?( Increasing memory to 512 in both PCIe1 and PCIe2 and assuming PCIe3 is still removed from the system). GPU is plugged in PCIe2 which is X1 PCIE slot on demo board ( right side)

0 Kudos
14,875 Views
yipingwang
NXP TechSupport
NXP TechSupport

Hello Yusuf Altıparmak,

If GPU is plugged in PCIe2, please configure the above setting for PCIe2 and comment PCIe1.

Thanks,

Yiping

0 Kudos
14,874 Views
yusufalti333
Contributor IV

Hello yipingwang‌,

Commenting PCIe1 is crashed U-boot. I couldn't even reach 'Please any key to stop autoboot' section.

Can I use 4 GB memory size for PCIe2 ?

and also can you give me a Sample about 4 GB PCIe2 configuration (.dts and .h file) with including other PCIe channels layout (PCIe1-2-3-4)  ?

0 Kudos
14,874 Views
yipingwang
NXP TechSupport
NXP TechSupport

It's impossible to use 4G memory size for PCIE, because of the 4G memory space.

0 Kudos
14,874 Views
yusufalti333
Contributor IV

Hello yipingwang,  Why it's impossible ? I have lx2160 which has 32 GB PCIe memory for each PCIe device. Isn't 2^64 byte is the largest amount CPU can reach in a 64 bit CPU system ?

I still don't get what is wrong with this memory layout.

0 Kudos
14,844 Views
yipingwang
NXP TechSupport
NXP TechSupport

Because u-boot is 32bit, total 4G memory space for memory mapped devices.

0 Kudos