IMX8MQ PCIe BAR 15: no space for [mem size 0x60000000 pref]

[mimi05633]

Hi NXP team,
I am trying to connect IMX8MQ with FPGA through mini PCIE.
I can detect it by lspci, but I have no idea to get address of FPGA to do read/write by devmem2.
I find fail log about PCIe and get "Memory at <unassigned>..." by lspci as below.
==============================================================
[ 2.310039] pci 0000:00:00.0: BAR 15: no space for [mem size 0x60000000 pref]
[ 2.316506] pci 0001:01:00.0: [168c:003c] type 00 class 0x028000
[ 2.322141] pci 0000:00:00.0: BAR 15: failed to assign [mem size 0x60000000 pref]
[ 2.326212] pci 0001:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[ 2.333361] pci 0000:00:00.0: BAR 0: assigned [mem 0x18000000-0x180fffff]
[ 2.333368] pci 0000:00:00.0: BAR 6: assigned [mem 0x18100000-0x1810ffff pref]

root@imx8mqevk:~# lspci -v
0000:00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 229
Memory at 18000000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: None
Prefetchable memory behind bridge: None
[virtual] Expansion ROM at 18100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] L1 PM Substates
Kernel driver in use: pcieport

0000:01:00.0 Non-VGA unclassified device: Altera Corporation Device 0000
Flags: fast devsel
[virtual] Memory at <unassigned> (64-bit, prefetchable) [size=1G]
[virtual] Memory at <unassigned> (64-bit, prefetchable) [size=4K]
Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [200] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
==============================================================
I notice that IMX8MQ PCIe memory space: 0x1800_0000 – 0x1fef_ffff (127M bytes).
Is FPGA PCIe failed for large size data(1G)?
If yes, how can I modify dtsi of PCIe to support large size? Or what can I do?

Attach imx8mq_pcie_fail_log.txt

Thank you for any help.
Minying

[vinuchandran]

@jamesbone @mimi05633 

I'm also interested in this thread.
I'm getting similar failed to assign message after around 16MBs.

[    4.877122] pci 0000:00:00.0: BAR 14: assigned [mem 0x18000000-0x185fffff]
[    4.884071] pci 0000:00:00.0: BAR 0: assigned [mem 0x18600000-0x186fffff 64bit]
[    4.891456] pci 0000:00:00.0: BAR 6: assigned [mem 0x18700000-0x1870ffff pref]
[    4.898754] pci 0000:01:00.0: BAR 2: assigned [mem 0x18000000-0x183fffff 64bit]
[    4.906116] pci 0000:01:00.0: BAR 0: assigned [mem 0x18400000-0x18407fff 64bit]
[    5.122276] pci 0001:00:00.0: BAR 0: assigned [mem 0x20000000-0x200fffff 64bit]
[    5.129598] pci 0001:00:00.0: BAR 14: assigned [mem 0x20100000-0x202fffff]
[    5.136542] pci 0001:00:00.0: BAR 6: assigned [mem 0x20300000-0x2030ffff pref]
[    5.143829] pci 0001:01:00.0: BAR 14: assigned [mem 0x20100000-0x201fffff]
[    5.150767] pci 0001:01:00.0: BAR 0: assigned [mem 0x20200000-0x2021ffff]
[    5.157572] pci 0001:01:00.1: BAR 0: assigned [mem 0x20220000-0x2023ffff]
[    5.164432] pci 0001:02:01.0: BAR 14: assigned [mem 0x20100000-0x201fffff]
[    5.171318] pci 0001:03:00.0: BAR 0: assigned [mem 0x20100000-0x2011ffff]
[    5.178177] pci 0001:03:00.1: BAR 0: assigned [mem 0x20120000-0x2013ffff]
[  239.307003] pcieport 0001:03:00.0: BAR 14: failed to assign [mem size 0x00300000]
[  239.322155] pcieport 0001:04:00.0: BAR 14: failed to assign [mem size 0x00300000]
[  239.337393] pci 0001:05:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit]
[  239.352532] pci 0001:05:00.0: BAR 2: failed to assign [mem size 0x00100000 64bit]
[  239.367058] pci 0001:05:00.0: BAR 4: failed to assign [mem size 0x00001000]
[  324.213416] pcieport 0001:04:00.0: BAR 14: failed to assign [mem size 0x00300000]
[  324.228415] pci 0001:05:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit]
[  324.243452] pci 0001:05:00.0: BAR 2: failed to assign [mem size 0x00100000 64bit]
[  324.251110] pci 0001:05:00.0: BAR 4: no space for [mem size 0x00001000]
[  324.257952] pci 0001:05:00.0: BAR 4: failed to assign [mem size 0x00001000]

AFAIK, iMX8MQ has 127MB each for pcie0 and pcie1.
Any pointers on why this is happening?

[vinuchandran]

My issue is resolved.

I was using an FPGA-> PCIe Switch (With DMA) -> iMX8.

The error (No space for mem and Failed to assign mem)was encountered when I program the FPGA after iMX boot and do a manual PCIe rescan with /sys/bus/pci/rescan.

I'm not getting this issue, if I program the FPGA before booting the iMX (The device will be available at the time of boot itself).
My total mem requirement ~20MB. 
Seems like the device should be present at the time of boot itself for correct memory assignments.

[jamesbone]

The problem is caused by the LAW and MMU memory allocations for PCIe in

u-boot and Linux Kernel are not enough for your PCIe devices.
Please modify the memory map in u-boot and dts.

u-boot:

#ifdef CONFIG_PCIE1
#define CONFIG_SYS_PCIE1_MEM_VIRT       0x80000000
#define CONFIG_SYS_PCIE1_MEM_BUS        0xe0000000
#define CONFIG_SYS_PCIE1_MEM_PHYS       0xc00000000ull
#define CONFIG_SYS_PCIE1_MEM_SIZE       0x10000000      /* 256M */
#define CONFIG_SYS_PCIE1_IO_VIRT        0xf8000000
#define CONFIG_SYS_PCIE1_IO_BUS         0x00000000
#define CONFIG_SYS_PCIE1_IO_PHYS        0xff8000000ull
#define CONFIG_SYS_PCIE1_IO_SIZE        0x00010000      /* 64k */
#endif

In dts:

       pci0: pcie@ffe240000 {
                reg = <0xf 0xfe240000 0 0x10000>;
                ranges = <0x02000000 0 0xe0000000 0xc 0x0 0x0 0x10000000
                          0x01000000 0 0x0 0xf 0xf8000000 0x0 0x00010000>;
                pcie@0 {
                        ranges = <0x02000000 0 0xe0000000
                                  0x02000000 0 0xe0000000
                                  0 0x10000000

                                  0x01000000 0 0x00000000
                                  0x01000000 0 0x00000000
                                  0 0x00010000>;
                };
        };

[mimi05633]

Hi Jamesbone,

Thank you for your reply.
My customize board are base on imx8mq evk and linux version is LF_L5.4.3_1.0.0.
Does the same method to modify the memory map as you say on imx8mq evk?
Because of I cannot find CONFIG_SYS_PCIE1_MEM_SIZE for 127MB in u-boot, and my dts is as below.

pcie0: pcie@33800000 {
    compatible = "fsl,imx8mq-pcie";
    reg = <0x33800000 0x400000>,
             <0x1ff00000 0x80000>;
    reg-names = "dbi", "config";
    #address-cells = <3>;
    #size-cells = <2>;
    device_type = "pci";
    bus-range = <0x00 0xff>;
    ranges = <0x81000000 0 0x00000000 0x1ff80000 0 0x00010000 /* downstream I/O 64KB */
                     0x82000000 0 0x18000000 0x18000000 0 0x07f00000>; /* non-prefetchable memory */
    ...

And the ranges about non-prefetchable memory setting can be used on prefetchable memory type?

Looking forward to your reply.
Best regards,
minying

[jamesbone]

Please check PCI EP examples (0001-ENGR00268112-pcie-emaluate-the-pcie-ep-as-ram-device.patch.zip) on

i.MX6Q PCIe EP/RC Validation System  ( it works the same in the i.Mx8M since they have the same PCIe module)

for latest kernels one can look at sect.40.4 Using PCIe Endpoint and Running Tests in Linux Reference Manual

[mimi05633]

Hi @jamesbone,

Thank you for your reply.
I am sorry my explanation is not clear.

In my topology, IMX8MQ(RC)--mini PCIe--FPGA(EP, Altera)

I notice that IMX8MQ PCIe memory space: 0x1800_0000 – 0x1fef_ffff (127M bytes).
Can IMX8MQ(RC) memory space be modified more than 127 MB?
If yes, how can I do?

Looking forward to your reply.
Best regards,
minying