PCIe BAR length limit

[Nevyn75]

Hi.

I have a custom FPGA connected to i.mx6q on pcie bus.

I need to open 2 BARs with lengths of 64MB and 256KB but after some tests I found that it seems to be a limit of 4MB for memory regions under which everything's ok and over which I get these messages on kernel startup:

[    1.307700] pci 0000:01:00.0: BAR 0: can't assign mem pref (size 0x4000000)

[    1.314698] pci 0000:01:00.0: BAR 1: can't assign mem pref (size 0x40000)

I've found a similar problem on a previous application with Marvell Armada Xp and I solved it by changing a couple of parameters inside kernel board drivers, but I can't find similar defines inside linux sources for i.mx6.

Can someone help me in finding a solution please?

Thank you very much.

[idor]

I have similar issues, but the patch is not reachable, can you please post again?

[richard_zhu]

Hi:
Here it is.

Can you reach it?

[idor]

Thank you for the patch its accessible now.

My problem is somehow different:

I have a pci device connected  as device 0000:01:00.0, and I need access to bars 0 and 2.

In order to remap the bars, I tried using the pci_resource_start/end/len functions (which seems to work fine over my ubuntu PC).

The bars resources are all set to zero (including start/end/flags), therefore I cant call the ioremap functions, as there are no addresses to remap.

Can someone help me in finding a solution please?

Thank you very much.

[richard_zhu]

Hi:

As I remember that you should write 0xffff_ffff to the bar# firstly, then reads it back if you want to read the actual values contained in the bar#.

Please refer to the chapter 6.2.5 base address  of PCI 3.0 spec.

Best Regards

Richard

[idor]

thank you for your answer.

you are correct regarding the pci spec, but this is done by the kernel probe of the pci.

during probe, it should allocate the resources for bars of mem/io for drivers access.

I'm using the Nit6X_PCIE and the mPCIe to PCI-E DB from boundary devices to connect to my PCIe device.

I'm able to probe my device, read the configuration, enable msi, etc, but when calling: pci_ioremap_bar, or any other pci_resource_* apis, it returns 0 for them.

running the same from windows/ubuntu machines works just fine.

[richard_zhu]

Hi:
Welcome.

Can you read out the correct values of the Bar# of your ep device manually?

Best Regards

Richard.

[idor]

yes, the data is there alright, device ask for 128MB for each bar (0+2).

after drilling into kernel code, I can tell that its not size that get rejected, but if fails on the match flags check:

in driver/pci/bus.c, pci_bus_alloc_resource: "if ((res->flags ^ r->flags) & type_mask)"

the flags are always zero.

[richard_zhu]

O, IC.
The max space size can be allocated by imx6 pcie rc is 8Mbytes.

128MB size spaces can't be allocated.

So, you can't see them either.

Best Regards

Richard

[salil]

Hello,

We are building a PCIe bus based system using iMX6. iMX6 RC is connected to a 21 port PLX switch. We have just realized that 14M+1M+(1M-16k)+16k of total space has been allocated to the PCIe controller for mapping and configuration purposes. Now, since virtual PCI-to-PCI bridge associated with each port of the PLX switch would have alignment and size constraints of 1MB (dictated by its Memory Base/Limit registers). We will not be able to memory-map more than 7 bridge ports. This we have cross verified by placing more than 7 NIC cards on the PLX switch ports. Memory-mapping for anything more than 7 fails and hence the NICs beyond that does not work.

We have few questions here (maybe peculiar :smileygrin: & difficult to digest):

1) Can we increase the size of the macro in file mx6.h. Though we totally understand if peripherals on the SoC sitting at particular address have been defined in such a way that they respond to those particular addresses represented in mx6.h then this might not be possible.

2) Is it by anyway possible to get away with 14M limit of memory-mapping of the iMX6? Is it possible to reconfigure the PCIe Controller for new set of addresses or maybe increased range?

We could see from file mx6.h below macros dictate the size of the mapping:

file: mx6.h

#define PCIE_ARB_BASE_ADDR	0x01000000
#define PCIE_ARB_END_ADDR	0x01FFFFFF

I could also see memory beyond 0x01FFFFFF i.e. 0x2000000 is being used:

#define AIPS1_ARB_BASE_ADDR	0x02000000
#define AIPS1_ARB_END_ADDR	0x020FFFFF

regards

Salil

NOTE: Please find the  memory-mapping file attached with 4 cards.

[idor]

okay, so I hacked the pci probing a bit, hard coding the requested size and override the 128MB with a different smaller value.

after some trial & error, I managed to set the bars requested size to a 512KB which were remapped successfully by the driver.

question is, why just 512KB, and not 4MB (only bars 0 and 2 are active), initially I thought these are i/o bars (which 1MB of ram could be explained - it is the limit), but that's not the case, both are of type memory.

[karina_valencia · Accepted solution]

Re: PCIe BAR length limit

richard.zhu Jun 14, 2013 2:45 AM (in response to imxcommunityscout)

Based on the imx_3.0.35.4.0  FSL Linux BSP release, the layout of the 16MB address space of PCIe RC is listed below:

         * i.MX6 defines 16MB in the AXI address map for PCIe.

         *

         * That address space excepted the pcie registers is

         * split and defined into different regions by iATU,

         * with sizes and offsets as follows:

         *

         * RC:

         * 0x0100_0000 --- 0x010F_FFFF 1MB IORESOURCE_IO

         * 0x0110_0000 --- 0x01EF_FFFF 14MB IORESOURCE_MEM

         * 0x01F0_0000 --- 0x01FF_FFFF 1MB Cfg + MSI + Registers

         *

0x0110_0000 ~ 0x01EF_FFFF 14MB would be used for MEM allocation.

But the "IORESOURCE_SIZEALIGN" would be used during the Linux PCI/PCIe subsystem probe/scan the bus and allocate the resources.

If the 8MB MEM is required, the start address 0x0180_0000 would be used by Linux PCI/PCIe subsystem, trying to

allocate the 8MB MEM space (0x0180_0000 ~ 0x01FF_FFFF), this operation would be failed.

Because that the limitation of the MEM space of iMX6 PCIe RC is

* 0x0110_0000 --- 0x01EF_FFFF 14MB IORESOURCE_MEM

So, One method to allocate the 8MB(the biggest size of IO/MEM space) MEM space on iMX6 PCIe RC.

Adjust the layout of the 16MB address space of iMX6 PCIe RC, like this:

         * RC:

         * 0x0100_0000 --- 0x01DF_FFFF 14MB IORESOURCE_MEM

         * 0x01E0_0000 --- 0x01EF_FFFF 1MB IORESOURCE_IO

         * 0x01F0_0000 --- 0x01FF_FFFF 1MB Cfg + MSI + Registers

The 8MB space would be allocated from 0x0100_0000 ~ 0x017F_FFFF.

Based two imx6 platforms, one is used as RC, the other is used as RC.

Here is the test log at RC side:

iMX6 PCIe PCIe RC mode imx_pcie_pltfm_probe entering.

PCIE: imx_pcie_pltfm_probe start link up.

IMX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: BAR 0: assigned [mem 0x01800000-0x018fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01800000-0x018fffff 64bit pref] (PCI address [0x1800000-0x18fffff])

pci 0000:00:00.0: BAR 9: assigned [mem 0x01900000-0x019fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01a00000-0x01a0ffff pref]

pci 0000:00:00.0: BAR 7: assigned [io  0x1e00000-0x1e00fff]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01000000-0x017fffff] (PCI address [0x1000000-0x17fffff])

pci 0000:01:00.0: BAR 6: assigned [mem 0x01900000-0x0190ffff pref]

pci 0000:01:00.0: BAR 2: assigned [io  0x1e00000-0x1e00fff]

pci 0000:01:00.0: BAR 2: set to [io  0x1e00000-0x1e00fff] (PCI address [0x1e00000-0x1e00fff])

pci 0000:01:00.0: BAR 3: assigned [mem 0x01910000-0x019100ff pref]

pci 0000:01:00.0: BAR 3: set to [mem 0x01910000-0x019100ff pref] (PCI address [0x1910000-0x19100ff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0:   bridge window [io  0x1e00000-0x1e00fff]

pci 0000:00:00.0:   bridge window [mem 0x01000000-0x017fffff]

pci 0000:00:00.0:   bridge window [mem 0x01900000-0x019fffff pref]

Based on the imx_3.0.35.4.0 release, attach the patch

• 0001-imx-pcie-fix-the-8MB-size-MEM-space-allocation-failu.patch.zip(1.3 K)

[flamenca]

how can i view the patch

[claudechaussé]

Hi,

I have a similar issue where I try to connect a TI-KeyStone (6657) DSP to the i.MX6. I want to boot the DSP off the PCIe interface. The closest BAR Configuration that I can set on the TI side to make the i.mx6 happy is

BAR1:     4M

BAR2:     128M

BAR3:     128M

BAR4:     256M

BAR5:     256M

I know that the i.mx6 can not map all these BAR at once but I would have thought that it could handle at least the 4M of BAR1.

I tried to apply the 8M Patch and it did not help

Here is what the I.MX6 shows on boot up

IMX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x34000000)

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01000000-0x010fffff 64bit pref] (PCI address [0x1000000-0x10fffff])

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:01:00.0: BAR 4: can't assign mem pref (size 0x10000000)

pci 0000:01:00.0: BAR 5: can't assign mem pref (size 0x10000000)

pci 0000:01:00.0: BAR 2: can't assign mem pref (size 0x8000000)

pci 0000:01:00.0: BAR 3: can't assign mem pref (size 0x8000000)

pci 0000:01:00.0: BAR 1: can't assign mem pref (size 0x400000)

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x01100fff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01100000-0x01100fff] (PCI address [0x1100000-0x1100fff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0:   bridge window [io  disabled]

pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]

pci 0000:00:00.0:   bridge window [mem pref disabled]

And This is confirmed in lspci

01:00.0 Multimedia controller: Texas Instruments Device b006 (rev 01)

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 32 bytes

        Interrupt: pin A routed to IRQ 155

        Region 0: Memory at 01100000 (32-bit, non-prefetchable) [size=4K]

        Region 1: Memory at <unassigned> (32-bit, prefetchable)

        Region 2: Memory at <unassigned> (32-bit, prefetchable)

        Region 3: Memory at <unassigned> (32-bit, prefetchable)

        Region 4: Memory at <unassigned> (32-bit, prefetchable)

        Region 5: Memory at <unassigned> (32-bit, prefetchable)

Can someone help me on that ?

[richard_zhu]

Hi Claude:
Can you trace the memory allocation procedure in the PCI/PCIe subsystem of Linux?

And find out why the 4MB memory can't be allocated?

As we discussed before, I used to allocate the 8MB memory space

in the i.MX6 PCIe EP/RC validation system(One i.MX6 used as RC, the other one used as EP).

About the details, please refer to the previous messages.

Best Regards.

Richard.

[claudechaussé]

I added some traces in the resource allocation and I got this

pci 0000:01:00.0: pci_alloc_ bus:bfe99c00 size:400000 min:0 align:400000

pci 0000:01:00.0: pci_alloc_ fail trying non-prefetch

pci_bus_alloc_resource: Trying allocate sz:400000 mx:ffffffff mn:0

allocate_resource: call find_resource( root.start  1100000, root.end 11fffff | new.start 0, new.end 3fffff size:400000

Find Resource root->start 1100000, root->end 11fffff

Find Resource constraint->min 1100000, constraint->max ffffffff, constraint->align 400000

Find Resource Loop 1 tmp>start 1100000, tmp->end 11fffff

Find Resource Loop 2 avail>start 1400000, avail->end 11fffff

Find Resource Loop 3 alloc>start 1400000, alloc->end 17fffff

allocate_resource: Find Resource error -16

pci 0000:01:00.0: BAR 1: can't assign mem pref (size 0x400000)

As you can see, in bold, there is not enough room between root.start and root.end to fit my 4M of memory.

It does not seem to look at the whole regions which should be 0x0100_0000 --- 0x01DF_FFFF after the patch.

[richard_zhu]

Hi Claude:
Can you masked all the Bars excepted the Bar1(4MByte memory)?

In your previous log, it seems that Linux PCI/PCIe subsystem only let PCIe RC allocate the 0x0100_0000 ~ 0x010F_FFFF when "pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x34000000) ” is occurred.

MX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x34000000)

pci0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff 64bit pref]  ßOnly 1M bytes memory space is allocated. That would be a problem when your 4M byes memory space specified by Bar1 is required to be allocated.

pci 0000:00:00.0: BAR 0: set to [mem 0x01000000-0x010fffff 64bit pref] (PCI address [0x1000000-0x10fffff])

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

In my previous i.MX6 PCIe RC/EP validation system, RC would allocate the responding memory regions required by PCIe EP.
IMX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: BAR 0: assigned [mem 0x01800000-0x018fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01800000-0x018fffff 64bit pref] (PCI address [0x1800000-0x18fffff])

pci 0000:00:00.0: BAR 9: assigned [mem 0x01900000-0x019fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01a00000-0x01a0ffff pref]

pci 0000:00:00.0: BAR 7: assigned [io 0x1e00000-0x1e00fff]

pci0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01000000-0x017fffff] (PCI address [0x1000000-0x17fffff])

pci 0000:01:00.0: BAR 6: assigned [mem 0x01900000-0x0190ffff pref]

pci 0000:01:00.0: BAR 2: assigned [io 0x1e00000-0x1e00fff]

pci 0000:01:00.0: BAR 2: set to [io 0x1e00000-0x1e00fff] (PCI address [0x1e00000-0x1e00fff])

pci 0000:01:00.0: BAR 3: assigned [mem 0x01910000-0x019100ff pref]

pci 0000:01:00.0: BAR 3: set to [mem 0x01910000-0x019100ff pref] (PCI address [0x1910000-0x19100ff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0: bridge window [io 0x1e00000-0x1e00fff]

pci 0000:00:00.0: bridge window [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: bridge window [mem 0x01900000-0x019fffff pref]

[claudechaussé]

Hi,

Unfortunately I cannot change the BAR requirement of the card since these are hardcoded BAR Configurations that the TI DSP is setting at boot time.

So I had to patch the kernel to reject any BAR that greater than 4M.

Si far so good.

thanks for you help

[richard_zhu]

Welcome.

Best Regards

Richard

[karina_valencia]

Hongxing Zhu

Please continue with the follow up.

[richard_zhu]

Thanks for your reminder. :smileyhappy:

Best Regard

Richard.