i.MX6 PCIe: supporting devices larger than 16MB

[kitz36]

Has anyone been able to enumerate PCIe devices summing to more than 14MB (and use them)?  Is it possible?

Freescale's BSP allocates 14MB (0x0110_0000 - 0x01DF_FFFF) to the PCIe root controller.  See linux/arch/arm/mach-mx6/pcie.c, imx_pcie_setup().  Our design includes an FPGA that wants to use 256MB of PCIe memory.  In order to get Linux to enumerate the large device I allocated an additional 256MB resource block to PCI.  I arbitrarily chose the address range 0x8000_0000 - 0x8FFF_FFFF: 

if (pp->index == 0) {
    pp->res[2].start = 0x80000000;
    pp->res[2].end = pp->res[2].start + SZ_256M - 1;
}
pp->res[2].flags = IORESOURCE_MEM;
if (request_resource(&iomem_resource, &pp->res[2]))
    panic("Request PCIe FPGA resource failed\n");
sys->resource[2] = &pp->res[2];

I expect that I need to somehow associate this address range with PCIe so AXI accesses are routed to the PCIe core.  Right now we don't see any PCIe traffic from the i.MX6 to the FPGA.

Anyone have any ideas?

-Charlie

[alexandero_]

you mentioned that you managed to map the entire 4GB memory region for inbound TCP. how did you manage to do that?

[kitz36 · Accepted solution]

The i.MX6 cannot map more memory for outbound PCI than exists in the memory map (16MB).  Some of that memory is used for i.MX6 internal registers.  Given that PCI sizes must be a power of two, the maximum outbound PCI memory supported by the i.MX6 is 8MB.  Mapping all 16MB might be possible assuming the overlapped memory is well managed.

For inbound PCI, the entire 4GB memory range may be mapped.

I was able to map an 8MB endpoint.  This required a small modification to the BSP to make the assignable PCI memory start on an 8MB boundary (0x0100_0000).

-Charlie

[travisr]

Charlie,

I know it's almost six years later, but you indicated that 4GB was available for inbound and I haven't seen any indication elsewhere that this is possible. Can you elaborate on how this would be achieved? Thanks.

[FrankLi1z]

Do you have specific reason why need 256 Memory?

Generally FPAG is bus master, which can use RC's syste memory directly.

8MB Space is used for register access.

[erkanaltan]

Hi Zhi,

I have the same issue.  I am using Linux Kernel  3.14.28 and my FPGA BAR 2 requests 64MB outbound memory.   Is there no way to assign this much memory

for outbound PCIe traffic, or did I miss something ??

E.

[kitz36]

Our implementation uses the processor as the bus master with an FPGA has the only endpoint. The FPGA register space was larger than 8MB. We since worked around the issue by reducing the size of the FPGA register space.

[volki]

We thought about using an i.mx6 with a NVidia GPU and a FPGA both connected using PCIe (via a switch). As far as I understood the description of the RDMA concept for CUDA the NVidia graphics card has a BAR size of 256 MB. So this is impossible to be used with an iMX6. Is this correct?