i.MX ProcessorsのトピックRe: Re: PCIe BAR length limit

PCIe BAR length limit

Nevyn75 — Wed, 12 Jun 2013 07:19:06 GMT

Hi.

I have a custom FPGA connected to i.mx6q on pcie bus.

I need to open 2 BARs with lengths of 64MB and 256KB but after some tests I found that it seems to be a limit of 4MB for memory regions under which everything's ok and over which I get these messages on kernel startup:

[ 1.307700] pci 0000:01:00.0: BAR 0: can't assign mem pref (size 0x4000000)

[ 1.314698] pci 0000:01:00.0: BAR 1: can't assign mem pref (size 0x40000)

I've found a similar problem on a previous application with Marvell Armada Xp and I solved it by changing a couple of parameters inside kernel board drivers, but I can't find similar defines inside linux sources for i.mx6.

Can someone help me in finding a solution please?

Thank you very much.

Re: PCIe BAR length limit

karina_valencia — Fri, 14 Jun 2013 17:24:35 GMT

Re: PCIe BAR length limit

richard.zhu Jun 14, 2013 2:45 AM (in response to imxcommunityscout)

Based on the imx_3.0.35.4.0 FSL Linux BSP release, the layout of the 16MB address space of PCIe RC is listed below:

* i.MX6 defines 16MB in the AXI address map for PCIe.

* That address space excepted the pcie registers is

* split and defined into different regions by iATU,

* with sizes and offsets as follows:

* RC:

* 0x0100_0000 --- 0x010F_FFFF 1MB IORESOURCE_IO

* 0x0110_0000 --- 0x01EF_FFFF 14MB IORESOURCE_MEM

* 0x01F0_0000 --- 0x01FF_FFFF 1MB Cfg + MSI + Registers

0x0110_0000 ~ 0x01EF_FFFF 14MB would be used for MEM allocation.

But the "IORESOURCE_SIZEALIGN" would be used during the Linux PCI/PCIe subsystem probe/scan the bus and allocate the resources.

If the 8MB MEM is required, the start address 0x0180_0000 would be used by Linux PCI/PCIe subsystem, trying to

allocate the 8MB MEM space (0x0180_0000 ~ 0x01FF_FFFF), this operation would be failed.

Because that the limitation of the MEM space of iMX6 PCIe RC is

* 0x0110_0000 --- 0x01EF_FFFF 14MB IORESOURCE_MEM

So, One method to allocate the 8MB(the biggest size of IO/MEM space) MEM space on iMX6 PCIe RC.

Adjust the layout of the 16MB address space of iMX6 PCIe RC, like this:

* RC:

* 0x0100_0000 --- 0x01DF_FFFF 14MB IORESOURCE_MEM

* 0x01E0_0000 --- 0x01EF_FFFF 1MB IORESOURCE_IO

* 0x01F0_0000 --- 0x01FF_FFFF 1MB Cfg + MSI + Registers

The 8MB space would be allocated from 0x0100_0000 ~ 0x017F_FFFF.

Based two imx6 platforms, one is used as RC, the other is used as RC.

Here is the test log at RC side:

iMX6 PCIe PCIe RC mode imx_pcie_pltfm_probe entering.

PCIE: imx_pcie_pltfm_probe start link up.

IMX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: BAR 0: assigned [mem 0x01800000-0x018fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01800000-0x018fffff 64bit pref] (PCI address [0x1800000-0x18fffff])

pci 0000:00:00.0: BAR 9: assigned [mem 0x01900000-0x019fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01a00000-0x01a0ffff pref]

pci 0000:00:00.0: BAR 7: assigned [io 0x1e00000-0x1e00fff]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01000000-0x017fffff] (PCI address [0x1000000-0x17fffff])

pci 0000:01:00.0: BAR 6: assigned [mem 0x01900000-0x0190ffff pref]

pci 0000:01:00.0: BAR 2: assigned [io 0x1e00000-0x1e00fff]

pci 0000:01:00.0: BAR 2: set to [io 0x1e00000-0x1e00fff] (PCI address [0x1e00000-0x1e00fff])

pci 0000:01:00.0: BAR 3: assigned [mem 0x01910000-0x019100ff pref]

pci 0000:01:00.0: BAR 3: set to [mem 0x01910000-0x019100ff pref] (PCI address [0x1910000-0x19100ff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0: bridge window [io 0x1e00000-0x1e00fff]

pci 0000:00:00.0: bridge window [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: bridge window [mem 0x01900000-0x019fffff pref]

Based on the imx_3.0.35.4.0 release, attach the patch

0001-imx-pcie-fix-the-8MB-size-MEM-space-allocation-failu.patch.zip(1.3 K)

Re: PCIe BAR length limit

claudechaussé — Thu, 12 Sep 2013 19:22:24 GMT

Hi,

Would it be possible to have access to this patch ? (I get access denied when I try to get it)

I have a similar problem trying to connect Imx6 to a DSP.

Thank you

Re: PCIe BAR length limit

karina_valencia — Thu, 12 Sep 2013 21:30:33 GMT

Can you try again?

Re: PCIe BAR length limit

claudechaussé — Thu, 12 Sep 2013 21:36:14 GMT

Hi,

I did but I get: https://community.freescale.com/login.jspa?authzFailed=true

Maybe my profile does not allow me to download stuff "???

Re: PCIe BAR length limit

karina_valencia — Thu, 12 Sep 2013 21:41:25 GMT

Maybe you can try with other browser to see if this works.

Re: PCIe BAR length limit

claudechaussé — Thu, 12 Sep 2013 21:47:11 GMT

I am sorry but I tried Safari, Chrome and Firefox and it does not work. Can you email me the patch since it is only 1.3K ? I really need it and I don't want to screw up the PCIe driver if I try to do it myself.

Re: PCIe BAR length limit

karina_valencia — Thu, 12 Sep 2013 21:55:25 GMT

Hi Claude,

I tried by community but it doesnt work.

you can send me a private message with your email

Re: PCIe BAR length limit

karina_valencia — Thu, 12 Sep 2013 22:11:16 GMT

Did you get it?

Re: PCIe BAR length limit

claudechaussé — Thu, 12 Sep 2013 22:13:38 GMT

I got it.

Fantastic :smileyhappy:

Re: PCIe BAR length limit

claudechaussé — Fri, 13 Sep 2013 21:56:36 GMT

Hi,

I have a similar issue where I try to connect a TI-KeyStone (6657) DSP to the i.MX6. I want to boot the DSP off the PCIe interface. The closest BAR Configuration that I can set on the TI side to make the i.mx6 happy is

BAR1: 4M

BAR2: 128M

BAR3: 128M

BAR4: 256M

BAR5: 256M

I know that the i.mx6 can not map all these BAR at once but I would have thought that it could handle at least the 4M of BAR1.

I tried to apply the 8M Patch and it did not help

Here is what the I.MX6 shows on boot up

IMX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x34000000)

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01000000-0x010fffff 64bit pref] (PCI address [0x1000000-0x10fffff])

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:01:00.0: BAR 4: can't assign mem pref (size 0x10000000)

pci 0000:01:00.0: BAR 5: can't assign mem pref (size 0x10000000)

pci 0000:01:00.0: BAR 2: can't assign mem pref (size 0x8000000)

pci 0000:01:00.0: BAR 3: can't assign mem pref (size 0x8000000)

pci 0000:01:00.0: BAR 1: can't assign mem pref (size 0x400000)

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x01100fff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01100000-0x01100fff] (PCI address [0x1100000-0x1100fff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0: bridge window [io disabled]

pci 0000:00:00.0: bridge window [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: bridge window [mem pref disabled]

And This is confirmed in lspci

01:00.0 Multimedia controller: Texas Instruments Device b006 (rev 01)

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 32 bytes

Interrupt: pin A routed to IRQ 155

Region 0: Memory at 01100000 (32-bit, non-prefetchable) [size=4K]

Region 1: Memory at <unassigned> (32-bit, prefetchable)

Region 2: Memory at <unassigned> (32-bit, prefetchable)

Region 3: Memory at <unassigned> (32-bit, prefetchable)

Region 4: Memory at <unassigned> (32-bit, prefetchable)

Region 5: Memory at <unassigned> (32-bit, prefetchable)

Can someone help me on that ?

Re: PCIe BAR length limit

karina_valencia — Tue, 17 Sep 2013 15:42:29 GMT

Hongxing Zhu

Please continue with the follow up.

Re: PCIe BAR length limit

richard_zhu — Wed, 18 Sep 2013 02:21:34 GMT

Hi Claude:
Can you trace the memory allocation procedure in the PCI/PCIe subsystem of Linux?

And find out why the 4MB memory can't be allocated?

As we discussed before, I used to allocate the 8MB memory space

in the i.MX6 PCIe EP/RC validation system(One i.MX6 used as RC, the other one used as EP).

About the details, please refer to the previous messages.

Best Regards.

Richard.

Re: PCIe BAR length limit

richard_zhu — Wed, 18 Sep 2013 02:22:18 GMT

Thanks for your reminder. :smileyhappy:

Best Regard

Richard.

Re: PCIe BAR length limit

claudechaussé — Tue, 24 Sep 2013 21:28:17 GMT

I added some traces in the resource allocation and I got this

pci 0000:01:00.0: pci_alloc_ bus:bfe99c00 size:400000 min:0 align:400000

pci 0000:01:00.0: pci_alloc_ fail trying non-prefetch

pci_bus_alloc_resource: Trying allocate sz:400000 mx:ffffffff mn:0

allocate_resource: call find_resource( root.start 1100000, root.end 11fffff | new.start 0, new.end 3fffff size:400000

Find Resource root->start 1100000, root->end 11fffff

Find Resource constraint->min 1100000, constraint->max ffffffff, constraint->align 400000

Find Resource Loop 1 tmp>start 1100000, tmp->end 11fffff

Find Resource Loop 2 avail>start 1400000, avail->end 11fffff

Find Resource Loop 3 alloc>start 1400000, alloc->end 17fffff

allocate_resource: Find Resource error -16

pci 0000:01:00.0: BAR 1: can't assign mem pref (size 0x400000)

As you can see, in bold, there is not enough room between root.start and root.end to fit my 4M of memory.

It does not seem to look at the whole regions which should be 0x0100_0000 --- 0x01DF_FFFF after the patch.

Re: PCIe BAR length limit

richard_zhu — Wed, 25 Sep 2013 02:11:36 GMT

Hi Claude:
Can you masked all the Bars excepted the Bar1(4MByte memory)?

In your previous log, it seems that Linux PCI/PCIe subsystem only let PCIe RC allocate the 0x0100_0000 ~ 0x010F_FFFF when "pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x34000000) ” is occurred.

MX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x34000000)

pci0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff 64bit pref] ßOnly 1M bytes memory space is allocated. That would be a problem when your 4M byes memory space specified by Bar1 is required to be allocated.

pci 0000:00:00.0: BAR 0: set to [mem 0x01000000-0x010fffff 64bit pref] (PCI address [0x1000000-0x10fffff])

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

In my previous i.MX6 PCIe RC/EP validation system, RC would allocate the responding memory regions required by PCIe EP.
IMX PCIe port: link up.

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: BAR 0: assigned [mem 0x01800000-0x018fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01800000-0x018fffff 64bit pref] (PCI address [0x1800000-0x18fffff])

pci 0000:00:00.0: BAR 9: assigned [mem 0x01900000-0x019fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01a00000-0x01a0ffff pref]

pci 0000:00:00.0: BAR 7: assigned [io 0x1e00000-0x1e00fff]

pci0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01000000-0x017fffff] (PCI address [0x1000000-0x17fffff])

pci 0000:01:00.0: BAR 6: assigned [mem 0x01900000-0x0190ffff pref]

pci 0000:01:00.0: BAR 2: assigned [io 0x1e00000-0x1e00fff]

pci 0000:01:00.0: BAR 2: set to [io 0x1e00000-0x1e00fff] (PCI address [0x1e00000-0x1e00fff])

pci 0000:01:00.0: BAR 3: assigned [mem 0x01910000-0x019100ff pref]

pci 0000:01:00.0: BAR 3: set to [mem 0x01910000-0x019100ff pref] (PCI address [0x1910000-0x19100ff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0: bridge window [io 0x1e00000-0x1e00fff]

pci 0000:00:00.0: bridge window [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: bridge window [mem 0x01900000-0x019fffff pref]

Re: PCIe BAR length limit

claudechaussé — Mon, 30 Sep 2013 16:50:09 GMT

Hi,

Unfortunately I cannot change the BAR requirement of the card since these are hardcoded BAR Configurations that the TI DSP is setting at boot time.

So I had to patch the kernel to reject any BAR that greater than 4M.

Si far so good.

thanks for you help

Re: PCIe BAR length limit

richard_zhu — Tue, 08 Oct 2013 02:04:55 GMT

Welcome.

Best Regards

Richard

Re: PCIe BAR length limit

idor — Wed, 20 Nov 2013 10:13:25 GMT

I have similar issues, but the patch is not reachable, can you please post again?

Re: Re: PCIe BAR length limit

richard_zhu — Wed, 20 Nov 2013 10:18:07 GMT

Hi:
Here it is.

Can you reach it?