Can not allocate PCIe region when size is 8M on Sabre platform

chrishossack · ‎11-04-2015

Hi

I’ve recompiled the imx6 kernel to support PCIe (see Re: Changing the Kernel configuration for i.MX6 SABRE ) and connected my PCI TMS320C6424 via a PCIe-toPCI adaptor to the sabre mini-PCIe slot. I can see my device correctly attached to the PCIe bus

$ lspci

00:00.0 PCI bridge: Device 16c3:abcd (rev 01)

01:00.0 PCI bridge: Pericom Semiconductor PI7C9X111SL PCIe-to-PCI Reversible Bridge (rev 02)

02:04.0 Signal processing controller: Texas Instruments TMS320C6424

But the kernel has failed to assigned any PCIe regions greater than 4M.

$ lspci -vvv -s 02:04.0

02:04.0 Signal processing controller: Texas Instruments TMS320C6424

Subsystem: Device 04b8:0136

Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Interrupt: pin A routed to IRQ 155

Region 0: Memory at <unassigned> (32-bit, prefetchable) [disabled]

Region 1: Memory at 01400000 (32-bit, non-prefetchable) [disabled] [size=64K]

Region 2: Memory at 01000000 (32-bit, non-prefetchable) [disabled] [size=4M]

Region 3: Memory at <unassigned> (32-bit, prefetchable) [disabled]

Region 4: Memory at <unassigned> (32-bit, prefetchable) [disabled]

Region 5: Memory at <unassigned> (32-bit, prefetchable) [disabled]

When I plug in my TMS320C6424 board in to 386 Ubuntu PC it does allocate the 8M regions.

$ lspci -vvv -s 01:04.0

01:04.0 Signal processing controller: Texas Instruments TMS320C6424

Subsystem: Device 04b8:0136

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-

Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 64, Cache Line Size: 32 bytes

Interrupt: pin A routed to IRQ 16

Region 0: Memory at fd800000 (32-bit, prefetchable) [size=8M]

Region 1: Memory at febf0000 (32-bit, non-prefetchable) [size=64K]

Region 2: Memory at fe400000 (32-bit, non-prefetchable) [size=4M]

Region 3: Memory at fd000000 (32-bit, prefetchable) [size=8M]

Region 4: Memory at fc800000 (32-bit, prefetchable) [size=8M]

Region 5: Memory at fc000000 (32-bit, prefetchable) [size=8M]

I looked around for other solutions and found this possible patch PCIe BAR length limit written by karinavalencia for imx_3.0.35.4.0 but when I added this to my linux-imx_3.10.53.bbappend I get the following error

ERROR: Command Error: exit status: 1 Output:

Applying patch 0001-imx-pcie-fix-the-8MB-size-MEM-space-allocation-failu.patch

can't find file to patch at input line 33

Perhaps you used the wrong -p or --strip option?

The text leading up to this was:

--------------------------

|From da2b24e9270da50cc1547e436dfdb648b9ce6b3a Mon Sep 17 00:00:00 2001

|From: Richard Zhu <r65037@freescale.com>

|Date: Fri, 14 Jun 2013 15:50:55 +0800

|Subject: [PATCH] imx: pcie: fix the 8MB size MEM space allocation failure

|

|"IORESOURCE_SIZEALIGN" would be used during the Linux PCI/PCIe

|subsystem probe/scan the bus and allocate the resources.

|

|If the 8MB MEM is required, the start address 0x0180_0000 would be

|used by Linux PCI/PCIe subsystem, trying to allocate the 8MB MEM

|space(0x0180_0000 ~ 0x01FF_FFFF), this operation would be failed.

|Because the address if outof 0x0110_0000 ~ 0x01EF_FFFF limitaion.

|

|solution:

|One method to allocate the 8MB(the biggest size of IO/MEM space) MEM

|space on iMX6 PCIe RC.

|Adjust the layout of the 16MB address space of iMX6 PCIe RC, like this:

| * 0x0100_0000 --- 0x01DF_FFFF 14MB IORESOURCE_MEM

| * 0x01E0_0000 --- 0x01EF_FFFF 1MB IORESOURCE_IO

| * 0x01F0_0000 --- 0x01FF_FFFF 1MB Cfg + MSI +

| * Registers

|The 8MB space would be allocated from 0x0100_0000 ~ 0x017F_FFFF.

|

|Signed-off-by: Richard Zhu <r65037@freescale.com>

|---

| arch/arm/mach-mx6/pcie.c | 8 ++++----

| 1 files changed, 4 insertions(+), 4 deletions(-)

|

|diff --git a/arch/arm/mach-mx6/pcie.c b/arch/arm/mach-mx6/pcie.c

|index c4e8f7d..6f38b0a 100644

|--- a/arch/arm/mach-mx6/pcie.c

|+++ b/arch/arm/mach-mx6/pcie.c

--------------------------

No file to patch. Skipping patch.

3 out of 3 hunks ignored

Patch 0001-imx-pcie-fix-the-8MB-size-MEM-space-allocation-failu.patch does not apply (enforce with -f)

ERROR: Function failed: patch_do_patch

ERROR: Logfile of failure stored in: /mnt/data/imx6/fsl-release-bsp-3-10-53/build-x11/tmp/work/imx6qsabresd-poky-linux-gnueabi/linux-imx/3.10.53-r0/temp/log.do_patch.4295

ERROR: Task 1 (/mnt/data/imx6/fsl-release-bsp-3-10-53/sources/meta-fsl-bsp-release/imx/meta-fsl-arm/recipes-kernel/linux/linux-imx_3.10.53.bb, do_patch) failed with exit code '1'

It looks like the file arch/arm/mach-mx6/pcie.c no longer exists at that location and grep-ing for a similar one produces no winners.

$ find ./tmp/work/imx6qsabresd-poky-linux-gnueabi/linux-imx/3.10.53-r0/git -name pcie.c

./drivers/net/wireless/mwifiex/pcie.c

./arch/arm/mach-cns3xxx/pcie.c

./arch/arm/mach-tegra/pcie.c

./arch/arm/plat-orion/pcie.c

./arch/arm/mach-dove/pcie.c

./arch/arm/mach-mv78xx0/pcie.c

./arch/arm/mach-kirkwood/pcie.c

I need these 8M PCIe regions to access the TMS320C6424 on-board memory, and these are fixed at power up so I can’t change them.

Can anybody help?

Cheers

Chris

Yuri · ‎11-05-2015

Hello,

The following may be helpful :

Re: PCIe BAR length limit

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

chrishossack · ‎11-06-2015

Hi Yuri,

thank you for your answer, but if you read my question you will see that over half the question is actually dealing with the link you sent me. I've even used the suggested patch from the link which generated errors which I pasted above ;-(

cheers

Chris

MT · ‎12-29-2015

Hi Chris

You do not mention which version of the BSP you are using, but I suspect it is a later one. The ranges for PCIe are configured in the dts files.

arch/arm/boot/dts/imx6qdl.dtsi

BR

Mark

chrishossack · ‎12-30-2015

Hi Mark,

thank you for your help. I'm using imx-3.14.28-1.0.0_ga. I'm happy to use any branch since we are still in development.

I've looked at imx6qdl.dtsi and found the pcie section below

pcie: pcie@0x01000000 {

compatible = "fsl,imx6q-pcie", "snps,dw-pcie";

reg = <0x01ffc000 0x4000>, <0x01f00000 0x80000>;

reg-names = "dbi", "config";

#address-cells = <3>;

#size-cells = <2>;

device_type = "pci";

ranges = <0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x01000000 0 0x00f00000>; /* non-prefetchable memory */

num-lanes = <1>;

I’ve read the following link Device Tree Usage - FDTWiki and it seems that we have 0xf00000 (15 Mbytes) of non-prefetchable memory. I’ve tried to increase this to 40MB,

ranges = <0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x02000000 0 0x02800000>; /* non-prefetchable memory */

but then all regions fail.

lspci –vvv

Region 0: Memory at 02000000 (32-bit, prefetchable) [disabled] [size=8M]

Region 1: Memory at 04400000 (32-bit, non-prefetchable) [disabled] [size=64K]

Region 2: Memory at 04000000 (32-bit, non-prefetchable) [disabled] [size=4M]

Region 3: Memory at 02800000 (32-bit, prefetchable) [disabled] [size=8M]

Region 4: Memory at 03000000 (32-bit, prefetchable) [disabled] [size=8M]

Region 5: Memory at 03800000 (32-bit, prefetchable) [disabled] [size=8M]

Then I tried to add a prefetechable region, eg

ranges = <0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x01000000 0 0x00f00000 /* 15MB non-prefetchable memory */

0xC2000000 0 0x02000000 0x01fa0000 0 0x02800000>; /* 40MB prefetchable memory */

But then this fails to even see the non-prefetchable.

lspci –vvv

Region 0: Memory at 02000000 (32-bit, prefetchable) [disabled] [size=8M]

Region 3: Memory at 02800000 (32-bit, prefetchable) [disabled] [size=8M]

Region 4: Memory at 03000000 (32-bit, prefetchable) [disabled] [size=8M]

Region 5: Memory at 03800000 (32-bit, prefetchable) [disabled] [size=8M]

Am I using an invalid PCI/host address? How can I locate unused address spaces?

Can you give me any clues as to what I’m doing wrong?

Cheers

Chris

MT · ‎01-04-2016

Hi Chris,

You can't allocate more than 16MB space as the physical limit in the memory map is 16MB. The PCIe memory is physically located from 0x100_0000 to 0x1FF_FFFF.

Looking at the patch and comparing with the dtsi file then it should be able to allocate 8MB, without modification as the 3.14.28 BSP appears to have applied the changes already. Let me do some more investigation.

BR

Mark

chrishossack · ‎01-05-2016

Hi Mark,

Is the 16MB an actual physical limit of the imx6 chip? I've looked at the i.MX6 ref manual and I couldn't see any 16MB limit in the pcie section so I'm hoping there's still a chance ;-)

The pci card I'm using requests 32M (4*8M) prefetchable and 4.064M (4M+64K) non-prefetchable regions. Also I need two of these cards plugs in ;-(

I think that my PCI cards will work if I can get at least one 8M prefetchable and 4.064M (4M+64K) non-prefetchable region per card.

Is the 16M limit per PCI card or for all PCI cards attached?

I've tried to snatch 8M from non-prefetchable to prefetchable, but the kernel still didn't allocate any 8M prefetchable region.

ranges = < 0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0xC2000000 0 0x01000000 0x01700000 0 0x00800000 /* 8MB prefetchable memory */

0x82000000 0 0x02000000 0x01000000 0 0x00700000>; /* 7MB non-prefetchable memory */

lspci -vvv

Region 0: Memory at (32-bit, prefetchable) [disabled]

Region 1: Memory at 01400000 (32-bit, non-prefetchable) [disabled] [size=64K]

Region 2: Memory at 01000000 (32-bit, non-prefetchable) [disabled] [size=4M]

Region 3: Memory at (32-bit, prefetchable) [disabled]

Region 4: Memory at (32-bit, prefetchable) [disabled]

Region 5: Memory at (32-bit, prefetchable) [disabled]

I then tried to snatch 10M from non-prefetchable to prefetchable, and then my non-prefetchable failed to appear at all. I think this is because my PCIE to PCI bridge allocated 1M non-prefetchable for itself. But I don't know why my 64K non-prefetchable didn't get allocated or why these regions weren't displayed using the "lspci -vvv" command.

ranges = < 0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0xC2000000 0 0x01000000 0x01500000 0 0x00A00000 /* 10MB prefetchable memory */

0x82000000 0 0x02000000 0x01000000 0 0x00500000>; /* 5MB non-prefetchable memory */

lspci -vvv

Region 0: Memory at (32-bit, prefetchable) [disabled]

Region 3: Memory at (32-bit, prefetchable) [disabled]

Region 4: Memory at (32-bit, prefetchable) [disabled]

Region 5: Memory at (32-bit, prefetchable) [disabled]

I then reordered the memory so the 8M was located on a 8M boundary and my non-prefetchable failed to appear at all.

ranges = < 0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x01800000 0 0x00700000 /* 7MB non-prefetchable memory */

0xC2000000 0 0x02000000 0x01000000 0 0x00800000>; /* 8MB prefetchable memory */

lspci -vvv

Region 0: Memory at (32-bit, prefetchable) [disabled]

Region 3: Memory at (32-bit, prefetchable) [disabled]

Region 4: Memory at (32-bit, prefetchable) [disabled]

Region 5: Memory at (32-bit, prefetchable) [disabled]

So I'm guessing I'm still not understanding what's happening ;-(

Do I need to specify a prefetchable region or should the kernel allocate prefetchable and non-prefetchable from a single 15M memory range in the dtsi file?

cheers

Chris

MT · ‎01-07-2016

Hi Chris

Yes, 16MB is the physical limit. At onne time the total remap size cannot be bigger then 16MB.

I think your approach here was correct:

ranges = < 0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x01800000 0 0x00700000 /* 7MB non-prefetchable memory */

0xC2000000 0 0x02000000 0x01000000 0 0x00800000>; /* 8MB prefetch

but I wonder if the ranges are just over lapping. Can you try:

ranges = < 0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x01800000 0 0x00700000 /* 7MB non-prefetchable memory */

0xC2000000 0 0x02000000 0x01000000 0 0x007FFFF>; /* 8MB prefetch

BR

Mark

chrishossack · ‎01-13-2016

Hi Mark,

sorry for the delay, but I was re-writing my kernel driver to use only the two allocated non-prefechable regions and ignore the others.

I've tried with 0x007FFFFF but it still failed.

ranges = <0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */

0x82000000 0 0x01000000 0x01800000 0 0x00700000 /* 7MB non-prefetchable memory */

0xC2000000 0 0x02000000 0x01000000 0 0x007fffff>;/* 8MB prefetch */

lscpi -vvv

02:04.0 Signal processing controller: Texas Instruments TMS320C6424

Subsystem: Device 04b8:0136

Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Interrupt: pin A routed to IRQ 20

Region 0: Memory at <unassigned> (32-bit, prefetchable) [disabled]

Region 3: Memory at <unassigned> (32-bit, prefetchable) [disabled]

Region 4: Memory at <unassigned> (32-bit, prefetchable) [disabled]

Region 5: Memory at <unassigned> (32-bit, prefetchable) [disabled]

I think even if we get this working I will still have problems getting two of these cards working, since each card tries to allocate 4x8MB regions. This means the first card will consume at least 2x8MB and leave nothing for the second card.

At least I know now there is a hard 16MB limit that I can't exceed. It's a shame we couldn't get the prefechable regions working, but by re-working my driver I've managed to find a work around.

If you do have any more ideas about getting the prefechable regions working I'll be happy to try them out in case it helps somebody else.

cheers

Chris

Can not allocate PCIe region when size is 8M on Sabre platform

Can not allocate PCIe region when size is 8M on Sabre platform

i.MX6_All

Linux