Unable to clear PCIe BARs 2 and 3 on i.MX6

[ronsandel]

I am using the Boundary Devices Sabre Lite for i.MX6 with VxWorks and the BSP for that board it comes with.  It doesn't support PCIe so I am putting together my own driver.  The post I am using as guidance is: https://community.freescale.com/docs/DOC-95014

Basically I set all the registers and PCIe is alive and the link is good and the BAR registers show up like this:

0x0000000c

0x00000000

0x00000008

0x00000001

0x00000000

0x00000000

And when my root complex assigns memory to the them they show up like this:

0x8000000c

0x00000000

0x80100008

0xe8000001

0x00000000

0x00000000

I only need 1MB of memory in a single BAR, so similar to 0001-ENGR00268112-pcie-emaluate-the-pcie-ep-as-ram-device.patch I do this:

*(volatile unsigned int *)(0x01ffc000 + 0x10) = 0;

*(volatile unsigned int *)(0x01ffc000 + (1 << 12) + 0x10) = 0x100000 - 1;

*(volatile unsigned int *)(0x01ffc000 + 0x14) = 0;

*(volatile unsigned int *)(0x01ffc000 + (1 << 12) + 0x14) = 0;

*(volatile unsigned int *)(0x01ffc000 + 0x18) = 0;

*(volatile unsigned int *)(0x01ffc000 + (1 << 12) + 0x18) = 0;

*(volatile unsigned int *)(0x01ffc000 + 0x1c) = 0;

*(volatile unsigned int *)(0x01ffc000 + (1 << 12) + 0x1c) = 0;

*(volatile unsigned int *)(0x01ffc000 + 0x20) = 0;

*(volatile unsigned int *)(0x01ffc000 + (1 << 12) + 0x20) = 0;

*(volatile unsigned int *)(0x01ffc000 + 0x24) = 0;

*(volatile unsigned int *)(0x01ffc000 + (1 << 12) + 0x24) = 0;

And then the registers show up as this:

0x00000000

0x00000000

0x00000008

0x00000000

0x00000000

So it looks like my write of 0 to BAR0 and BAR3 worked but my write of 0 to BAR2 didn't.

And when my root complex assigns memory they show up like this:

0xa0100000

0x00000000

0x00000008

0xa0200000

0x00000000

0x00000000

So it looks like the 8 in BAR2 isn't getting cleared, although it's not a big deal because memory isn't actually being assigned to it.  But the BAR3 is still getting assigned.  I noticed a comment in the patch says:

/*

* 32bit prefetchable 1M bytes memory on bar3

* FIXME BAR MASK3 is not changable, the size

* is fixed to 256 bytes.

*/

Is that some kind of hardcoded limitation in the processor?

[richard_zhu]

Hi Ron Sandel:

First of all, there is one IC design limitation but I don't know the reason.

The size of BAR3 specified is fixed-256 bytes, whatever the size configured into the BAR3_MASK register.

here is the log derived from RC side when the BAR3 is configured to be prefetchable 1Mbytes size memory:

...

pci 0000:01:00.0: BAR 2: set to [io  0x1e00000-0x1e00fff] (PCI address [0x1e00000-0x1e00fff])

pci 0000:01:00.0: BAR 3: assigned [mem 0x01910000-0x019100ff pref]

pci 0000:01:00.0: BAR 3: set to [mem 0x01910000-0x019100ff pref] (PCI address [0x1910000-0x19100ff])

Regarding to " https://community.freescale.com/docs/DOC-95014", the the BARs values in the PCIe EP

side are listed below when the BAR2 is configured as 4K bytes IO.

We can see that the value of the BAR2 can be changed/configured.

root@freescale ~$ ./memtool -32 01ffc000 40

Reading 0x40 count starting at address 0x01FFC000

0x01FFC000:  DEADBEAF 00100147 05000001 00000008

0x01FFC010:  01000000 00000000 01E00001 01910008

When cleared the configurations of BAR2/BAR2_MASK, RC wouldn't allocate the responded space, although the read-out value of BAR2 register

at EP side is none-zero.

Here is the log:

RC side:

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x017fffff]

pci 0000:00:00.0: BAR 0: assigned [mem 0x01800000-0x018fffff 64bit pref]

pci 0000:00:00.0: BAR 0: set to [mem 0x01800000-0x018fffff 64bit pref] (PCI address [0x1800000-0x18fffff])

pci 0000:00:00.0: BAR 9: assigned [mem 0x01900000-0x019fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01a00000-0x01a0ffff pref]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]

pci 0000:01:00.0: BAR 0: set to [mem 0x01000000-0x017fffff] (PCI address [0x1000000-0x17fffff])

pci 0000:01:00.0: BAR 6: assigned [mem 0x01900000-0x0190ffff pref]

pci 0000:01:00.0: BAR 3: assigned [mem 0x01910000-0x019100ff pref]

pci 0000:01:00.0: BAR 3: set to [mem 0x01910000-0x019100ff pref] (PCI address [0x1910000-0x19100ff])

pci 0000:00:00.0: PCI bridge to [bus 01-01]

pci 0000:00:00.0:   bridge window [io  disabled]

pci 0000:00:00.0:   bridge window [mem 0x01000000-0x017fffff]

pci 0000:00:00.0:   bridge window [mem 0x01900000-0x019fffff pref]

EP BAR2 value:

root@freescale ~$ ./memtool -32 01ffc000 40

Reading 0x40 count starting at address 0x01FFC000

0x01FFC000:  DEADBEAF 00100146 05000001 00000008

0x01FFC010:  01000000 00000000 00000008 01910008

[ronsandel]

Okay so you did confirm that BAR3 is fixed and there is nothing that can be done about it.  In the documentation is called out this 'BAR3_MASK_WRITABLE_N' but it's not really explained what it is.  And what you are saying does match what I saw, where the master was not actually allocating memory on the BAR2.  So I suppose I can just accept this and deal with the extra pci space memory allocation.

Thanks,

[karina_valencia]

EricNelson could you help on this  question?

[karina_valencia]

EricNelson please  confirm if you are available  to  help  on this case.

[EricNelson]

Hi Karina,

Unfortunately, I don't have cycles to spare and I have limited expertise in this area (mostly involving the use of the Freescale Linux drivers).

TroyKisky has messed around with the interrupts in some of the older kernels.

[troy_kisky]

I have no experience with endpoints, but as a general

statement, if writes appear to not take, check to make

sure the appropriate clocks are enabled.

Troy

[ronsandel]

The peripheral works fine, so all the clocks are there, it's just that I can't clear these 2 particular BARs.

[karina_valencia]

richard.zhu can you  help on this case?

[richard_zhu]

Hi Karina:
Sure, I‘m checking it now, would post my comments a moment later.

Best Regards
Richard Zhu

[karina_valencia]

Ok thanks Eric.

Let me check internally.

[karina_valencia]

let me check with  Frank Li