PCIe Inbound Window Configuration on P1011

[ericfornstedt]

Hi,

We have a working PCIe configuration between out P1011 CPU and an FPGA, where P1011 is the Root Complex and FPGA is the Endpoint. One outbound window is defined and working properly when P1011 is the initiator. But, we are having a problem with the inbound window. A detailed description of relevant registers for the working outbound configuration looks like this:

PEX1_PEXOTAR1	0x000A0000	Translation address 0xA0000000
PEX1_PEXOWBAR1	0x000A0000	Base Address 0xA0000000
PEX1_PEXOWAR1	0x80044012	Enable, Default order, Snoopable, TC0, Mem Read, Mem write, 512 kB Size

LAW_LAWBAR0	0x000A0000	Local Access Window 0xA0000000
LAW_LAWAR0	0x8020001C	Enable, Target PEX1, 512 MB Size
L2MMU_CAM12	0xA0007FCA1C0800008000000080000001	A window at 0x80000000 of size 1 GB

Bus 0, Device 0, Fuction 0 - Configuration space 

0x04	0x00100106	Command Status: Capabilities List - Command Register: SERR, Bus Master, Memory Space
0x08	0x0B200011	Class Code: Processor, PowerPC, RC mode. Revision: 0x11
0x10	0xFFF00000	PEXCSRBAR: The fixed 1 MB window
0x18	0x00010100	Sec latency: 0, Subordinate bus: 1, Secondary bus 1, Primary bus 0

Bus 1, Device 0, Fuction 0 - Configuration space 

0x04	0x00100106	Command Status: Capabilities List - Command Register: SERR, Bus Master, Memory Space
0x10	0xA0000000	Base address 0xA00000000

As far as we can see this works fine. Our problem starts when we try to define an inbound window to enable the FPGA to be initiator of messages to read/write directly into the DDR memory connected to P1011. The approach we have tried looks like this:

PEX1_PEXITAR1	0x00000030	Let Endpoint read from address 0x00003000 (DDR). There is also a TBL entry for 0x00003000 of size 16 MB.
PEX1_PEXIWBAR1	0x000A0000	Base address 0xA0000000 (we have also tried 0x00000000)
PEX1_PEXIWAR1	0xA0F4401B	Enable, Prefetchable, Local Mem Space, 1 MB Size

Bus 0, Device 0, Fuction 0 - Configuration space additions 

0x24

0xC0010001

Prefetchable base 0x00000000, limit 0xC0000000, type 64-bit (also tried with default limit 0x00000000)

What happens is that P1011 receives a memory read request which seems valid according to what we can see, but for some reason it is not accepted and an "Unsupported Request" error is raised.

Bus 0, Device 0, Fuction 0 - Configuration space

0x11C	0x00000001	PCI Express Header Log Reg 1: Mem read request from Endpoint
0x120	0x0100000F	PCI Express Header Log Reg 2: Requester 0x0100 (bus 1, device 0, function 0), 1st BE 0xF
0x124	0x000E0000	PCI Express Header Log Reg 3: Address requested from Endpoint
0x04	0x40100106	Command Status: Signaled System Error, Capabilities List - Command Register: SERR, Bus Master, Memory Space
0x104	0x00100000	Uncorrectable Error Status Register: Unsupported request error
0x118	0x000000B4	PCIe Advanced Error Capabilities and Ctrl Reg: First_Error_Pointer: 0x14, ECRC generation capable: 1, ECRC checking capable: 1
0x130	0x0000002C	Root Error Status Register: NFEMR, MEFNFR, EFNFR

Bus 1, Device 0, Fuction 0 - Configuration space

0x04

0x20100106

Command Status: Received Master Abort, Capabilities List - Command Register: SERR, Bus Master, Memory Space

We have tried to investigate why this happens without success and would appreciate any help to guide us where to dig further.

BR
/ Eric

[ericfornstedt]

The FPGA (EP) internal address range 0x000E0000-0x001E0000 is mapped to PCIe, and due to a technical limitation in the FPGA at the moment this is the address range it will read over PCIe instead of the more natural 0x00000000-0x00100000. For now, we will have to live with that.

Just for an overview, here is the current MMU setup we have from RTEMS. The marked one is the one we try to access from EP in this initial step.

L2MMU_CAM0  0x300000c0040800000001000000010001
L2MMU_CAM1  0x300000c0040800000002000000020001
L2MMU_CAM2  0x300000c0040800000003000000030001
L2MMU_CAM3  0x300000c0100800000100000001000001
L2MMU_CAM4  0x70000fc4180800000200000002000001
L2MMU_CAM5  0x70000fc4180800000300000003000001
L2MMU_CAM6  0x80001fc4180800000400000004000001
L2MMU_CAM7  0x80001fc4180800000800000008000001
L2MMU_CAM8  0x80001fc4180800000c0000000c000001
L2MMU_CAM9  0x70000fc4180800001000000010000001
L2MMU_CAM10 0x500003ca18080000ffe00000ffe00001
L2MMU_CAM12 0xa0007fca1c0800008000000080000001

LAW_LAWBAR0  0x000a0000
LAW_LAWAR0   0x8020001c
LAW_LAWBAR11 0x00000000
LAW_LAWAR11  0x80f0001d

The Inbound Windows registers looks like this:

PEX1_PEXITAR1  0x00030000 - TA[12:31] corresponds to internal address bits [4:23] (reference manual 14.6.19)
PEX1_PEXIWBAR1 0x00000000 
PEX1_PEXIWAR1  0xa0f44015

As I understand it, these settings will make a read request for address 0x00000000 (PEX1WBAR1) to be translated to physical address 0x03000000 (PEXITAR1:TA << 8). That means the physical address of the request EP tries to read in our case would be 0x030E0000, which should be a valid address.

NOTE: Inbound windwow register PEX1_PEXITAR1 is slightly different from original post, now pointing at another physical address described above. The logged error is however still identical.

[ufedor]

> PEX1_PEXITAR1  0x00030000 

This corresponds to internal platform address 0x30000000.

[ericfornstedt]

> This corresponds to internal platform address 0x30000000.

How is that possible when PEX1_PEXITAR1:TA[12:31] corresponds to internal address bits [4:23], according to refernece manual?

Anyway, the error we get is the same even if we change PEX1_PEXITAR1 to 0x00003000, which according to your comment above would correspond to internal platform address 0x03000000.

[ufedor · Accepted solution]

> 0x20: 00000000 00010001 00000000 00000000

Memory_Base_Register=0 and Memory_Limit_Register=0 define 1MB memory range starting from address 0 - refer to the P1020RM, 14.9.8 PCI Express Memory Base Register and 14.9.9 PCI Express Memory Limit Register.

This effectively exclude the 1MB area from the defined Inbound Window.

Check this by accessing addresses above 0x100000 instead of 0xE0000.

[ericfornstedt]

Thank you ufedor! That solved our problem.

Due to a bug in the FPGA I couldn't access any address above 0xF0000 (0xE0000 + 64 KiB), but as a test I moved the Memory Base/Limit and the Prefetchable Memory Base/Limit away from 0x00000000 by using:

0x20: 00300020 00510041 00000000 00000000

and that enables us to both read and write from 0x000E0000, which translates to internal physical address 0x030E0000.

It's not a final solution, but the concept works and from here we can clean up the configuration.

[ufedor]

> PCIe Header Log Reg 3: 0x000E0000

Why read address is 0x000E0000?

What are values of the PCIe1 Inbound Windows registers?

[ufedor]

In Power Architecture it is required to set base address of a window aligned to its size.

In other words: Base Address = N * Size (N- integer).

In the described setup the PEX1_PEXIWAR1 = 0xA0F4401B - i.e. window size is 256 MB (IWS = 0x1B)

16 MB corresponds to IWS = 0x17

1 MB - to 0x13

Also, it is written:

> PEX1_PEXITAR1    0x00000030    Let Endpoint read from address 0x00003000 (DDR).

but the PEX1_PEXITAR1 = 0x00000030 corresponds to 0x00030000 and could be valid for IWS = 0xF (64 kB) or smaller.

[ericfornstedt]

Hi,

Yes, the IWS = 0x1B was a mistake during debugging, I have set it to 0xF instead for now to make PEX1_PEXITAR1 easier to choose.

I'm not sure how PEX1_PEXITAR1 = 0x00000030 can correspond to 0x00030000 as PEX1_PEXITAR1[12:31] corresponds to platform bits [4:23] (I assume 4 is MSB) according to the reference manual. For PEX1_PEXOTARn the TA field is [31:12] (31 MSB) and there I agree with you about the address range. But anyway, I made sure both 0x00003000 and 0x00030000 are valid ranges in the TLBs to read from, and both are also multiples of IWS = 0xF.

Still no success though.

There are a some things we are not quite sure about and where we can use some guidance:

• The Prefetchable Memory Base/Limit registers in the Type 1 header for the RC, do I need to handle those or is it ok to leave them to their default values for this scenario?
• The PEXCSRBAR in Type 0 header for EP is only for handling outbound accesses as I understood. Its base address (0xA0000000 in our case), is not connected to any inbound access since all inbound windows are handled through the PEXn_PEXIxxxx registers. Is that correct?
  
  
• In our case, the EP will access the range of maximum 1 MB from 0x000E0000 (See e.g PCI Express Header Log Reg 3 from original post) and nothing else. Does that mean we can set PEX1_PEXIWBAR1 to address 0x000E0000 and translate it to where we want it to access with PEX1_PEXITAR1 (IWS needs to be 0x13 for full range of course)?
  
  
• We have a LAWBAR for PEX1 for the outbound address range (0xA0000000), but do we also need a PEX1 LAWBAR for inbound address range, e.i 0x000E0000 plus 1 MB?
• As I understood it we need a TLB entry for the inbound translated DDR address range to make it accessable for PEX1. Do we also need a TLB entry fo the inbound window base adress (0x000E0000) ?

Do you see any other areas we can look into?

Thank you for your support
/ Eric

[ufedor]

> Still no success though.

To which exactly configuration this corresponds?

[ericfornstedt]

The same configuration as in the original post, but with IWS = 0xF. PEX1_PEXITAR1 = 0x00000030 according to original post should then be a multiple of IWS.

BR

/ Eric

[ufedor]

Just a note (MMU is not involved in the investigated PCIe operation):

> There is also a TBL entry for 0x00003000 of size 16 MB.

This is also a window with incorrect base address.

Please ensure that this is the only TLB for the memory region - note that hit to multiple matching TLB entries is considered a programming error (PowerPC e500 Core Family Reference Manual, 12.2.3 Checking for TLB Entry Hit).

Please provide:

1) values of the P1011 PCI configuration header 0x00-0x3F

2) PCI Express Header Log values after unsuccessful EP read attempt for the current RC configuration

[ericfornstedt]

Hi,

Here is th the requested info:

P1011 PCI Configuration space 0x00-0x3F (before read attempt from EP)

0x00: 01091957 00100106 0B200011 00010000
0x10: FFF00000 00000000 00010100 20000000
0x20: 00000000 00010001 00000000 00000000
0x30: 00000000 00000044 00000000 00000000

P1011 PCI Configuration space 0x00-0x3F and PCI Log headers (after read attempt from EP)

0x00: 01091957 40100106 0B200011 00010000
0x10: FFF00000 00000000 00010100 20000000
0x20: 00000000 00010001 00000000 00000000
0x30: 00000000 00000044 00000000 00000000

PCIe Header Log Reg 1: 0x00000001
PCIe Header Log Reg 2: 0x0100000F
PCIe Header Log Reg 3: 0x000E0000
PCIe Header Log Reg 4: 0x76A2A06C

The MMU is automatically setup by RTEMS in our case and should be corrrect. What we try to do is to map an inbound PEX1 window into one of the already existing memory windows of the DDR memory.