PCIe RC configuration problem on t2080

[int0dh]

Hi All,

I am trying to configure PCIe RCs on t2080-based board. I have LaWs set up by u-boot as follows

target PCIe0 - 0xc:0x00000000, size 512M

target PCIe1 - 0xc:0x20000000, size 256M

target PCIe2 - 0xc:0x40000000, size 256M,

target PCIe3 - 0xc:0x60000000, size 256M

I leave LaWs as configured and inside OS I configure PCIe1 ATMUs as follows

PEXOTEAR2(PEX1) = 0xc20000000ULL >> 44 (aka 0)

PEXOTAR2(PEX1) = 0xc20000000ULL >> 12

PEXOWBAR2(PEX1) = PCI1_MEM_BUS (0xe0000000) >> 4

PEXOWAR2(PEX1) = 0x80000000 | 0x4 << 16 | 0x4 << 12 | 0x1c

PEXITAR1(PEX1) = 0x0

PEXIWBAR1(PEX1) = 0x0

PEXIWAR1(PEX1) = 0x80000000 | 0xf << 20 (target - DRAM) | 0x5 << 16 | 0x5 << 12 | 0x1F;

Then I assign bus address 0xe000000 to PCIe device`s BAR0, there are no any bridges between device and RC.

Then I map BAR0 into OS virtual space (0xe000000 bus address gets mapped to 0xc2000000ULL host physical address through outbound ATMU which in turn gets mapped into OS virtual address space)

Then I'm trying to access BAR0 from device driver (does not matter read or write) and immediately receive Machine Check exception.  Looks like something was missed in the RC configuration, but I do not see what exactly, does anyone have an idea what may be wrong ?

[int0dh]

T2080RM.pdf, 20.4.20.2 

>>

20.4.20 PCI Express outbound translation address register n (PFa_PEXOTARb)

The PCI Express outbound translation address registers select the starting addresses in the system address space for window hits within the PCI Express outbound address translation windows. The new translated address is created by concatenating the transaction offset to this translation address. PEXOTAR0 exists in RC mode or for PF0- only in EP mode. PEXOTAR1-4 are unique per PF.

>>

>>

20.4.23 PCI Express outbound window base address register (PFa_PEXOWBARb)

20.4.23.2 Function

The PCI Express outbound window base address registers select the base address for the windows which are translated to the external address space. Addresses for outbound transactions are compared to these windows. If such a transaction does not fall within one of these spaces the transaction is forwarded through a default register set. These registers are unique per PF.

>>>

[ufedor]

I agree that 20.4.20 has to be corrected.

The 20.4.23 is OK.

You wrote:

> PCIe1 ATMUs after OS started (reversed, PCIe outbound works)

Is my understanding correct that the issue is only with inbound transactions?

Please describe the attempted inbound transaction.

Is PAMU configured/used by the OS?

[int0dh]

We have e1000 device 0x8086:0x105E attached to PCIe1 RC. When we configure outbound ATMUs in reverse order,

we are able to access device registers through BAR0. if_em driver successfully establishes the ethernet link at correct speed and duplex, but there are no DMA packet transfers device -> to/from host memory although ethernet traffic present on the wire. Given that transaction initiator is e1000 device behind PCIe1 RC we made conclusion that inbound PCIe transactions do not work and this is why we see no DMA exchanges. We have the same problem (no DMA/no bus-mastering) for any device we attach to PCIe. Neither u-boot nor our OS configure PAMU, so it shall be in pass-through mode. The same if_em driver works pretty well with the same device on another platform.

[int0dh]

In addition to that inbound problem we see no legacy INTx interrupts coming. Our OS uses legacy INTx for PCIe devices. MPIC is configured to let them go

[ufedor]

Do you see an error detected by the PCIe controller in case of an inbound transaction?

[int0dh]

Nope. Do you have any special registers we could take look at ?

[ufedor]

RM, 20.5.6 Advanced error reporting capability structure

[int0dh]

Do not see anything bad there

[ufedor]

Please use a PCIe analyzer to capture the problem inbound transaction.

[int0dh]

LaWs

Local Access Window Configuration
LAWBARH00: 0x0000000f LAWBARL00: 0xe0000000 LAWAR00: 0x81f0001b
(EN: 1 TGT: 0x1f SIZE: 256 MiB)
LAWBARH01: 0x0000000f LAWBARL01: 0xf4000000 LAWAR01: 0x81800018
(EN: 1 TGT: 0x18 SIZE: 32 MiB)
LAWBARH02: 0x0000000f LAWBARL02: 0xf6000000 LAWAR02: 0x83c00018
(EN: 1 TGT: 0x3c SIZE: 32 MiB)
LAWBARH03: 0x0000000f LAWBARL03: 0xffdf0000 LAWAR03: 0x81f0000b
(EN: 1 TGT: 0x1f SIZE: 4 KiB)
LAWBARH04: 0x0000000f LAWBARL04: 0x00000000 LAWAR04: 0x81d00018
(EN: 1 TGT: 0x1d SIZE: 32 MiB)
LAWBARH05: 0x0000000f LAWBARL05: 0xff800000 LAWAR05: 0x81f00013
(EN: 1 TGT: 0x1f SIZE: 1 MiB)
LAWBARH06: 0x0000000c LAWBARL06: 0x00000000 LAWAR06: 0x8000001c
(EN: 1 TGT: 0x00 SIZE: 512 MiB)
LAWBARH07: 0x0000000f LAWBARL07: 0xf8000000 LAWAR07: 0x8000000f
(EN: 1 TGT: 0x00 SIZE: 64 KiB)
LAWBARH08: 0x0000000c LAWBARL08: 0x20000000 LAWAR08: 0x8010001b
(EN: 1 TGT: 0x01 SIZE: 256 MiB)
LAWBARH09: 0x0000000f LAWBARL09: 0xf8010000 LAWAR09: 0x8010000f
(EN: 1 TGT: 0x01 SIZE: 64 KiB)
LAWBARH10: 0x0000000c LAWBARL10: 0x40000000 LAWAR10: 0x8030001b
(EN: 1 TGT: 0x03 SIZE: 256 MiB)
LAWBARH11: 0x0000000f LAWBARL11: 0xf8030000 LAWAR11: 0x8030000f
(EN: 1 TGT: 0x03 SIZE: 64 KiB)

Our OS leaves them as they were programmed by u-boot

PCIe1 ATMUs before OS started

u-boot ATMUs for PCIe1 (PCIe2 in u-boot notation)

outbound

fe250c00: 00000000 00000000 00000000 00000000 ................
fe250c10: 80044027 00000000 00000000 00000000 ..@'............
fe250c20: 000e0000 00000000 00c20000 00000000 ................
fe250c30: 8004401b 00000000 00000000 00000000 ..@.............
fe250c40: 00000000 00000000 00ff8010 00000000 ................
fe250c50: 8008800f 00000000 00000000 00000000 ................
fe250c60: 00000000 00000000 00000000 00000000 ................
fe250c70: 00000000 00000000 00000000 00000000 ................
fe250c80: 00000000 00000000 00000000 00000000 ................
fe250c90: 00044027 00000000 00000000 00000000

inbound

fe250da0: 00000000 00000000 01000000 00000000 ................
fe250db0: a0f5501e 00000000 00000000 00000000 ..P.............
fe250dc0: 00000000 00000000 00000000 00000000 ................
fe250dd0: a0f5501e 00000000 00000000 00000000 ..P.............

Please pay atttention that PEXOTAR/PEXOTEAR and PEXOWBAR are reversed in  comparison

to said in documentation. PEXOWBAR holds system bus address 0xc20000000ULL >> 12

while PEXOTAR/PEXOTEAR have PCIe BUS address 0x:0xe0000000 >> 12. According to documentation PEXOTAR:PEXOTEAR shall have address of system bus window assigned to PCIe while PEXOWBAR shall have PCIe BUS address 

PCIe1 ATMUs after OS started (reversed, PCIe outbound works)

73250c00:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250c10:80 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250c20:00 0e 00 00 00 00 00 00:00 c2 00 00 00 00 00 00 ........ ........
73250c30:80 04 40 1b 00 00 00 00:00 00 00 00 00 00 00 00 ..@..... ........
73250c40:00 04 00 00 00 00 00 00:00 ff 80 10 00 00 00 00 ........ ........
73250c50:80 08 80 0f 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250c60:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250c70:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250c80:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250c90:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250ca0:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250cb0:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........

Only inbound window (target - DRAM, size 2G)

73250da0:00 00 00 00 00 00 00 00:01 00 00 00 00 00 00 00 ........ ........
73250db0:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250dc0:00 00 00 00 00 00 00 00:00 00 00 00 00 00 00 00 ........ ........
73250dd0:a0 f5 50 1e 00 00 00 00:00 00 00 00 00 00 00 00 ..P..... ........

[ufedor]

> According to documentation PEXOTAR:PEXOTEAR shall have address of system bus window assigned to PCIe

> while PEXOWBAR shall have PCIe BUS address

Where in the RM this is stated?

[int0dh]

Yes, I can confirm that all the configured windows have base addresses aligned to its size. We may get outbound transactions work when we swap PEXOWBAR2 and PEXOTAR2/PEXOTEAR2, so PEXOWBAR2 has system bus address (0xc:0x20000000 >> 12) while PEXOTAR2/PEXOTEAR2 have PCI bus address although this is against documentation. But we still have no luck with inbound transactions - PCIe device cannot do DMA transfers to host memory for example.

[ufedor]

Please provide values of all Inbound & Outbound ATMUs and LAWs registers.

[ufedor]

Please ensure that each configured window has base address aligned to its size - i.e. Base Address = N * Window Size where N is a whole number.