PCIe: RC cannot write into EP

[elijahbrown]

The IMX6 is configured as a PCIe EP, which is connected to a RC (not an IMX6).  I've got two iATU configurations, as follows:

    pcie_map_inbound_addr(PCIE_IATU_VIEWPORT_0,

                                            TLP_TYPE_MemRdWr,

                                            (uint32_t)endpointBuffer,

                                            0x90500000,

                                            SZ_64K);    

    pcie_map_outbound(PCIE_IATU_VIEWPORT_1,

                                          TLP_TYPE_MemRdWr,

                                           PCIE_ARB_BASE_ADDR,

                                           0x310000,

                                           SZ_64K);

Here's what the inbound mapping function does:

uint32_t pcie_map_inbound_addr(uint32_t viewport, uint32_t tlp_type,

                               uint32_t addr_base_cpu_side, uint32_t addr_base_pcie_side, uint32_t size)

{

    // configure as an inbound region

    HW_PCIE_PL_IATUVR_WR((viewport & 0x0F) | (1 << 31));

    // configure region's base and limit address

    HW_PCIE_PL_IATURLBA_WR(addr_base_pcie_side);

    HW_PCIE_PL_IATURUBA_WR(0);

    HW_PCIE_PL_IATURLA_WR(addr_base_pcie_side + size - 1);

    // configure target address

    HW_PCIE_PL_IATURUTA_WR(0);

    HW_PCIE_PL_IATURLTA_WR(addr_base_cpu_side);

    // configure TLP type

    HW_PCIE_PL_IATURC1_WR(tlp_type & 0x0F);

    // enable region

    HW_PCIE_PL_IATURC2_WR(((unsigned int)(1 << 31)));

    return addr_base_cpu_side;

}

Bus mastering is configured in the EP.  The outbound transactions (EP to RC) work fine, I can read and write memory in the RC by reading or writing PCIE_ARB_BASE_ADDR.  But the other way around, inbound transactions (RC to EP) doesn't work.  The RC is sending TLPs with address 0x90500000, which I want to map into the IMX's DRAM, specifically a 64K buffer named endpointBuffer.  This buffer is 1M aligned to meet the iATU requirements of 64k aligning and the MMU requirements of 1M aligning so I can turn caching off for it... The RC sets BAR0 to 0x90500000 and BAR2 to 0x310000.  For whatever reason I can't make the BAR1 mask anything non-zero so I'm using BAR2 instead. 

What am I missing?  The fact that outbound transactions are working makes me think this has to be close, and just an error in the mapping.  I've tried setting up the inbound mapping to do BAR matching and address matching, neither seems to work.

[elijahbrown]

Yes, but the platform SDK sets up the PCIe core as an RC not as an EP, so I've modified it from there.  I've attached my code to this post, hopefully we can be on the same page this way.

Just to reiterate, the IMX6 configured as an EP (using this code) is able to write into the RC's memory.  But when the RC tries to do a memory read/write into the IMX6, it fails.  The RC, an intel board, is known to fully work with an FPGA device so we're pretty sure it's configured correctly.  I have tried setting the iATU up to do BAR matching for inbound TLPs but that didn't work, so I also tried setting up inbound address mapping - no luck there either.  But that's why you'll see the hardcoded address for the inbound mapping setup...

[gfine]

elijahbrown​

Did Hongxing's reply help?

BR,

Glen

[elijahbrown]

Unfortunately not really, we are not using linux and they do not describe any actual configuration of the PCIe EP.  And the linux patches do not setup any inbound mapping on the EP, they assume the RC will send TLPs with the correct addresses.  This will not work for us, we need to remap the TLPs to the ARM's native address space.  Besides it is kind of silly to require the RC to know the EPs memory map, that's the whole point of setting up the BARs in the first place... the iATU should remap inbound TLPs based on the BAR. 

Probably what needs to happen is you guys should take my bare metal project and try to setup the inbound mapping so you can duplicate my issue as it does not appear any testing has been done with inbound mapping setup.  Please advise if you're willing to do this and I'll take some time to make a stripped down project that demonstrates the issue.