PCIe and data cache on LS1046A

[mariapalmqvist]

Hi

In our system we are using an LS1046A connected to an FPGA by PCIe. The LS1046A is acting as RC, and the FPGA as EP. We are using the LS1046A-RDB and running bare metal.

Accesses from CPU to the FPGA is working correctly. It is also possible to access the CPU RAM from the FPGA, however when doing so we do not seem to get the data cache to update correctly.

The behaviour we see is:
After a write to an address in CPU RAM (at 00_8000_0000) from the FPGA, the FPGA can read back the value it wrote but the CPU will read the old value.
After invalidation of the cache (dc ivac), both the CPU and the FPGA reads the value written by the FPGA.

From reading the Reference Manual I find the following stated in the CCI Module section: "Snoop transactions are dependent on PCI Express protocol and controlled by PCI Express IATU registers. Refer PEX module internal configuration space for details."  But when reading the details on the IATU registers I can not find any way to control snooping.

I have set the Coherency Control Register 1 to 0x10000000, but that had no effect.

Settings of the NO_SNOOP bit in TLP from FPGA does not change the behaviour.

Do you have any suggestions on what to do or which settings to look at to get the cache updated without SW doing the invalidation?

[ufedor]

As long as any outbound MWr or MRd request TLP hits a defined iATU outbound region, the CTRL_1_OFF_Outbound [ATTR] bit 9 should be used for these TLP headers’ DW0, Byte 2, Bit 4, ATTR’s 'No Snoop' bit, when the PEX controller’s Device Control Register [NSE = Enable No Snoop] bit is left as default 1b.

[mariapalmqvist]

Hi Ufedor. My question only concerns inbound transactions (from FPGA to LS1046A) as the outbound transactions are already working, so the configuration of CTRL_1_OFF_Outbound will not matter in this case right?

Do you have any suggestions on how to make inbound transactions update the LS1046A cache?

[ufedor]

For Inbound request TLPs, our decision is to hide the ATTR bit in the CTRL_1_OFF_Inbound register. As such, we will not perform any checking for the ATTR attribute of the inbound TLP at the PEX controller level. Instead, we allow all the inbound request TLPs to come in without blocking them. It’s up to the host system’s cache coherency setup to handle the actual coherency by means of SMMU.