LS1012ARDB - PCIe IO Coherency

[bvermeul]

First some context:
I am connecting an AMD Radeon GPU to the LS1012ARDB's mini PCIe slot in order to get it working with linux.

The GPU is visible on the PCIe bus, enumerates, gets it's BAR's assigned, and is recognized by the driver.

However, when it tries to get the GPU to write to a specific memory location, it fails.

Reading the reference manual (LS1012ARM.pdf, page 1237, section 25.1) it says the following:

The PCI Express controller as instantiated on this chip does not support hardware coherency. All incoming PCI Express transactions are made non IO-coherent.

Does this mean that when a PCIe device writes something into main memory, the caches aren't updated? If not, what does it mean? ARM - Cache coherency fundamentals seems to imply this.

Regards,

Bas Vermeulen

[ufedor]

For inbound requests received by the PEX controller, it is downstream EP’s job to ensure the correct “No Snoop” bit setting for all the inbound request TLPs. The hardware coherency of all inbound requests is managed by SMMU. (Please consider that LS1012A does not possess SMMU).

The default SMMU driver in Linux depends on hardware for coherent signals. The SMMU is not configured to enforce coherency by default. When the link partner sends us memory requests targeting non-CCSR space (above 0x0100_0000) with TLP header’s NO_SNOOP = 0, we will treat the transactions as hardware coherency required.  

[shuangjunzhu]

Hi Ufedor,

Could you please point out how I could change the behavior of "default SMMU driver" to enforce coherency? Because we could not know whether the partner's request's NO_SNOOP is zero, but we want to make it always coherency?

Thanks,

Jeff