E9171 AMDPU with T1040 RDB Board

[Ganesh3955]

I Am using E9171 Amdgpu with T1040 NXP Board, here does this GPU supports per to peer data transfer using amdgpu.? this NXP is connected to FPGA and GPU via PCIe switch. this GPU should be able able to get data directly from FPGA using QDMA driver and also  does this NXP board supports direct peer to peer feature using AMDGPU driver?

[yipingwang]

Do not assume that E9171 + AMDGPU on a T1040 platform supports FPGA→GPU PCIe P2P DMA. Based on currently available AMDGPU information, direct FPGA-to-AMDGPU P2P is not generally supported as a standard AMDGPU feature in the same way that NVIDIA GPUDirect RDMA is.

1. Does AMDGPU support PCIe Peer-to-Peer (P2P)?

AMDGPU does contain Linux P2P infrastructure support (PCI_P2PDMA) and AMD KFD has an HSA_AMD_P2P option, but this support is primarily documented for:

• AMD GPU ↔ AMD GPU communication
• ROCm/HSA compute environments
• Platforms where the GPU exposes a large BAR and the platform/chipset allows PCIe P2P routing

The Linux Kconfig description explicitly mentions P2P communication between AMD GPUs.

2. FPGA → AMD GPU direct DMA?

AMD engineers have publicly stated that:

achieving P2P between Xilinx FPGA and AMD GPU is currently not directly supported

and suggested a host-memory registration workaround instead of true device-to-device PCIe DMA.

Therefore:

Path	Status
AMD GPU ↔ AMD GPU	Supported on specific ROCm platforms
FPGA ↔ AMD GPU direct PCIe DMA	Not generally supported by AMDGPU
FPGA → Host DDR → GPU	Supported
FPGA P2P buffer mapped into host memory and registered with GPU

3. Does T1040 support PCIe P2P?

From the T1040 side, PCIe hardware itself can forward Memory Read/Write TLPs between endpoints through a switch if:

• the PCIe switch allows P2P routing,
• ACS redirect is disabled (depends on switch),
• address translation is configured correctly.

PCIe as a protocol does not prevent endpoint-to-endpoint transfers. However, T1040/NXP software does not automatically provide AMDGPU-FPGA P2P support. The critical question is whether:

1. AMDGPU exports GPU memory for third-party DMA access.
2. FPGA QDMA can obtain GPU BAR/VRAM physical addresses.
3. Linux IOMMU/P2PDMA path accepts the transaction.

The AMDGPU limitation is typically the blocking factor rather than the T1040 PCIe controller itself.

4. What is likely to work on your system?

Current topology:

          PCIe Switch

           /                     \

FPGA(QDMA)    E9171 GPU

           \                       /

             T1040 RC

Most likely supported flow:

FPGA ---> DDR (T1040 memory)

|

v

AMDGPU DMA

|

VRAM

Not guaranteed to work:

FPGA(QDMA) ---> GPU VRAM

because AMDGPU generally does not expose a GPUDirect-RDMA-like interface for arbitrary FPGA devices.