AnsweredAssumed Answered

PCIe Peer(SSD) to Peer(FPGA) Communication Error

Question asked by Ce Zhu on Apr 28, 2019
Latest reply on Jul 7, 2019 by Yiping Wang

We make a T2080 custom board refer to T2080RDB-PC, we use PEX4 in RC mode and we have a PLX PCIe Switch PEX8764 in board, follow is our pcie subsystem diagram:

root@t2080rdb:~# lspci -tv
-[0000:00]---00.0-[01-0f]----00.0-[02-0f]--+-01.0-[03]--
+-02.0-[04]--
+-03.0-[05]--
+-04.0-[06]----00.0 Device 0731:8000
+-05.0-[07]----00.0 Device 0731:8000
+-08.0-[08]--
+-09.0-[09]----00.0 Samsung Electronics Co Ltd Device a808
+-0a.0-[0a]----00.0 Samsung Electronics Co Ltd Device a808
+-0b.0-[0b]--
+-0c.0-[0c]--
+-0d.0-[0d]--
+-0e.0-[0e]----00.0 Samsung Electronics Co Ltd Device a804
\-0f.0-[0f]----00.0 Samsung Electronics Co Ltd Device a804

Device 0731:8000 is Xilinx Vertex7 FPGA, which has a 64MB BAR0, the other four devices is Samsung NVMe SSD, follow are our system memory space allocation: 

c40000000-c4fffffff : /pcie@ffe270000 c40000000-c4fffffff : PCI Bus 0000:01
c40000000-c407fffff : PCI Bus 0000:02
c40000000-c401fffff : PCI Bus 0000:03
c40200000-c403fffff : PCI Bus 0000:07
c40400000-c405fffff : PCI Bus 0000:09
c40600000-c407fffff : PCI Bus 0000:0d
c41000000-c4103ffff : 0000:01:00.0
c41100000-c463fffff : PCI Bus 0000:02
c41100000-c43ffffff : PCI Bus 0000:06
c42000000-c43ffffff : 0000:06:00.0
c42000000-c43ffffff : pcie_ep
c44000000-c45ffffff : PCI Bus 0000:07
c44000000-c45ffffff : 0000:07:00.0
c44000000-c45ffffff : pcie_ep
c46000000-c460fffff : PCI Bus 0000:09
c46000000-c46003fff : 0000:09:00.0
c46000000-c46003fff : nvme
c46100000-c461fffff : PCI Bus 0000:0a
c46100000-c46103fff : 0000:0a:00.0
c46100000-c46103fff : nvme
c46200000-c462fffff : PCI Bus 0000:0e
c46200000-c46203fff : 0000:0e:00.0
c46200000-c46203fff : nvme
c46300000-c463fffff : PCI Bus 0000:0f
c46300000-c46303fff : 0000:0f:00.0
c46300000-c46303fff : nvme

now we want to put Xilinx Vertex7 FPGA BAR0's data into SSD, BAR0's physical address is 0xc42000000, so we fire NVMe DMA request to read 0xc42000000, we got follow message in T2080 console:

 

PCIe error(s) detected
PCIe ERR_DR register: 0x80100000
PCIe ERR_CAP_STAT register: 0x80000001
PCIe ERR_CAP_R0 register: 0x00000800
PCIe ERR_CAP_R1 register: 0x00000000
PCIe ERR_CAP_R2 register: 0x00000000
PCIe ERR_CAP_R3 register: 0x00000000


When we write 0xc42000000, we got follow message in T2080 console:

 

 

PCIe error(s) detected
nvme 0000:09:00.0: async event result 00010300
PCIe ERR_DR register: 0x80100000
PCIe ERR_CAP_STAT register: 0x80000001
PCIe ERR_CAP_R0 register: 0x00000800
PCIe ERR_CAP_R1 register: 0x00000000
PCIe ERR_CAP_R2 register: 0x00000000
PCIe ERR_CAP_R3 register: 0x00000000
nvme 0000:09:00.0: Failed status: 3, reset controller
nvme 0000:09:00.0: Cancelling I/O 4 QID 2


Then we check the T2080RM section 20.4.36.4, we got:

 

T2080

 

it seems like the PCIe P2P Communication is failed, this is our question.

Outcomes