PCIe Peer(SSD) to Peer(FPGA) Communication Error

[cezhu]

We make a T2080 custom board refer to T2080RDB-PC, we use PEX4 in RC mode and we have a PLX PCIe Switch PEX8764 in board, follow is our pcie subsystem diagram:

root@t2080rdb:~# lspci -tv
-[0000:00]---00.0-[01-0f]----00.0-[02-0f]--+-01.0-[03]--
+-02.0-[04]--
+-03.0-[05]--
+-04.0-[06]----00.0 Device 0731:8000
+-05.0-[07]----00.0 Device 0731:8000
+-08.0-[08]--
+-09.0-[09]----00.0 Samsung Electronics Co Ltd Device a808
+-0a.0-[0a]----00.0 Samsung Electronics Co Ltd Device a808
+-0b.0-[0b]--
+-0c.0-[0c]--
+-0d.0-[0d]--
+-0e.0-[0e]----00.0 Samsung Electronics Co Ltd Device a804
\-0f.0-[0f]----00.0 Samsung Electronics Co Ltd Device a804

Device 0731:8000 is Xilinx Vertex7 FPGA, which has a 64MB BAR0, the other four devices is Samsung NVMe SSD, follow are our system memory space allocation: 

c40000000-c4fffffff : /pcie@ffe270000 c40000000-c4fffffff : PCI Bus 0000:01
c40000000-c407fffff : PCI Bus 0000:02
c40000000-c401fffff : PCI Bus 0000:03
c40200000-c403fffff : PCI Bus 0000:07
c40400000-c405fffff : PCI Bus 0000:09
c40600000-c407fffff : PCI Bus 0000:0d
c41000000-c4103ffff : 0000:01:00.0
c41100000-c463fffff : PCI Bus 0000:02
c41100000-c43ffffff : PCI Bus 0000:06
c42000000-c43ffffff : 0000:06:00.0
c42000000-c43ffffff : pcie_ep
c44000000-c45ffffff : PCI Bus 0000:07
c44000000-c45ffffff : 0000:07:00.0
c44000000-c45ffffff : pcie_ep
c46000000-c460fffff : PCI Bus 0000:09
c46000000-c46003fff : 0000:09:00.0
c46000000-c46003fff : nvme
c46100000-c461fffff : PCI Bus 0000:0a
c46100000-c46103fff : 0000:0a:00.0
c46100000-c46103fff : nvme
c46200000-c462fffff : PCI Bus 0000:0e
c46200000-c46203fff : 0000:0e:00.0
c46200000-c46203fff : nvme
c46300000-c463fffff : PCI Bus 0000:0f
c46300000-c46303fff : 0000:0f:00.0
c46300000-c46303fff : nvme

now we want to put Xilinx Vertex7 FPGA BAR0's data into SSD, BAR0's physical address is 0xc42000000, so we fire NVMe DMA request to read 0xc42000000, we got follow message in T2080 console:

PCIe error(s) detected
PCIe ERR_DR register: 0x80100000
PCIe ERR_CAP_STAT register: 0x80000001
PCIe ERR_CAP_R0 register: 0x00000800
PCIe ERR_CAP_R1 register: 0x00000000
PCIe ERR_CAP_R2 register: 0x00000000
PCIe ERR_CAP_R3 register: 0x00000000

When we write 0xc42000000, we got follow message in T2080 console:

PCIe error(s) detected
nvme 0000:09:00.0: async event result 00010300
PCIe ERR_DR register: 0x80100000
PCIe ERR_CAP_STAT register: 0x80000001
PCIe ERR_CAP_R0 register: 0x00000800
PCIe ERR_CAP_R1 register: 0x00000000
PCIe ERR_CAP_R2 register: 0x00000000
PCIe ERR_CAP_R3 register: 0x00000000
nvme 0000:09:00.0: Failed status: 3, reset controller
nvme 0000:09:00.0: Cancelling I/O 4 QID 2

Then we check the T2080RM section 20.4.36.4, we got:

it seems like the PCIe P2P Communication is failed, this is our question.

[yipingwang]

Please refer to "PCI Express error detect register (ERR_DR)", "PCI Express error capture status register (ERR_CAP_STAT)" and "PCI Express error capture register n (ERR_CAP_R0)" in T2080 Reference Manual.

The PCI Express error detect register contains error status bits that are detected by hardware, it is 0x80100000 indicates that Multiple errors were detected and A no-map transaction was detected in RC mode.

PCI Express no map. An inbound transaction that is not mapped to any inbound windows was detected.
In RC mode, a posted transaction will be dropped silently and this bit will be set. A nonposted transaction
will return a completion without data (Cpl) packet with a UR completion status to the requester and this bit
is set. For EP mode, a Cpl packet with a UR completion status is sent back to the requester but does not
set this bit.This bit is also set when an IO Write or IO Read transaction is received in RC mode and the
transaction is not hitting into IO Base/Limit registers.

Thanks,

Yiping