PCIe Endpoint to Endpoint data transfer

[TrinathK]

Hi,

I have a PCIe Rootcomplex (Custom LS1046A board configured as RC), Endpoint-1 (Custom LS1046A board configured as EP) and Endpoint-2 (Custom IMX8QM board configured as EP). Endpoints are connected to the Root complex through switch (PI7C9X2G1224GP).

I want to perform Endpoint-1 to Endpoint-2 data transfer. For that, i configured ATU registers of Endpoint-1 and Endpoint-2 as mentioned below.

Endpoint-1:

#BAR-0 region-0 as Inbound
0x0340_1010 = 0x00000FFF        -- BAR0 mask register
0x0340_0900 = 0x80000000         -- iATU Index register
0x0340_0908 = 0xC0000000        -- iATU Region Control 2 Register
0x0340_0918 = 0xE0000000        -- iATU Lower target address offset inbound
0x0340_091C = 0x00000000        -- iATU Upper target address offset inbound

#BAR-1 region-1 as Outbound
0x0340_1014 = 0x00000FFF        -- BAR1 mask register
0x0340_0900 = 0x00000001        -- iATU Index register
0x0340_0904 = 0x00000000        -- iATU Region Control 1 Register
0x0340_090C = 0xE1000000       -- iATU Lower base address offset outbound
0x0340_0910 = 0x00000000        -- iATU Upper base address offset outbound
0x0340_0914 = 0xE1000FFF       -- iATU Limit address offset outbound
0x0340_0908 = 0x80000000        -- iATU Region Control 2 Register
0x0340_0918 = 0x41800000        -- iATU Lower target address offset outbound
0x0340_091C = 0x00000040       -- iATU Upper target address offset outbound

#BAR-2,3 region-2 as Inbound
0x0340_1018 = 0x00000FFF        -- BAR2 mask register
0x0340_0900 = 0x80000002        -- iATU Index register
0x0340_0908 = 0xC0000200        -- iATU Region Control 2 Register
0x0340_0918 = 0x80000000        -- iATU Lower target address offset inbound
0x0340_091C = 0x00000008       -- iATU Upper target address offset inbound

#BAR-4,5 region-3 as Inbound
0x0340_1020 = 0x00000FFF       -- BAR4 mask register
0x0340_0900 = 0x80000003       -- iATU Index register
0x0340_0908 = 0xC0000400       -- iATU Region Control 2 Register
0x0340_0918 = 0x81000000       -- iATU Lower target address offset inbound
0x0340_091C = 0x00000008      -- iATU Upper target address offset inbound

Endpoint-2 :

0x5F00_1010 = 0x00000FFF          -- BAR0 mask register
0x5F00_0900 = 0x80000000          -- iATU Index register
0x5F00_0904 = 0x00000000          -- iATU Region Control 1 Register
0x5F00_0908 = 0xC0000000          -- iATU Region Control 2 Register
0x5F00_0918 = 0xA0000000          -- iATU Lower target address offset inbound
0x5F00_091C = 0x00000000          -- iATU Upper target address offset inbound

On Root complex side, i can see below information using "lspci -v" command.

00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c0 (rev 10) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 77
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
Memory behind bridge: 40000000-433fffff
Prefetchable memory behind bridge: 0000000043400000-00000000434fffff
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 PCI bridge: Pericom Semiconductor Device 1224 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 78
Memory at 4043000000 (32-bit, non-prefetchable) [size=128K]
Bus: primary=01, secondary=02, subordinate=07, sec-latency=0
Memory behind bridge: 40000000-42ffffff
Prefetchable memory behind bridge: 0000000043400000-00000000434fffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [148] Virtual Channel
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12
Kernel driver in use: pcieport

01:00.1 System peripheral: Pericom Semiconductor Device 1224
Subsystem: Pericom Semiconductor Device 1224
Flags: bus master, fast devsel, latency 0
Memory at 4043020000 (32-bit, non-prefetchable) [size=512]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [148] Virtual Channel
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12

02:01.0 PCI bridge: Pericom Semiconductor Device 1224 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 79
Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12
Kernel driver in use: pcieport

02:02.0 PCI bridge: Pericom Semiconductor Device 1224 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 80
Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
Memory behind bridge: 40000000-417fffff
Prefetchable memory behind bridge: 0000000043400000-00000000434fffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12
Kernel driver in use: pcieport

02:03.0 PCI bridge: Pericom Semiconductor Device 1224 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 81
Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
Memory behind bridge: 41800000-42ffffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12
Kernel driver in use: pcieport

02:04.0 PCI bridge: Pericom Semiconductor Device 1224 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 82
Bus: primary=02, secondary=06, subordinate=06, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12
Kernel driver in use: pcieport

02:05.0 PCI bridge: Pericom Semiconductor Device 1224 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 83
Bus: primary=02, secondary=07, subordinate=07, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: Pericom Semiconductor Device 1224
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [270] L1 PM Substates
Capabilities: [900] #12
Kernel driver in use: pcieport

04:00.0 Power PC: Freescale Semiconductor Inc Device 81c0 (rev 10)
Flags: bus master, fast devsel, latency 0, IRQ 84
Memory at 4041000000 (32-bit, non-prefetchable) [size=4K]
Memory at 4041001000 (32-bit, non-prefetchable) [size=4K]
Memory at 4043400000 (64-bit, prefetchable) [size=4K]
Memory at 4043401000 (64-bit, prefetchable) [size=4K]
Expansion ROM at 4040000000 [disabled] [size=16M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [168] Address Translation Service (ATS)

05:00.0 Power PC: Freescale Semiconductor Inc Device 0000 (rev 01)
Flags: bus master, fast devsel, latency 0, IRQ 85
Memory at 4041800000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at 4042000000 [disabled] [size=16M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [158] Address Translation Service (ATS)
Capabilities: [168] L1 PM Substates

Here in the above,

04:00.0 is Endpoint-1 and 05:00.0 is Endpoint-2.

As per my understanding,

I am writing to address 0xE100_0000 which is iATU base address offset outbound (Upper, Lower) of BAR1 in Endpoint-1. 0xE100_0000 after address translation is 0x40_4180_0000 which is the iATU Target address (upper, lower).

So the data written to 0xE100_0000 should go to 0x40_4180_0000 which is the base address (BAR) of Endpoint-2 (05:00.0).

Again as per the Inbound translation (BAR match mode) in Endpoint-2, data received from 0x40_4180_0000 should go to 0xA000_0000 which is iATU Target address offset inbound (Upper, Lower)

But I don't see it working as expected. Please firstly let me know, Is my understanding correct ?

Thanks.

[TrinathK]

Hi @Hector_Villarruel 

This document doesn't have PCIe EP to EP communication example. Please provide EP to EP example document if you have any.

Thanks.

[Hector_Villarruel]

Hello @TrinathK 

Hope this email finds you well,

I would like to inform you that we do have a documentation regarding your last reply, nevertheless such documentation can is NXP Confidential Proprietary and can only be provided under non-disclosure agreement (NDA).

In order to provide the requested information, please create a new ticket using the technical support web located in the following link:

https://support.nxp.com/s/

Best Regards,

Hector Villarruel

[Hector_Villarruel]

Hello @TrinathK 

I would like to provide you a documentation that can be useful to this post, 

Please find in the following link the "PCIe EP/RC Validation and Throughput" Using an i.MX6Q.

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX6Q-PCIe-EP-RC-Validation-System/ta-p...

Have a great day.

Hector V

[TrinathK]

Hi @ufedor 

Please check this query.

Thanks.