I started with reproducing the i.MX6Q PCIe EP/RC validation done by NXP using iMX6Q and i.MX6SoloX development board. At both boards Linux imx-4.9.11-1.0.0_ga SDK is used.
The i.MX6SoloX is configured as RC and the i.MX6Q as EP. The test run automatically by the EP is successful and produces the following result:
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Starting data transfer...
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Data transfer is successful. tv_count1 34682us, tv_count2 104120us.
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Data write speed:59MB/s.
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Data read speed:19MB/s.
lspci at the RC is also as expected:
root@imx6sxsabresd:~# lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 309
Memory at 08800000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00001000-00001fff
Memory behind bridge: 08000000-087fffff
Prefetchable memory behind bridge: 08900000-089fffff
[virtual] Expansion ROM at 08a00000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: pcieport
01:00.0 RAM memory: Device beaf:dead (rev 01)
Subsystem: Device beaf:dead
Flags: bus master, fast devsel, latency 0, IRQ 310
Memory at 08000000 (32-bit, non-prefetchable) [size=8M]
I/O ports at 1000 [size=4K]
Memory at 08910000 (32-bit, prefetchable) [size=256]
[virtual] Expansion ROM at 08900000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: imx_pcie_ep
So far, so good. My next step is to access the EP RAM from the RC. Because the outbound iATU at the RC added by the NXP PCIe EP compliance test code in drivers/pci/host/pci-imx6.c did work did not work this outbound iATU has been removed from the RC. Instead using memtool like commands the inbound iATU has been added at the EP:
root@imx6qsabresd:~# rio 32 1ffc900 20
<0x1ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x1ffc910> 0x00000000 0x0000ffff 0x4f000000 0x00000000
This is an inbound iATU that translates the BAR0 region to the EPs test RAM region. This works fine, the RAM at the EP can be accessed using the PCIe memory region at the RC.
RC:
root@imx6sxsabresd:~# wio 32 8000000 deadbeef
<0x8000000> 0xdeadbeef
root@imx6sxsabresd:~# rio 32 8000000 20
<0x8000000> 0xdeadbeef 0xf79ef79e 0xf79ef79e 0xf79ef79e
<0x8000010> 0xf79ef79e 0xf79ef79e 0xf79ef79e 0xf79ef79e
EP:
root@imx6qsabresd:~# rio 32 4f000000 20
<0x4f000000> 0xdeadbeef 0xf79ef79e 0xf79ef79e 0xf79ef79e
<0x4f000010> 0xf79ef79e 0xf79ef79e 0xf79ef79e 0xf79ef79e
Note: Because the kernel did not start with the "mem=768M" therefore "mem=1008" is used and the ddr_test_region is moved 0x4f000000 (i.MX6Q) or 0xbf000000 (i.MX6SoloX).
For our purpose we want to access the EP peripherals (e.g. USB) from the RC, so I changed the target address of the inbound iATU to the AIPS (0x2000000)
EP:
root@imx6qsabresd:~# rio 32 1ffc900 20
<0x1ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x1ffc910> 0x00000000 0x0000ffff 0x02000000 0x00000000
root@imx6qsabresd:~# rio 32 2184200 18
<0x2184200> 0xe401fa05 0x00000115 0x10020001 0x00000000
<0x2184210> 0x80080808 0x00000808
RC:
root@imx6sxsabresd:~# rio 32 8184200 18
<0x8184200> 0xe401fa05 0x00000115 0x10020001 0x00000000
<0x8184210> 0x80080808 0x00000808
So when the i.MX6SoloX is the RC and the i.MX6Q the EP everything is working as expected, but when the EP and RC roles are swapped it is possible to access the EPs RAM from the RC but its is not possible to access the EPs peripherals.
EP (set inbound iATU to test RAM and read test RAM):
root@imx6sxsabresd:~# rio 32 8ffc900 20
<0x8ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x8ffc910> 0x00000000 0x0000ffff 0xbf000000 0x00000000
root@imx6sxsabresd:~# rio 32 bf184200 18
<0xbf184200> 0x00000000 0x00000000 0x00000000 0x00000000
<0xbf184210> 0x00000000 0x00000000
RC (write and read back EPs test RAM):
root@imx6qsabresd:~# wio 32 1184200 deadbeef
<0x1184200> 0xdeadbeef
root@imx6qsabresd:~# rio 32 1184200 18
<0x1184200> 0xdeadbeef 0x00000000 0x00000000 0x00000000
<0x1184210> 0x00000000 0x00000000
EP (set inbound iATU to AIPS and read USB registers):
root@imx6sxsabresd:~# wio 32 8ffc918 2000000
<0x8ffc918> 0x2000000
root@imx6sxsabresd:~# rio 32 8ffc900 20
<0x8ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x8ffc910> 0x00000000 0x0000ffff 0x02000000 0x00000000
root@imx6sxsabresd:~# rio 32 2184200 18
<0x2184200> 0xe401fa05 0x00000015 0x10020001 0x00000011
<0x2184210> 0x80080b08 0x00000808
RC (read EPs USB registers):
root@imx6qsabresd:~# rio 32 1184200 18
Unhandled fault: external abort on non-linefetch (0x1018) at 0x76f47200
pgd = a2724000
[76f47200] *pgd=32875831, *pte=01184703, *ppte=01184e33
Bus error (core dumped)
Did I miss something when using the i.MC6SoloX as EP or is this an iMX6SoloX issue?
Hello,
in the original thread https://community.nxp.com/docs/DOC-95014
the follwoing is mentioned "In EP's system, EP can access the reserved ddr memory ..."
and the patch and kernel line settings are shown. Have You performed it ?
Have a great day,
Yuri
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------
In the latest linux-imx (4.9.11-1.0.0_ga) the patches you mentioned are already merged to main. So I did not need to apply the patches. Anyhow as described I could access the other boards RAM both from the the EP and RC. The issue is when I try to access the USB registers. This does not work if the i.MX6SoloX is the EP and the i.MX6Q is the RC. If the EP and RC roles are swapped it is possible.