AnsweredAssumed Answered

Issue with i.MX6SoloX as PCIe EP

Question asked by Kees Trommel on Oct 17, 2017
Latest reply on Nov 2, 2017 by Kees Trommel

I started with reproducing the i.MX6Q PCIe EP/RC validation done by NXP using iMX6Q and i.MX6SoloX development board. At both boards Linux imx-4.9.11-1.0.0_ga SDK is used.

The i.MX6SoloX is configured as RC and the i.MX6Q as EP.  The test run automatically by the EP is successful and produces the following result:

Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Starting data transfer...
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Data transfer is successful. tv_count1 34682us, tv_count2 104120us.
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Data write speed:59MB/s.
Oct 19 01:51:26 imx6qsabresd user.info kernel: pcie ep: Data read speed:19MB/s.

lspci at the RC is also as expected:

root@imx6sxsabresd:~# lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 309
        Memory at 08800000 (32-bit, non-prefetchable) [size=1M]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 08000000-087fffff
        Prefetchable memory behind bridge: 08900000-089fffff
        [virtual] Expansion ROM at 08a00000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport

01:00.0 RAM memory: Device beaf:dead (rev 01)
        Subsystem: Device beaf:dead
        Flags: bus master, fast devsel, latency 0, IRQ 310
        Memory at 08000000 (32-bit, non-prefetchable) [size=8M]
        I/O ports at 1000 [size=4K]
        Memory at 08910000 (32-bit, prefetchable) [size=256]
        [virtual] Expansion ROM at 08900000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Kernel driver in use: imx_pcie_ep

So far, so good. My next step is to access the EP RAM from the RC. Because the outbound iATU at the RC added by the NXP PCIe EP compliance test code in drivers/pci/host/pci-imx6.c did work did not work this outbound iATU has been removed from the RC. Instead using memtool like commands the inbound iATU has been added at the EP:

root@imx6qsabresd:~# rio 32 1ffc900 20

<0x1ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x1ffc910> 0x00000000 0x0000ffff 0x4f000000 0x00000000

This is an inbound iATU that translates the BAR0 region to the EPs test RAM region. This works fine, the RAM at the EP can be accessed using the PCIe memory region at the RC.

RC:

root@imx6sxsabresd:~# wio 32 8000000 deadbeef
<0x8000000> 0xdeadbeef
root@imx6sxsabresd:~# rio 32 8000000 20
<0x8000000> 0xdeadbeef 0xf79ef79e 0xf79ef79e 0xf79ef79e
<0x8000010> 0xf79ef79e 0xf79ef79e 0xf79ef79e 0xf79ef79e

EP:

root@imx6qsabresd:~# rio 32 4f000000 20
<0x4f000000> 0xdeadbeef 0xf79ef79e 0xf79ef79e 0xf79ef79e
<0x4f000010> 0xf79ef79e 0xf79ef79e 0xf79ef79e 0xf79ef79e

Note: Because the kernel did not start with the "mem=768M" therefore "mem=1008" is used and the ddr_test_region is moved 0x4f000000 (i.MX6Q) or 0xbf000000 (i.MX6SoloX).

For our purpose we want to access the EP peripherals (e.g. USB) from the RC, so I changed the target address of the inbound iATU to the AIPS (0x2000000)

EP:

root@imx6qsabresd:~# rio 32 1ffc900 20
<0x1ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x1ffc910> 0x00000000 0x0000ffff 0x02000000 0x00000000
root@imx6qsabresd:~# rio 32 2184200 18
<0x2184200> 0xe401fa05 0x00000115 0x10020001 0x00000000
<0x2184210> 0x80080808 0x00000808

RC:

root@imx6sxsabresd:~# rio 32 8184200 18
<0x8184200> 0xe401fa05 0x00000115 0x10020001 0x00000000
<0x8184210> 0x80080808 0x00000808

So when the i.MX6SoloX is the RC and the i.MX6Q the EP everything is working as expected, but when the EP and RC roles are swapped it is possible to access the EPs RAM from the RC but its is not possible to access the EPs peripherals.

EP (set inbound iATU to test RAM and read test RAM):

root@imx6sxsabresd:~# rio 32 8ffc900 20
<0x8ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x8ffc910> 0x00000000 0x0000ffff 0xbf000000 0x00000000
root@imx6sxsabresd:~# rio 32 bf184200 18
<0xbf184200> 0x00000000 0x00000000 0x00000000 0x00000000
<0xbf184210> 0x00000000 0x00000000

RC (write and read back EPs test RAM):

root@imx6qsabresd:~# wio 32 1184200 deadbeef
<0x1184200> 0xdeadbeef
root@imx6qsabresd:~# rio 32 1184200 18
<0x1184200> 0xdeadbeef 0x00000000 0x00000000 0x00000000
<0x1184210> 0x00000000 0x00000000

EP (set inbound iATU to AIPS and read USB registers):

root@imx6sxsabresd:~# wio 32 8ffc918 2000000
<0x8ffc918> 0x2000000
root@imx6sxsabresd:~# rio 32 8ffc900 20
<0x8ffc900> 0x80000000 0x00000000 0xc0000000 0x00000000
<0x8ffc910> 0x00000000 0x0000ffff 0x02000000 0x00000000
root@imx6sxsabresd:~# rio 32 2184200 18
<0x2184200> 0xe401fa05 0x00000015 0x10020001 0x00000011
<0x2184210> 0x80080b08 0x00000808


RC (read EPs USB registers):

root@imx6qsabresd:~# rio 32 1184200 18
Unhandled fault: external abort on non-linefetch (0x1018) at 0x76f47200
pgd = a2724000
[76f47200] *pgd=32875831, *pte=01184703, *ppte=01184e33
Bus error (core dumped)


Did I miss something when using the i.MC6SoloX as EP or is this an iMX6SoloX issue?

Outcomes