imx8mm - no pcie end point controller

[antho]

Hi all,

I'm testing PCIe in End Point mode on an imx8mm. Linux kernel is 4.19. I cannot find a way to make it work. Please comment if anything seems wrong. Any advice could be useful.

Following kernel documentation, I should be able to play with pcie endpoint controller in /sys/class/pci_epc and test the interface with lspci and pcitest on the root complex side (x86).

https://www.kernel.org/doc/html/latest/PCI/endpoint/pci-test-howto.html 

The thing is, PCIe endpoint controller device does not exist (nothing in /sys/class/pci_epc). It should be created by a driver activated in the kernel (drivers/pci/controller/dwc/pcie-designware-ep.c ?). Is it created after pcie link is brought up ?

PCIe link is never established (timeout !). Are we talking about a link between RC and EP (see logs below) ? Or is it a phy link between two modules inside the iMX ? I tried to power up the x86 RC connected to the iMX when it is waiting, but nothing happened.

imx kernel logs :

[ 2.266698] imx6q-pcie 33800000.pcie: host bridge /hsio/pcie@33800000 ranges:
[ 2.273869] imx6q-pcie 33800000.pcie: No bus range found for /hsio/pcie@33800000, using [bus 00-ff]
[ 2.283110] imx6q-pcie 33800000.pcie: IO 0x1ff80000..0x1ff8ffff -> 0x00000000
[ 2.290521] imx6q-pcie 33800000.pcie: MEM 0x18000000..0x1fefffff -> 0x18000000
[ 2.297926] imx6q-pcie 33800000.pcie: try to initialize pcie ep.
[ 2.304036] imx6q-pcie 33800000.pcie: Initialize PHY with PLL REfCLK!.
[ 2.310951] imx6q-pcie 33800000.pcie: PHY Initialization End!.
[ 2.317119] imx6q-pcie 33800000.pcie: iATU unroll: enabled
[ 2.322617] imx6q-pcie 33800000.pcie: ddr_test_region is 0xb8200000.
[ 2.328982] imx6q-pcie 33800000.pcie: PCIe EP: waiting for link up...

[timeout]

[ 302.268463] imx6q-pcie 33800000.pcie: PCIe EP: link down.

device tree pcie node (including kernel 4.19 fsl-imx8mm.dtsi) :

&pcie0{
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_pcie0>;
    //disable-gpio = <&gpio1 5 GPIO_ACTIVE_LOW>;
    //reset-gpio = <&gpio4 21 GPIO_ACTIVE_LOW>;
    ext_osc = <0>; // use external clk or internal pll if 0
    status = "okay";
};

Here's a diff of my kernel .config :

+CONFIG_PCIEPORTBUS=y
+CONFIG_HOTPLUG_PCI_PCIE=y
+CONFIG_PCIEAER=y
+CONFIG_PCIEASPM=y
+CONFIG_PCIEASPM_DEFAULT=y
+CONFIG_PCIE_PME=y
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
CONFIG_PCI_QUIRKS=y
CONFIG_PCIE_DW=y
CONFIG_PCIE_DW_HOST=y
+CONFIG_PCIE_DW_EP=y
+CONFIG_PCIE_DW_PLAT=y
# CONFIG_PCIE_DW_PLAT_HOST is not set
+CONFIG_PCIE_DW_PLAT_EP=y
CONFIG_PCI_IMX6=y
# CONFIG_PCI_IMX6_COMPLIANCE_TEST is not set
+CONFIG_EP_MODE_IN_EP_RC_SYS=y
# CONFIG_PCI_LAYERSCAPE is not set
CONFIG_PCI_HISI=y
# CONFIG_PCIE_QCOM is not set
+CONFIG_PCI_ENDPOINT=y
+CONFIG_PCI_ENDPOINT_CONFIGFS=y
+CONFIG_PCI_EPF_TEST=y
+CONFIG_PCI_ENDPOINT_TEST=y

[Guohua_Liu]

Hello ancho,

Have you managed to make your board working in EP mode? I was working on a board that is bought from other company because our own hardware is not ready for now, but their kernel is 4.14.78, and their u-boot is close-sourced, that makes it very difficult to port kernel 5.x. Now we have a new board with kernel 5.4.70. 

We have modifiied the device tree and kernel config, but it still does not work. It seems that the kernel stops when reading data from the EP controller.

Please give some help if you have done it, thanks!

[pete_sensoray]

Hi @Guohua_Liu @antho 

Were you able to make the PCIE work in endpoint mode with Kernel 5.4?  I'm trying get endpoint mode working under 5.10.72.

I've modified the device tree to use the endpoint mode as follows:

/* Switch PCIE to endpoint mode */
&pcie {
status = "disabled";
};

&pcie_ep {
ext_osc = <0>;
status = "okay";
};

&pcie_phy {
ext_osc = <0>;
/* clocks = <&clk IMX8MP_CLK_PCIE_PHY>; */
status = "okay";
};

Should the EP board use the REF_CLK or EXT_CLK?

Should the RC board be booted before the EP board?

Thanks for any help you can provide.

[Guohua_Liu]

Sorry for my late reply. We've finally succeed to make the PCIE work. The EP board should use ext-clock, it should be disconnected from the host's clock source on hardware.

[Guohua_Liu]

Hello antho,

We are also working on a board which is supposed to work in EP mode and have some problems. Have you made some progress?

[antho]

Hi,

I fixed a hardware problem on my board. I had swapped RX lanes on the pcb. Then, I could go further in the link detection.

[ 2.285062] imx6q-pcie 33800000.pcie: host bridge /hsio/pcie@33800000 ranges:
[ 2.292244] imx6q-pcie 33800000.pcie: No bus range found for /hsio/pcie@33800000, using [bus 00-ff]
[ 2.301502] imx6q-pcie 33800000.pcie: IO 0x1ff80000..0x1ff8ffff -> 0x00000000
[ 2.308945] imx6q-pcie 33800000.pcie: MEM 0x18000000..0x1fefffff -> 0x18000000
[ 2.316374] imx6q-pcie 33800000.pcie: try to initialize pcie ep.
[ 2.322492] imx6q-pcie 33800000.pcie: Initialize PHY with PLL REfCLK!.
[ 2.329350] imx6q-pcie 33800000.pcie: PHY Initialization End!.
[ 2.335549] imx6q-pcie 33800000.pcie: iATU unroll: enabled
[ 2.341068] imx6q-pcie 33800000.pcie: ddr_test_region is 0xb8200000.
[ 2.347446] imx6q-pcie 33800000.pcie: PCIe EP: waiting for link up...
[ 2.482082] imx6q-pcie 33800000.pcie: pcie ep: Starting data transfer...
[ 2.791039] imx6q-pcie 33800000.pcie: dma write no end ...
[ 2.848045] imx pcie dma read error 0x10000.
[ 2.848052] imx pcie dma read error 0x10000.
[ 2.852322] imx pcie dma read error 0x10000.
[ 2.856592] imx pcie dma read error 0x10000.
[ 2.860862] imx pcie dma read error 0x10000.
[ 2.865131] imx pcie dma read error 0x10000.
[ 2.869401] imx pcie dma read error 0x10000.
[ 2.873670] imx pcie dma read error 0x10000.
[ 2.877939] imx pcie dma read error 0x10000.
[ 2.882211] imx pcie dma read error 0x10000.
[ 2.886481] imx pcie dma read error 0x10000.
[ 2.890750] imx pcie dma read error 0x10000.
[ 3.099038] imx6q-pcie 33800000.pcie: dma read no end
[ 3.108369] imx6q-pcie 33800000.pcie: pcie ep: Data transfer is failed.

I was told to port my layer in kernel 5.4 to investigate. So I did.

Then, I had only a few kernel logs concerning pcie :

[ 2.153297] imx6q-pcie 33800000.pcie_ep: 33800000.pcie_ep supply epdev_on not found, using dummy regulator
[ 2.163173] imx6q-pcie 33800000.pcie_ep: PLL REF_CLK is used!.
[ 2.173755] imx6q-pcie 33800000.pcie_ep: PCIe PLL locked after 20 us.

and /sys/class/pci_epc/ controller is working, so I can set a pcie test function.

The problem is, the x86 root complex of my system does not detect the imx8 configured in end point.

Now I'm stuck here. Do you guys know what to check ? Any advice to help me debugging ?

Edit :
I had to use the EXT REF_CLK and boot the EP after the RC, so I suppose its clk is available on the PCIe connector. Then I can set a PCIe test function thanks to /sys/class/pci_epc/ controller and detect a device on the RC side.

[kajolkharbanda]

hello

I am facing same issue, cannot find epc controller on imx8qxpc0mek

root@imx8qxpc0mek:~# dmesg | grep pci
[ 1.366617] ehci-pci: EHCI PCI platform driver
[ 1.383328] ohci-pci: OHCI PCI platform driver
[ 3.225507] imx6q-pcie 5f010000.pcie: 5f010000.pcie supply epdev_on not found, using dummy regulator
[ 3.234971] imx6q-pcie 5f010000.pcie: No cache used with register defaults set!
[ 3.291857] imx6q-pcie 5f010000.pcie: PCIe PLL locked after 0 us.
[ 3.298050] imx6q-pcie 5f010000.pcie: host bridge /bus@5f000000/pcie@0x5f010000 ranges:
[ 3.306403] imx6q-pcie 5f010000.pcie: IO 0x7ff80000..0x7ff8ffff -> 0x00000000
[ 3.313827] imx6q-pcie 5f010000.pcie: MEM 0x70000000..0x7fefffff -> 0x70000000
[ 3.421474] imx6q-pcie 5f010000.pcie: Link up
[ 3.525900] imx6q-pcie 5f010000.pcie: Link up
[ 3.530293] imx6q-pcie 5f010000.pcie: Link up, Gen1
[ 3.535652] imx6q-pcie 5f010000.pcie: PCI host bridge to bus 0000:00
[ 3.542028] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.553275] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 3.553283] pci_bus 0000:00: root bus resource [mem 0x70000000-0x7fefffff]
[ 3.564896] pci 0000:00:00.0: [1957:0000] type 01 class 0x060400
[ 3.578396] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[ 3.590346] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x00ffffff pref]
[ 3.604751] pci 0000:00:00.0: supports D1 D2
[ 3.618681] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 3.629425] pci 0000:00:00.0: BAR 0: assigned [mem 0x70000000-0x70ffffff]
[ 3.636230] pci 0000:00:00.0: BAR 6: assigned [mem 0x71000000-0x71ffffff pref]
[ 3.636239] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 3.651905] pcieport 0000:00:00.0: PME: Signaling with IRQ 466
[ 3.664263] pcieport 0000:00:00.0: AER: enabled with IRQ 466
root@imx8qxpc0mek:~#

[dwang2]

Did you solve the problem at the end? Can you share the solution here?

Thank you!

[Guohua_Liu]

Yeah, we finally got that board working. But because it was so long ago, I couldn't remember the exact details. In general, we have done these things: 1. Use external clock sources on the hardware instead of the clock sources on the host; 2. Port the PCIE controller driver from the Linux 5.10 source code to Linux 5.4, because the PCIE controller was not initialized when PCIE worked as an EP on Linux 5.4.

[Guohua_Liu]

Sorry for replying so late. Because of some high priority tasks we have not been working on it for some days. Before my last reply, our hardware engineer modified the hardware, he swapped the rx/tx, cut off power supply of pcie port and disconnected the clock source from the pcie. We also enabled EP feature in kernel configuration. It seems that host PC can detect the board, but the R/W test fails as you mentioned above.

We just compiled a new boot image with yocto, the kernel version of which is 5.10.72, but unfortunately we failed to burn it with uuu even to official imx8mm-evk board. A strange thing is that the size of our "imx-boot-imx8mmddr4evk-sd.bin-flash_ddr4_evk" is 3552KB which is much bigger than the size (1962KB) of the one I download from official web site, the later can easily be burned  to the official evk board.

Any one who can give some advice? Thanks in advance.

[antho]

Hi,

Do you flash into an emmc storage ? I had a very similar problem :

MMC memory chips have a boot hardware partition (write protected). You can check it size by running in u-boot=> mmc info
Look for "Boot Capacity". Mine was 2MB, which was okay for yocto Warrior imx-boot image but not big enough for yocto Zeus image of 3.5MB.

Solutions :
-you can resize the mmc boot partition (it didn't work)
-you can move the bootloader to the main partition (but it won't be protected)
-you can reduce the imx-boot size by removing some feature. I removed the optee binary (2.1MB) in the distro config file. And it worked !

[antho]

Hi Igor,
I know this article from 2013. It is more an old validation and results note than a real tutorial to me.

It is base on an old kernel (imx_3.0.35 ?). I don't even find the drivers files that have to be patched (arch/arm/mach-mx6/pcie.c or mxc_ipudev_test.c). We don't even know what the attached test program is supposed to do (no source code).

Sorry, but this is not helping me.

Please, do you have more recent application notes about PCIe in EP mode ?

I'm planning to jump to kernel 5.4, but it is going to take a while.

[igorpadykov]

Hi Anthony

for testing pcie one can look at tutorial

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX6Q-PCIe-EP-RC-Validation-System/ta-p...

and sect.4.8 PCI Express Root Complex  i.MX Linux Reference Manual​

Best regards
igor