PCIe EndPoint iMX6SX

[paolo_minazzi]

Hello,

we are trying to connect

• iMX6SX cpu
• x86 module

by PCIe.

The iMX6SX must be a PCIe Endpoint.

I need only shared memory between iMX6SX and x86. I do not need interrupt.

Shared memory must be in a region of DDR RAM of iMX6SX.

On iMX6SX I use linux-imx-imx_4.9.88_2.0.0_ga-var01.

On x86 I'm using an EFI shell that allow me to see pci devices and addresses assigned.

I reserve some memory for shared memory, so Linux will not use this.

I boot with mem=128M.

I start using the endpoint configuration used from the validation test.

On Linux I have 

I power-on the iMX6SX and  I see on serial console :

PCIe EP: waiting for link up...

All is OK ... iMX6SX is waiting PCIe link.

Now I reset the x86 module.

As soon as I power-on x86, iMX6SX get the link.

For now all is OK.

Now I have a linux uart console.

On x86 I can see using the EFI shell the PCI configurazione.

I see:

• my endpoint as Memory controller - RAM memory controller.
• VID and PID are correct.
• the address assigned by x86 to BARs.
• the correct size of the BARs I assigned

The problem is that after some seconds linux freezes.

On the x86 I can see the PCI register of the iMX6SX endpoint.

x86 works OK, does not freeze.

I attach the complete dump of my PCI endpoint.

From the dump I see two strange things:

1. Command register go to 0.
2. Received Master Abort:          1

Only for testing, I tried linux-imx-rel_imx_4.1.15_2.1.0_ga and linux-imx-rel_imx_4.14.98_2.0.0_ga.

On 4.1.15 I have seen problem with the PCIe link, with 4.14.98 I have seen something similar what I have seen with 4.9.88.

I did not investigate.

On 4.14.98 I have seen an option in menuconfig.

but I don't think it is the right way to follow.

My final question is if I'm on the correct way using EP config used for Validation Test.

Is this OK as a base for PCIe Endpoint ? If is not complete, what it is missing ?

I realize that I could have setted badly some address in PCI controller, but for now, if none writes (x86 and iMX6SX), I think that this kind of error could not be important.

Thanks for your time,

Best regards,

Paolo

[paolo_minazzi]

Thanks Igor.

On x86 using both

• linux
• windows

I do not have the freezer on iMX6SX.

The problem was something in the BIOS of x86.

Maybe it writes some configuration that is not OK. Now I will investigate.

To use the shared memory I have setted the iATU registers in this way.

Maybe it could be useful to someone.

uint32_t viewport             = 0;
uint32_t tlp_type             = 0;
uint32_t addr_base_cpu_side   = 0xbff00000; // phy address
uint32_t size                 = SZ_1M;
 
HW_PCIE_PL_IATUVR_WR((viewport & 0x0F) | (1 << 31));  // orig 0<<31
HW_PCIE_PL_IATURLBA_WR(addr_base_cpu_side);
HW_PCIE_PL_IATURUBA_WR(0);
HW_PCIE_PL_IATURLA_WR(addr_base_cpu_side + size - 1);
HW_PCIE_PL_IATURC1_WR(tlp_type & 0x0F);
HW_PCIE_PL_IATURC2_WR(   ((unsigned int)(1 << 31))
                       | ((unsigned int)(1 << 30))  // BAR MODE
                     );

Thanks for your support

Best regards,

Paolo

[paolo_minazzi]

Hi Igor,

thanks for the link. I studied the code and I found pcie_map_space function.

I still cannot understand the linux freeze.

Now I have a question abount clocks.

The standard DTS of iMX6SX is

clocks = <&clks IMX6SX_CLK_PCIE_AXI>,

         <&clks IMX6SX_CLK_LVDS1_OUT>

         <&clks IMX6SX_CLK_PCIE_REF_125M>,

         <&clks IMX6SX_CLK_DISPLAY_AXI>;

clock-names = "pcie", "pcie_bus", "pcie_phy", "pcie_inbound_axi";

This is our PCI system:

The clk is the PCIexpress 100 MHz LVDS clock.

It is connected to the CLK1p, CLK1n pins.

Q1 : Is the standard DTS ok for this configuration ?

Using the standard DTS :

• the iMX6SX can see PCIe link
• the x86 can assign BAR0 to the endpoint

Q2 : Does it means that the clocks are already ok ?

The freeze depends on PCI. I'm sure of this, but for now I cannot find the source.

Thanks for support,

Best regards,

Paolo

[igorpadykov]

Hi Paolo

>Q1 : Is the standard DTS ok for this configuration ?

from log (" linux-imx-imx_4.9.88_2.0.0_ga-var01") seems you are using

some of variscite (Leading ARM System On Module (SoM) designer and manufacturer | Variscite )

boards. I am not familiar with them, NXP i.MX6SX reference board uses external PCIe clock, below

part of i.MX6SX Sabre SD schematic:

i.MX 6SoloX SABRE Development Board | NXP 

>Q2 : Does it means that the clocks are already ok ?

I believe yes. However based on your description :

"The problem is that after some seconds linux freezes."

There may be issues with long term clock instabilities appearing after some time,

so recommendation is to test this case on NXP i.MX6SX Sabre SD reference board

with official Linux from source.codeaurora.org/external/imx/linux-imx  repository
linux-imx - i.MX Linux kernel 

Also may be useful:

PCIe BAR length limit 

i.MX6 PCIe: supporting devices larger than 16MB 

Best regards
igor

[igorpadykov]

Hi Paolo

I believe EP config can be used for validation test:

i.MX6Q PCIe EP/RC Validation System 

To narrow down freeze problems one can try with other pcie cards

(check if this is caused by low signals). Just for test one can try using Gen1 mode

https://community.nxp.com/thread/313631 

i.MX6Q: Re-establishing a PCIe link 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[paolo_minazzi]

Thanks for you answers Igor.

1. PCIe LINK

I tried to force Gen1 (both BIOS x86 and iMX6SX) using the link you have show to me.

In UEFI shell I verify the link speed is Gen1 (that is 2.5GHZ).

Furthermore, I checked the link seems good.

To do this, after I have the link, I use simple

while(1) {

   printk("%d", (readl(pp->dbi_base + PCIE_PHY_DEBUG_R1) & 0x10) != 0);

}

And I see always "1". Then, after some seconds, I have the freeze. But the link seems OK.

2. PROBLEM FLAG PCI_COMMAND_MEMORY

I see in sequence:

• boot linux on iMX6
• iMX6 wait PCIe link
• boot x86
• iMX6 see link
• for some seconds flag PCI_COMMAND_MEMORY is setted
• someone (I think x86) reset flag PCI_COMMAND_MEMORY
• After 1 second I have linux freeze

Why PCI_COMMAND_MEMORY flag is resetted ? How reset it ? x86 ?

3. TEST : RESET ALL BARS

I tried a simple test : disable all BARs (putting 0 in the MASK of all BARs) .

writel(0, pp->dbi_base + (1 << 12) + PCI_BASE_ADDRESS_0..5);

Also in this case I see the linux freeze freeze.

This says it should not be a memory address problem. All BARs are disabled!

4. ADDRESS OR BAR

Latest question.

My iMX6SX has 1GB DDR.

The start physical address of DDR is 0x8000_0000.

The end physical address of DDR is 0xc000_0000.

If I want a shared memory of 1MB at the end of my DDR, my address is 0xbff0_0000.

writel(0xbff00000, pp->dbi_base + PCI_BASE_ADDRESS_0);

writel(SZ_1M, pp->dbi_base + (1 << 12) + PCI_BASE_ADDRESS_0);

Is it OK ?

Thanks for you time and support.

Best regards,

Paolo

[igorpadykov]

Hi Paolo

in general one can try to test without linux, baremetal test:

rt-thread/pcie.c at master · RT-Thread/rt-thread · GitHub 

Best regards
igor

[paolo_minazzi]

I tried to skip the memory speed test (adding a return, see code below).

Adding the return, I have :

• command 0x00
• Status(6) 0x0010 (that is, Receive Master Abort is not setted)

I verify,  after the link is OK, command is 0x07 (correct) for some seconds, than it become 0x00.

do {
   usleep_range(10, 20);
   if (time_after(jiffies, timeout)) {
   dev_info(dev, "PCIe EP: link down.\n");
   return 0;
   }
} while ((readl(pp->dbi_base + PCIE_PHY_DEBUG_R1) & 0x10) == 0);

return;