Implementing PCIe support for bridges in iMX6Q WEC7 BSPs

[ilkodossev]

People,

I am trying to implement full PCIe support for Windows CE BSPs, including bridges. My specific hardware is PLX 8606.

My references are:

- FSL documentation (IMX6DQRMr2 part1 and part2)

- Linux implementation for PCI bus enumeration

- PCIe implementation for x86 BSPs on Windows CE platform

- PCI.ORG documentation

From the very beginning I detected two problems in OAL support:

  - Only BDF 0:0:0 is accepted, all other are rejected. I corrected that.

  - When 0 bytes are read, OAL fills up the entire configuration structure with FFs thus simulating multi-function device and causing PCIBus driver to look up phantom functions. I corrected that, too.

I followed the ideas of Linux implementation for traversing PCI bus beyond the bridge, but this fails. I can only see the configuration of the 8606 bridge.

PCI controller is initialized in Root Complex mode, but its class code is "undefined" initially, setting up that configuration was my first step.

I have a few questions regarding this development.

1. Linux code sets up RC as PCI bridge (0604, rev. 0) and treats the configuration space as Header Type 1. This includes addr. 0x3C which in IMX6DQRM is described as PCIE_RC_EROMMASK.

  Is this correct, or changing EROMMASK just happens to be inconsequential because EROMBAR is disabled?

2. Linux sets up BAR0 and all BLRs with specific values. I can hard-code same values in IO, MEM, PREF_MEM as well, but I could not deduce how these values are calculated.

  Does they depend on the BARs of the bridge, and if not -- what would be the right method to calculate them? Writing FFs only brings back the masks and these masks allow very broad ranges.

3. Linux sets up ATU.0 for BDF 0:0:0 in "...\arch\arm\mach-mx6\pcie.c", but later some other module changes iATURLTA to BDF 1:0:0.

  I could not locate that part of the code in the sources -- any help on it?

4. So far, I can only see the configuration space for the first device -- the bridge. Even if there are no more devices on bus 0, I am getting response for all BDFs 0:x:0; the whole 64KB mapped configuration space are populated with repeated blocks.

  Is there way to determine from the Configuration Space Header whether a device truly exists, when for example BDF 0:1:0 is requested?

5. And at the end, when the bridge is pre-configured and the time comes to traverse it, I write to iATURLTA new BDF 1:0:0 and read the bus using the address of the already mapped configuration space at 0x01F00000.

  Yet in it the VID/PID of the bridge only shows. Do I need to map another memory space from bridge's BARs to get the info for downstream devices?

Sharing the experience for Linux Kernel PCI support is valuable, even if there is no direct correspondence between that Kernel and OAL in WEC7.

Best Regards,

Ilko Dossev

[ilkodossev · Accepted solution]

I found he solution for PCIe bridges connected to iMX6 Root Complex.

Now I can traverse, enumerate, initialize and operate endpoints behind the bridge.

Full PCIe Bridge Support for WEC7 is now available from Adeneo Embedded.

Modifications include:
1. Update of OAL PCI Configuration Space access using ATU.

2. Update of PCI Bus Driver to complete full initialization of Root Complex.

3. Update of PCI Bus Driver to enable interrupt sharing on PCIe bus.

4. Implementing proper ISR handlers for endpoints of a kind.

5. ..and, of course, implementing Device Driver for the endpoint with corresponding Registry entry template for PCI Bus Driver.

Ilko Dossev

Senior Software Engineer

Adeneo Embedded US

[ilkodossev]

There is a lot of progress I made, even if few outstanding questions about Root Complex registers still remain.

I will split the answers in three separate categories:

  1. Traversing PCIe bus and enumerating devices on it;

  2. MMIO configuration of enumerated devices;

  3. IRQ configuration of enumerated devices.

These posts are yet to come, so I cannot provide links to them yet.

I promise, however, to title them exactly as listed above, so search will turn these posts up when they are ready.

In general, current source code across all platforms provides only rudimentary support for PCIe, and support for bridges is effectively non-existent. Yes, PCIBus Driver does handle bridges, but underlying OAL layer does not.

There is a little neat utility in the COMMON branch, named "pcienum":

  ...\WINCE700\public\common\oak\drivers\ceddk\test\pcienum\

Building it and running it will print out full information about the devices currently existing on the PCIe bus; this reflects bridge support as well.

You must add this little fragment to the IOCTL support for the platform,

    switch (dwIoControlCode) {

        case IOCTL_HAL_DDK_CALL:

            if (pInBuf != NULL && nInBufSize >= 4) {

                // only peeking PCIe topology is allowed

                if (IOCTL_OAL_READBUSDATA != *((DWORD*)pInBuf)) {

                    SetLastError(ERROR_ACCESS_DENIED);

                    break;

                }

            }

        }

It will allow IOCTLs to propagate to OALPCICfgRead() function and will report currently found PCI devices.

All in all, full PCIe bridge support for Freescale iMX6 family will be coming soon in Adeneo platforms.

[ilkodossev]

As 2014 approaches its end, many, if not all, WEC7 platforms for iMX6 still do not support properly PCI bus.

All platforms I've come across so far only accept BDF 0:0:0 in their OAL PCI support, which effectively limits the connection to just one single endpoint.

Neither bridges, not multifunction endpoints could be used, thus WEC7 (and 2013) applications get limited and render creation of customized PCI-relying hardware quite difficult.

Using a PCI-to-PCI bridge, several PCI/PCIe devices can be connected easily to iMX6 host.

This can be accomplished if two requirements are met:

  1. Bus numbering is done properly; and

  2. ATU is re-programmed to provide access to busses and devices downstream.

Host controller must be initialized as Root Complex, and its bus number must be set to 0; its secondary bus should be 1, and subordinate bus number can be the highest possible 255.

This is what PCI topology might look like with bridge connected to the Root Complex:

  HOST

      |

      | bus 0

      |

    RC:  BDF 0.0.0
      |

      | bus 1

      |

BRIDGE:  BDF 1.0.0 for its upstream port
        |

     bus 2 is internal for the bridge, created implicitly for access to downstream ports

                   /                  |                    \

           BDF 2.0.0    BDF 2.1.0    BDF 2.2.0

                 |                    |                     |

                 | bus 3          | bus 4           | bus 5

                 |                    |                     |

           BDF 3.0.*      BDF 4.1.*      (empty)

         endpoint A   endpoint B

The diagram above shows that there will be 5 busses numbered from 0 to 4, and if the last slot of the bridge is also populated then there will be 6 busses.

Given the topology above, now ATU can be properly re-programmed each time when a configuration of a PCI device needs to be read or written.

For example, ATU viewport 0 can be used for that purpose, provided that access to that code fragment is fully serialized.

Access to the upstream port is simple -- ATU[0].iATURC1.TYPE needs to be  set to 4 for CFG0 type of transactions.

Access to all downstream ports and to the endpoints connected to them requires ATU[0].iATURC1.TYPE to be set to 5 for CFG1 transaction type.

There are several posts about programming ATU for LINUX platforms; they provide very fine details and useful patches.

As side benefit from implementation of such PCI handling, such scheme would provide support for multifunction endpoint connected directly to Root Complex as well.

CAVEAT:

Beware of bridge's port numbering - ports may not be compact but scattered; for example   x1xx45x7, so enumeration starting from 0 will begin with exception as ATU cannot connect to non-existing device.

This requires implementation of exception handling in the enumeration procedure.

Ilko Dossev

idossev@adeneo-embedded.us

[CarlosCasillas]

Deactivated user, could you please comment?

[ilkodossev]

Carlos, I am Adeneo too! ☺

And I just got one – O-N-E endpoint connected thru RootComplex and thru 8606 bridge connected to RC.

I will be updating my post because I now have important info to add; and I will summarize the 2 or 3 remaining Qs about RC registers.

Thanks for the heads-up,

~Ilko

[karina_valencia]

b47504 can you  attend this case?