WEC7/2013 - Traversing PCIe bus and enumerating devices on it

[ilkodossev]

This discussion is complementing my original "Implementing PCIe support for bridges in iMX6Q WEC7 BSPs" post.

iMX6 implementation has one single PCI connection, which can accommodate either one endpoint, or PCI-to-PCI Bridge.

When bridge is connected, traversing and enumerating the devices behind it requires understanding of PCIe bus topology, and re-programming the ATU.

Initial setup of iMX6 PCI controller is as "Root Complex" but the initialization may be incomplete.

The Driver must write all PCI Configuration registers so that Root Complex can meet PCI.ORG requirements.

OAL layer must be modified to:

- consider proper numbering of PCI busses (compliance with B:D:F addressing) and

- reprogram ATU each time a bus beyond the bridge is accessed.

This is a sample topology of a PCIe bus tree:

        HOST

         |

         | bus 0

         |

    RC:  BDF 0.0.0
         |

         | bus 1

         |

BRIDGE:  BDF 1.0.0 for its upstream port
         |

     bus 2 is internal for the bridge, created implicitly for access to downstream ports

                  /                   |                    \

             BDF 2.0.0            BDF 2.1.0             BDF 2.2.0

                 |                    |                     |

                 | bus 3              | bus 4               | bus 5

                 |                    |                     |

           BDF 3.0.*      BDF 4.1.*      (empty)

          endpoint A     endpoint B

For access to the bridge's configuration space at B:D:F 1.0.0, ATU must be programmed for CFG type 0 transaction (ATU[0].iATURC1.TYPE = 4)
For access to all downstream ports on the bridge and beyond, ATU must be programmed for CFG type 1 transaction (ATU[0].iATURC1.TYPE = 5)

Neither devices, nor functions on any bus need to be compact; the numbers may be scattered within the range of devices or functions. Because of this the OAL layer must be able to handle exceptions when attempt to access non-existing PCIe configuration space is made.

Once OAL implementation is complete, a simple Microsoft-provided application can be used to travers the bus and the devices on it - is a little neat utility in the COMMON branch, named "pcienum"

  ...\WINCE700\public\common\oak\drivers\ceddk\test\pcienum\

Building it and running it will print out full information about the devices currently existing on the PCIe bus; this includes all bridges too.

The numbering of busses and devices should reflect the expected topology of the PCIe tree.

It is possible that OAL would not permit even read access to PCI Configuration space. This little addition may be required to OAL IOCTL support for the platform in OALPCICfgRead() function:

    . . .

    switch (dwIoControlCode) {

        case IOCTL_HAL_DDK_CALL:

            if (pInBuf != NULL && nInBufSize >= 4) {

                // only peeking PCIe topology is allowed

                if (IOCTL_OAL_READBUSDATA != *((DWORD*)pInBuf)) {

                    SetLastError(ERROR_ACCESS_DENIED);

                    break;

                }

            }

        }

    . . .

Once this is completed and working, then the BSP is ready to handle PCI resources -- MMIO, I/O and interrupts; and assign them accordingly to bridges and endpoints.

Best luck in your development!

Ilko Dossev