LS2085A: Devices behind PCIe Switch not working

[ralfgoebel]

Hi,

I'm using the LS2085A RDB with EAR 6.0 and trying to use PCIe devices behind a PCIe switch. A PCIe expansion board with the Switch PEX8616 is plugged into the x8 connector of the RDB. Standard PCIe cards can be plugged into the expansion board, but I primarily use a custum FPGA board.

First, I had problems to get the devices behind the switch enumerated properly. After undefining CONFIG_PCI_PNP in U-Boot, the enumeration of the devices by the Linux kernel seems to work properly. I think there is a problem if U-Boot only scans the bus tree partially.

But now without CONFIG_PCI_PNP, I think that the devices are not able to generate MSI interrupts for some reason. I tested this with our custom FPGA implementation and driver. Interrupts are working if I plug the FPGA card directly into the RDB board, without using the switch. Other PCIe cards also don't work properly behind the switch, I tried different network cards.

This is the bus tree structure:

root@ls2085ardb:~# lspci -t -v

-+-[0001:00]---00.0-[01]--

\-[0000:00]---00.0-[01-05]----00.0-[02-05]--+-04.0-[03]--

                                             +-05.0-[04]--

                                             \-06.0-[05]----00.0  Altera Corporation Device a6e4

FPGA device configuration:

root@ls2085ardb:~# lspci -s 0:5:0.0 -vv

0000:05:00.0 Unassigned class [ff00]: Altera Corporation Device a6e4 (rev 01)

        Subsystem: Altera Corporation Device a6e4

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Interrupt: pin A routed to IRQ 215

        Region 0: Memory at 1446400000 (32-bit, non-prefetchable) [size=1M]

        Capabilities: [50] MSI: Enable+ Count=1/8 Maskable- 64bit+

                Address: 0000000006030040  Data: 0000

        Capabilities: [68] MSI-X: Enable- Count=2 Masked-

                Vector table: BAR=0 offset=00080000

                PBA: BAR=0 offset=00080000

        Capabilities: [78] Power Management version 3

                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

        Capabilities: [80] Express (v1) Endpoint, MSI 00

                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+

                        MaxPayload 128 bytes, MaxReadReq 512 bytes

                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

                LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited

                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

        Capabilities: [100 v1] Virtual Channel

                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1

                Arb:    Fixed- WRR32- WRR64- WRR128-

                Ctrl:   ArbSelect=Fixed

                Status: InProgress-

                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

                        Status: NegoPending- InProgress-

        Kernel driver in use: agexpcidrv

After the FPGA should have sent one interrupt:

root@ls2085ardb:~# cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7

...

215:          0          0          0          0          0          0          0          0   ITS-MSI 2621440 Edge      agexpcidrv

...

I'm not sure where the MSI gets lost and how to debug this problem further. Any help is appreciated.

Thanks,

Ralf

[bpe]

u-Boot normally recognizes and properly handles switch-based PCIe topology.

If neither u-Boot nor Linux generic code is able to bring it up propery,

check if any custom hardware initialization is necessary for the switch

to make standard configuration headers visible. Sometimes enumeration

fails because the switch is unable to get out of reset, or does it too

late. Try playing with CONFIG_PCI_BOOTDELAY and CONFIG_PCI_SCAN_SHOW

u-Boot config macros:

http://lists.denx.de/pipermail/u-boot/2010-October/080773.html

http://lists.denx.de/pipermail/u-boot/2010-March/068942.html

BTW, the lspci output you provided shows that the driver only registered

a leagcy PCI line A interrupt. This is not an error by itself, but

it is not MSI.

Have a great day,
Platon

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[ralfgoebel]

Hi,

After digging a lot within the kernel sources, I was finally able to fix the problem:

First, I discovered that the GIC Device ID used for setting up MSI in its_msi_prepare() in drivers/irqchip/irq-gic-v3-its.c was different between the working and the not working cases. The LS2 uses a stream ID for mapping of MSIs to LPIs. For the working case, the function pci_for_each_dma_alias() returned a valid stream ID with low values. In the not working case, the function simply returned the PCI Requester ID, e.g. 0x400 for PCI device 0000:04:00.0.

The reason for this was that the function ls_pcie_probe() in drivers/pci/host/pci-layerscape.c only allocated four stream IDs for each PCIe controller. But the ARM mmu code requests a stream ID for each PCI device (including all bridges) and set_pcie_streamid_translation() silently fails if no stream IDs are left!

The used PCIe switch has four internal bridges. Therefore, no IDs were left for the connected devices behind the switch.

In order to fix the problem, I decided to assign 8 stream IDs to two PCIe controllers, because the system only uses two controllers. The Stream ID assignments are defined in the U-Boot file arch/arm/include/asm/arch-fsl-lsch3/ls2080a_stream_id.h.

[ralfgoebel]

In U-Boot, the enumeration process finds the first bridge device inside the PCIe switch, but it doesn't find the other bridge devices on the next bus. I think the problem is located in ls_pcie_addr_valid() which only allows access to the first device on each bus:

static int ls_pcie_addr_valid(struct pci_controller *hose, pci_dev_t d)
{
    if (PCI_DEV(d) > 0)
        return -EINVAL;
    ...

After changing the if statement, all connected devices are now listed by U-Boot:

if (PCI_BUS(d) <= (hose->first_busno + 1) && PCI_DEV(d) > 0)
     return -EINVAL;

But I still cant't receive any interrupts with linux.

The lspci output always shows the PCI line A interrupt information, even when not using the PCIe switch and MSI is working properly. The output also shows that MSI is enabled and has a valid destination address.

Thanks,

Ralf