AnsweredAssumed Answered

LS2085A: Devices behind PCIe Switch not working

Question asked by Ralf Goebel on Mar 30, 2016
Latest reply on May 12, 2016 by Ralf Goebel

Hi,

I'm using the LS2085A RDB with EAR 6.0 and trying to use PCIe devices behind a PCIe switch. A PCIe expansion board with the Switch PEX8616 is plugged into the x8 connector of the RDB. Standard PCIe cards can be plugged into the expansion board, but I primarily use a custum FPGA board.

First, I had problems to get the devices behind the switch enumerated properly. After undefining CONFIG_PCI_PNP in U-Boot, the enumeration of the devices by the Linux kernel seems to work properly. I think there is a problem if U-Boot only scans the bus tree partially.

But now without CONFIG_PCI_PNP, I think that the devices are not able to generate MSI interrupts for some reason. I tested this with our custom FPGA implementation and driver. Interrupts are working if I plug the FPGA card directly into the RDB board, without using the switch. Other PCIe cards also don't work properly behind the switch, I tried different network cards.

 

This is the bus tree structure:

root@ls2085ardb:~# lspci -t -v

-+-[0001:00]---00.0-[01]--

\-[0000:00]---00.0-[01-05]----00.0-[02-05]--+-04.0-[03]--

                                             +-05.0-[04]--

                                             \-06.0-[05]----00.0  Altera Corporation Device a6e4

 

FPGA device configuration:

root@ls2085ardb:~# lspci -s 0:5:0.0 -vv

0000:05:00.0 Unassigned class [ff00]: Altera Corporation Device a6e4 (rev 01)

        Subsystem: Altera Corporation Device a6e4

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Interrupt: pin A routed to IRQ 215

        Region 0: Memory at 1446400000 (32-bit, non-prefetchable) [size=1M]

        Capabilities: [50] MSI: Enable+ Count=1/8 Maskable- 64bit+

                Address: 0000000006030040  Data: 0000

        Capabilities: [68] MSI-X: Enable- Count=2 Masked-

                Vector table: BAR=0 offset=00080000

                PBA: BAR=0 offset=00080000

        Capabilities: [78] Power Management version 3

                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

        Capabilities: [80] Express (v1) Endpoint, MSI 00

                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+

                        MaxPayload 128 bytes, MaxReadReq 512 bytes

                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

                LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited

                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

        Capabilities: [100 v1] Virtual Channel

                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1

                Arb:    Fixed- WRR32- WRR64- WRR128-

                Ctrl:   ArbSelect=Fixed

                Status: InProgress-

                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

                        Status: NegoPending- InProgress-

        Kernel driver in use: agexpcidrv

 

After the FPGA should have sent one interrupt:

root@ls2085ardb:~# cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7

...

215:          0          0          0          0          0          0          0          0   ITS-MSI 2621440 Edge      agexpcidrv

...

 

I'm not sure where the MSI gets lost and how to debug this problem further. Any help is appreciated.

Thanks,

Ralf

Outcomes