topic Re: How can we increase PCI burst size on LS1028A ? in Layerscape

How can we increase PCI burst size on LS1028A ?

tne — Fri, 29 Oct 2021 09:14:56 GMT

How can we increase PCI burst size on LS1028A ?

Hi,

We are developping an FPGA connected to the LS1028A over PCIe. We are currently facing performances issues regarding to burst size. Our problem is that we cannot make burst writes of more than four 32b words: 16 bytes instead of 128.

How can we improve the situation ?
We plan to use DMA for PCIe accesses, will that do ?

The `lspci` command output gives us the PCIe root complex configuration:

# lspci -vv -s 0001:00:00.0
0001:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 82c1 (rev 10) (prog-if 00 [Normal decod)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PE-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 255
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: None
        Prefetchable memory behind bridge: None
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PE-
        Expansion ROM at 8840000000 [disabled] [size=2K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIF+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- Complian-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhas-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- Uns-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- Uns-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- Uns-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn- NFERptEn- FERptEn-
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 1
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Secondary PCI Express <?>

As one can see, the bus Capability is correctly set up:

DevCap: MaxPayload 256 bytes

And what we needed for the MAX_PAYLOAD_SIZE and MAX_READ_SIZE field seems also to be the default:

DevCtl: [...] MaxPayload 128 bytes, MaxReadReq 512 bytes

Re: How can we increase PCI burst size on LS1028A ?

yipingwang — Thu, 04 Nov 2021 07:24:42 GMT

From the default LSDK , the two parameters(MAX_PAYLOAD_SIZE and MAX_READ_SIZE) are shown as their default reset value as listed in page 2983 of the LS1028RM, so it is not modified via the system boot by any code, it is the reason why the output of lspci show the results like:

MaxPayload 128 bytes, MaxReadReq 512 bytes (default reset values of the register)

If customer want to modify these parameters, they may try to modify PCI Express Device Control Register (Device_Control_Register) during the system boot and then have a try for performance tuning

Re: How can we increase PCI burst size on LS1028A ?

tne — Mon, 08 Nov 2021 06:52:36 GMT

Thank you for your reply.

As stated above, the default values for MAX_PAYLOAD_SIZE and MAX_READ_SIZE are just fine for us and we don't need to change their values. However our problem is that we cannot make burst writes of more than four 32b words: 16 bytes instead of 128.

We mmap() the device BAR0 memory into the application and access it with directe reads/writes or memcpy().

Re: How can we increase PCI burst size on LS1028A ?

yipingwang — Tue, 09 Nov 2021 07:47:19 GMT

May I know if you were using host CPU or DMA engine to do the data transaction via PCIe interface? what is your target of the data width?

With qDMA's help, the PCIe transaction width could be up to 256B per DMA operation.

Re: How can we increase PCI burst size on LS1028A ?

tne — Tue, 09 Nov 2021 09:36:10 GMT

Hi.

We are actually using the CPU to do the data transactions (through libc's implementation of memcpy() that uses ARM instructions). We are planning to implement DMA but we would like to maximize the performances with the actual implementation in order to have a good basis for performances comparisons.

Can you tell me more about qDMA ? What is its differences with common DMA ?

Re: How can we increase PCI burst size on LS1028A ?

yipingwang — Thu, 11 Nov 2021 06:16:26 GMT

I think DMA could be your option to improve the performance since it could support up to 256B width per transaction.

qDMA controller on ls1028 could access the PCIe bus, so that it could be used to help for the PCIe transaction implementation qDMA is a DMA controller implemented within the LS1028 SOC. the details could reference the chapter 27 of the LS1028RM.

There is also some reference code in LSDK based on other platform, you may reference the dpdk-qdma-demo application for its reference usage(chapter 9.2 of LSDKUG21.08)

Re: How can we increase PCI burst size on LS1028A ?

tne — Tue, 16 Nov 2021 10:13:20 GMT

Thank you yipingwang.

I suspected that this was the only possibility to overcome the CPU instructions limitation. We're implementing DMA transfert mode and testing. Thanks again.