PCIe Throughput Issue between i.MX8MP and LA9310

[IniterWorker]

Hi NXP Team, 

We are trying to achieve 2.52 Gbps (350 MB/s) between the i.MX8MP RC and LA9310 (EP) using DMA write, but our current setup is only achieving around 1.9 Gbps (240 MB/s). 

Current Setup: 

• Link Status: Speed 5 GT/s, Width x1 (overdriven). 

Hardware Configuration: 

• VAR-SOM-MX8MP on Variscite SBC Symphony with a 10-inch M.2 to PCIe x1 connection. 

• LA9310 (DMA write initiator) on a custom board with a PCIe x1 edge connector plugged into the SBC’s PCIe x1 slot. 

• External clock: The SBC hosts a 100 MHz external clock. 

We’ve encountered some issues with the LA9310, a boot delay when booting from an external SPI memory, during negotiation at Gen 2. To resolve this, we added a delay before the PCIe scan process to give the LA9310 enough time (approximately 8 seconds) to complete its boot sequence. However, we were unable to negotiate at Gen 3 speeds. 

Current Software Configuration: 

• Kernel: 6.1.24 from linux-fslc with Variscite patches. 

• Defconfig: CONFIG_PCIEAER=y is enabled. 

• No critical errors are observed during communication. 

スポイラ

[ 2.381195] imx6q-pcie 33800000.pcie: bus scan delay, fixed to 10000 ms...

[ 2.395369] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:

[ 2.402633] imx6q-pcie 33800000.pcie: Parsing ranges property...

[ 2.412208] imx6q-pcie 33800000.pcie: IO 0x001ff80000..0x001ff8ffff -> 0x0000000000

[ 2.420409] imx6q-pcie 33800000.pcie: MEM 0x0018000000..0x001fefffff -> 0x0018000000

[ 2.653100] imx6q-pcie 33800000.pcie: Waiting for 10000 milliseconds for PCIe stabilization...

[ 12.765215] imx6q-pcie 33800000.pcie: Using 32 MSI vectors

[ 12.765305] imx6q-pcie 33800000.pcie: iATU unroll: enabled

[ 12.770809] imx6q-pcie 33800000.pcie: iATU regions: 4 ob, 4 ib, align 64K, limit 16G

[ 12.878914] imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up

[ 12.984876] imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up

[ 12.990479] imx6q-pcie 33800000.pcie: Link up, Gen2

[ 12.995374] imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up

[ 13.001071] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00

[ 13.007452] pci_bus 0000:00: root bus resource [bus 00-ff]

[ 13.012962] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]

[ 13.019148] pci_bus 0000:00: root bus resource [mem 0x18000000-0x1fefffff]

[ 13.026034] pci_bus 0000:00: scanning bus

[ 13.026055] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400

[ 13.032071] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]

[ 13.038354] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]

[ 13.045089] pci 0000:00:00.0: supports D1

[ 13.049112] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold

[ 13.055469] pci 0000:00:00.0: PME# disabled

[ 13.055589] pci 0000:00:00.0: vgaarb: pci_notify

[ 13.057274] pci_bus 0000:00: fixups for bus

[ 13.057280] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 0

[ 13.057335] pci_bus 0000:01: scanning bus

[ 13.057387] pci 0000:01:00.0: [1957:aabb] type 00 class 0x028000

[ 13.063460] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]

[ 13.069753] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0001ffff]

[ 13.076066] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit pref]

[ 13.083371] pci 0000:01:00.0: Max Payload Size set to 128 (was 256, max 256)

[ 13.090582] pci 0000:01:00.0: PME# supported from D0 D3hot

[ 13.096084] pci 0000:01:00.0: PME# disabled

[ 13.096340] pci 0000:01:00.0: vgaarb: pci_notify

[ 13.096384] pci_bus 0000:01: fixups for bus

[ 13.096388] pci_bus 0000:01: bus scan returning with max=01

[ 13.096394] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 1

[ 13.096400] pci_bus 0000:00: bus scan returning with max=ff

[ 13.096416] pci 0000:00:00.0: BAR 14: assigned [mem 0x18000000-0x1dffffff]

[ 13.103306] pci 0000:00:00.0: BAR 0: assigned [mem 0x1e000000-0x1e0fffff]

[ 13.110099] pci 0000:00:00.0: BAR 15: assigned [mem 0x1e100000-0x1e1fffff pref]

[ 13.117439] pci 0000:00:00.0: BAR 6: assigned [mem 0x1e200000-0x1e20ffff pref]

[ 13.124668] pci 0000:01:00.0: BAR 0: assigned [mem 0x18000000-0x1bffffff]

[ 13.131479] pci 0000:01:00.0: BAR 1: assigned [mem 0x1c000000-0x1c01ffff]

[ 13.138281] pci 0000:01:00.0: BAR 2: assigned [mem 0x1e100000-0x1e10ffff 64bit pref]

[ 13.146056] pci 0000:00:00.0: PCI bridge to [bus 01-ff]

[ 13.151286] pci 0000:00:00.0: bridge window [mem 0x18000000-0x1dffffff]

[ 13.158086] pci 0000:00:00.0: bridge window [mem 0x1e100000-0x1e1fffff pref]

[ 13.165387] pcieport 0000:00:00.0: vgaarb: pci_notify

[ 13.165446] pcieport 0000:00:00.0: assign IRQ: got 229

[ 13.165495] imx6q-pcie 33800000.pcie: msi#0 address_hi 0x0 address_lo 0x4401c000

[ 13.165580] pcieport 0000:00:00.0: PME: Signaling with IRQ 230

[ 13.173081] pcieport 0000:00:00.0: AER: enabled with IRQ 230

[ 13.178839] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0xabcd16c3)

[ 13.178849] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100507)

[ 13.178854] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040001)

[ 13.178858] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10000)

[ 13.178862] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x1e000000)

[ 13.178867] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)

[ 13.178871] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0xff0100)

[ 13.178876] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0xf0)

[ 13.178880] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0x1df01800)

[ 13.178884] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1e101e10)

[ 13.178889] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)

[ 13.178893] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)

[ 13.178897] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)

[ 13.178901] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x40)

[ 13.178906] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)

[ 13.178910] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x201e5)

Throughput Experimentation Approach: 

We’ve been measuring throughput using the following method: 

• Once the PCIe link is configured, the DMA is set up to transfer 4 kB of data. After the transfer completes, another transfer is initiated, and this cycle continues. 

• The number of clock cycles is measured between the start and end of each transfer. 

We are currently reading 1.9 Gbps (240 MB/s), which is 100 MB/s lower than expected, based on the AN13164 iMX8MP PCIe Bandwidth Analysis. 

Observations: 

• We have checked the LA9310 register, and it shows Gen 2 speeds. 

Do you have any official or unofficial values or benchmarks that could validate whether our expectations of achieving 2.52 Gbps (350 MB/s) are reasonable with our hardware configuration? Any insights or advice would be greatly appreciated.

Do you have any suggestions or further steps we can take to diagnose or fix the throughput issue? Any insights or advice would be greatly appreciated. 

Thanks,

Best regards

[JosephAtNXP]

Hi,

Thank you for your interest in NXP Semiconductor products, as well for your detail explanation,

I would like to know the Variscite insights on your support case created, moreover you could experiment between two i.MX 8M Plus SOM and base boards, so you can be 100% sure that the processor is capable of such communication with current configs and patches, according to AN13164 as said.

Another strong recommendation is to use a greater buffer, instead of using a dedicated core, AN13164 shows that a greater buffer; equal or greater than 4 MB, shows a Gen3 bandwidth.

Regards

[IniterWorker · Accepted solution]

Hopefully, this topic helps others when choosing a swappable solution for high-bandwidth PCIe interfaces with an EP DMA write initiator.

Variscite Symphony SBC, Gen 3 Negotiation Issue

We have opened a support ticket with Variscite to investigate whether our hardware is capable of negotiating a Gen 3 link without running into signal integrity issues. We're awaiting their insights to confirm if the current hardware setup supports this capability reliably.

Throughput Limitation

From discussions with the NXP team, we have learned about a known issue with the i.MX8MP PCIe controller. It appears the controller does not saturate the PCIe link when the i.MX8MP (as RC) is communicating with an EP that is the DMA write initiator.

Options

• Use Cortex-M7 / i.MX8MP iATU DMA read
• Use Dedicated Linux Core
• Upgrade to a better PCIe controller 

Finally, for our use case, we're thinking about simply upgrading to a better PCIe controller with the i.MX8QuadMax.