Hello,
I got a question on PCIe speeds that I am able to achieve and what could their limiting factor be.
Test is as follows, got a Toradex iMX8QM on a custom carrier board which has a 1x PCIe Gen3 slot which us used to connect a intel X550 10Gb NIC.
NIC works on Gen2 speeds which the module and card correctly negotiate and pcie link runs on Gen2 speeds (max troughput of 4Gb/s).
Speed is tested using iperf3, using the same network card in host PC and connecting them directly. Both NICs were confirmed to run at their full speed so that is not the issue. Server would be the PC and client the module.
When running in upload test I was able to achieve speeds of 3.1Gb/s. Adding pci performance to boot args and playing around with jumbo frames I was able to achieve 3.5Gb/s.
When running in reverse download test I was able to achieve speeds of 1.5Gb/s. With before mentioned boot args and jumbo frames I was able to achieve 1.6Gb/s but sometimes the test wouldnt have any data troughput if jumbo frames were too big. Around 2200 seems the sweetspot, going any higher the test would report 0 bw or speed would be lower than original.
I have found some pdf, from nxp, about pci speed which had higher numbers that those for download test.
I have also found this post which has similar results but they don't match the pattern when on Gen3.
Both tests had 1 cpu core on 100% when running. Upload test ran using 4 connections and download test yielded highest numbers when runnign with 1 connection.
Also tried with iperf2 but its higly unstable. Getting same numbers for upload and for download i get around 1.8Gb/s but as soon as its ran by 2 connections the download side would lose connection, server would be reporting different transfer speeds or the server would already finish sending and few seconds later the client would finish.
Also did a loopback test where the server and client were both on module and it pushed trough 7.6GB/s both directions.
The number 1.5Gb/s also troubles me for other reasons. Its also the speed that the test return on Gen1 pcie link.
Is there anything that can be done or is this some hardware/firmware limitation of the bridge?
For our purposes where this will be used we need high download speeds. In terms of dw/ul ratio it would be 2:1. We got plenty of upload but would like to achieve download speeds preferably over 2.5Gb/s.
I do have a Aquantia Qnap QXG-5G1T-111C (Gen3 x1) but ever since Marvell acquired them the driver wasnt updated and its faulty on ARM causing module hard freezeups with failed memory page allocations.
Any help is appreciated.
root@apalis-imx8-07307404:~# uname -a
Linux apalis-imx8-07307404 5.4.193-5.7.0-devel+git.f78299297185
root@apalis-imx8-07307404:~# cat /etc/os-release
ID=tdx-xwayland
NAME="TDX Wayland with XWayland"
VERSION="5.7.0-devel-20221013091530+build.0 (dunfell)"
VERSION_ID=5.7.0-devel-20221013091530-build.0
PRETTY_NAME="TDX Wayland with XWayland 5.7.0-devel-20221013091530+build.0 (dunfell)"
DISTRO_CODENAME="dunfell"
Device tree was modified by using the following overlay.
/dts-v1/;
/plugin/;
&pciea {
status = "okay";
fsl,max-link-speed = <3>;
};
Best Regards,
Aleš
