Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp Hello, I’m trying to establish communication between two Imx95 verdin EVK A1 Silicon version boards, through their Aquantia 10 Gbps interfaces. Both boards are running Debian 12 ( linux kernel 6.12.3) with the aquantia10 G firmware (AQR-G4_v5.6.D-AQR_Marvell_NoSwap_XFI_ID44834_VER2068.cld ) properly installed using the nxp installer (aquantia-firmware-utility/aq_api_2_9_7 at master · nxp-qoriq/aquantia-firmware-utility · GitHub). They are physically connected using a Cat6a Ethernet cable. When I run performance tests using iperf3, I get around 5 Gbps in TCP and 1.2 Gbps in UDP with 0% of loss, even when specifying a target bandwidth of 7 Gbps: # TCP Test iperf3 -s # on the first board iperf3 -c -t 30 # on the second board # UDP Test iperf3 -s iperf3 -c -u -b 7G -t 30 when i try to activate jumbo frame with ip link set dev enp1s0 mtu 9000 but i get an error that i exceded the limit (it’s weird that a 10Gbps interface does not accept jumbo frames) also i tried to to increase the UDP buffer size, but i get the same bitrame the CPU load does not exceed 40% when i run iperf3 in both sides Are there any specific n configurations to apply or additional tools to install in order to reach the maximum throughput (close to 10 Gbps)? is the Aquantia FW version is good? the linux kernel version? the FW installer version??? SOmeone have already tried to use the 10Gbps interface on this Targets??? Regards, Abdelmonaem Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp 1. Try configuring following settings on both of your systems:
cpufreq-set -g performance
sysctl -w net.core.rmem_max=26214400 sysctl -w net.core.wmem_max=26214400 sysctl -w net.core.netdev_max_backlog=250000 sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216' sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'
2. If possible use a different reference system as iperf server (e.g. Intel Xeon)
3. iperf3 itself is single-threaded per test stream, try using -P option: e.g. iperf3 -c -u -b 10G -t 30 -P 6
(6 streams)
4. Check both forward and reverse stream (-R)
iperf3 -c 192.168.1.1 -t 10 -b 10G -u -R
Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp Hi, Thank you for your feedback, applying your configuration: cpufreq-set -g performance sysctl -w net.core.rmem_max=26214400 sysctl -w net.core.wmem_max=26214400 sysctl -w net.core.netdev_max_backlog=250000 sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216' sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216' Now i'm able to achieve 10Gbps some times 8,8 or 9,8Gbps on emitter side only and with only iperf and not iperf3, on the receiver side i can only achieve 5.59Gbps with 36% of loss frames, could you help to resolve this issue Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp it will be a minor version upgrade to 6.12.49
BTW, you may also want to consider DPDK or AF_XDP for better throughput? Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp => TCP vs UDP send performance
When you are using iperf3 to send TCP packets, each TCP packet is 128KB, and the packet will be fragmented by the LSO feature of ENETC Hardware. So you see higher TCP transfer performance.
iperf3 does not enable UDP_SEGMENT when creating UDP sockets. Therefore: - Each UDP packet is approximately MTU-sized (≈1500 bytes). - For the same data size, UDP requires to send many more packets than TCP. More packets → more kernel processing → lower performance compared to TCP with LSO.
=> Why receive is much lower in comparison to send for TCP?
- The TX and RX paths are not symmetrical in the Linux kernel, so the processing time for each RX packet and each TX packet in the kernel is different. Also TCP is using LSO offload in send. - RSC is not enabled by default in kernel; We need to disable TCP timestamp so that the RSC of ENETC can work properly. Currently, the RSC of i.MX95 is disabled by default. a) Enable RSC of i.MX95 (receiver side): ethtool -K eth1 large-receive-offload on b) Disabled TCP timestamp (sender side) : sysctl -w net.ipv4.tcp_timestamps=0 sysctl -p /etc/sysctl.conf After enabling RSC, you will see the TCP performance at receiver side will be increased.
Additionally you can try using Jumbo frames to get higher throughput. (Hope you are running latest release).
# Change MTU to 9000 on both side ip link set dev eth1 mtu 9000
# Change RX buffer length in the ethernet driver. ethtool -G eth1 rx-buf-len 16384
We can see between 8 to 10 Gbps UDP RX/TX with multi-stream mode. Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp Hi, I'm interested only on UDP for the moment and not TCP and i think that the offload mecanisms are not applicable for UDP I tried to activate the jumbo frames by setting mtu to 9000 but i get an error that i exceeded the limits which is 1500 (im using the kernel version 6.12.3) Regards, Abdelmonaem Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp Ok i will try this and told you what i get, Could you precise the linux kernel version for LF-Q4?? As i told you i'm on 6.12.3 and i cannot go to a superior version because i have a A1 silicon revision, i need to know if i have to upgrade to B0 revision or not Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp The RSC settings will also improve UDP performance.
The changes/fixes for Jumbo frames in ENETC should be available in upcoming LF-Q4 release in two weeks.
Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp With i don't have an issue i can go to 10Gbps sometime 9, 8Bps with iperf in parallel flows with only one flow i'm at 5Gbps , but my big issue is the Udp, in mono flow i'm at 2Gbps with no lost frame it's ok, but in parallel flows i'm at 5,5Gbps with 42% of loss Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp i don't think that AF_XDP or DPDK will help to reduce a 46% of loss to 0%, i verified also the IRQ affinty and i can see that on the 10G interface i have 6 IRQ each one is affected to a CPU, and i don't see any CPU load issue during the test, the maximum CPU load for 1 CPU is 40%, i still don't understand why i keep loosing frames , maybe the fact that the kernel version used does not support jumbo frames , did you have any performance tests on the 10G interface in your Side (NXP) i think you should test in your side and see if you have the same issue than mine Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp AF_XDP or DPDK do not use kernel networking stack.
DPDK specially have a different driver and work in userspace only. It is highly optimized for networking and packet processing. It can provide really fast performance for all IP packets. you may check it out at;
Chapter 10: https://www.nxp.com/docs/en/reference-manual/RM00293.pdf Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp this test is performed on which silicon revision A1 or B0?? do you confirm that the kernel version used for this test 6.12.49 is applicable only for B0 or could be applicable on A1??? i'm not seeing this kernel version in your bsp delivery, the last one is 6.12.34 https://www.nxp.com/pages/alpha-beta-bsps-for-microprocessors:IMXPRERELEASES Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp The new kernel tree and changes are available at:
https://github.com/nxp-imx/linux-imx/commits/lf-6.12.49-2.2.0
A1 support has been dropped from newer LF release.
You have following options;
1. Just build the kernel independently and replace the kernel only in your build. (It may work)
2. Ask your marketing contact to replace your boards with B0 - so that you can run LF-Q4'2025 release.
Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp Ok got it, but just to be sure, could you do the test with MTU = 1500 at reception side i need to know if you have the same result than me on this new version and also i see on your test you put TX at 1500 and reception at 9000 but this does not change anything it's like you are in 1500 in both side if you need to test jumbo frames you need to be at 9000 in both sides , please redo the tests with following configuration: 1- RX/TX with MTU = 1500 in both sides with mono and muti streams 2- RX/TX with MTU = 9000 in both sides with mono and muti streams thank you in advance Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp i don't understand that frome those lines: Single UDP Stream send (1500 MTU): 2 Gbps(this seems at tx with MTU 1500 in single stream you are at 2Gbps) Multi UDP stream send (1500 MTU): 10 Gbps (this seems at tx with MTU 1500 in multi streams you are at 10Gbps) Single UDP Stream send (9000 MTU): 8.2 Gbps(this seems at tx with MTU 900 in single stream you are at 8.2Gbps) Single UDP Stream receive(9000 MTU): 3.9 Gbps(this seems at Rx with MTU 9000 in single stream you are at 3.9Gbps) Multi UDP Stream receive(9000 MTU): 10Gbps(this seems at Rx with MTU 9000 in Multi streams you are at 2Gbps) for me i don't see : Single UDP Stream receive(1500 MTU) Multi UDP Streams receive(1500 MTU) Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp yes, this how the results were tested.
MTU was same on both side in all cases (either both 1500 or both 9000) Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp see attached,
Also note that I am also running it on A1 SOC
UBOOT logs: -
U-Boot 2025.04-g8c1de2e1deca (May 09 2025 - 15:29:26 +0000)
CPU: i.MX95 rev1.1 at 1800MHz CPU: Automotive temperature grade (-40C to 125C) at 30C LM Boot reason: sw, origin: 2, errid: 1 LM shutdown reason: sw, origin: 2, errid: 1 Model: NXP i.MX95 19X19 board DRAM: 15.8 GiB
--- SM logs
>$ info SM Version = Build 633, Commit c37b26da SM Config = mx95evk, mSel=0 Board = i.MX95 EVK, attr=0x00000000 Silicon = i.MX95 A1 Boot mode = normal Boot device = MMC1 Boot stage = primary Boot set = 1 ECID = 0x6E5F04BA0000000500041D0899123F81 PMIC 0 (0x08) = 0x20, 0x09, 0x10, 0x00, 0x01 PMIC 1 (0x2A) = 0x54, 0x22, 0x00, 0x0B PMIC 2 (0x29) = 0x55, 0x22, 0x00, 0x0A Compiler = gcc 14.2.1 20241119
記事全体を表示
