2251142_en-US

Yes

Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

Hello,

I’m trying to establish communication between two Imx95 verdin EVK A1 Silicon version boards, through their Aquantia 10 Gbps interfaces. Both boards are running Debian 12 ( linux kernel 6.12.3) with the aquantia10 G firmware (AQR-G4_v5.6.D-AQR_Marvell_NoSwap_XFI_ID44834_VER2068.cld ) properly installed using the nxp installer (aquantia-firmware-utility/aq_api_2_9_7 at master · nxp-qoriq/aquantia-firmware-utility · GitHub). They are physically connected using a Cat6a Ethernet cable.

When I run performance tests using iperf3, I get around 5 Gbps in TCP and 1.2 Gbps in UDP with 0% of loss, even when specifying a target bandwidth of 7 Gbps:

# TCP Test
iperf3 -s # on the first board
iperf3 -c -t 30 # on the second board

# UDP Test
iperf3 -s
iperf3 -c -u -b 7G -t 30

when i try to activate jumbo frame with ip link set dev enp1s0 mtu 9000 but i get an error that i exceded the limit (it’s weird that a 10Gbps interface does not accept jumbo frames)

also i tried to to increase the UDP buffer size, but i get the same bitrame

the CPU load does not exceed 40% when i run iperf3 in both sides
Are there any specific n configurations to apply or additional tools to install in order to reach the maximum throughput (close to 10 Gbps)? is the Aquantia FW version is good? the linux kernel version? the FW installer version???

SOmeone have already tried to use the 10Gbps interface on this Targets???

Regards,

Abdelmonaem

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

1. Try configuring following settings on both of your systems:

cpufreq-set -g performance

sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'

2. If possible use a different reference system as iperf server (e.g. Intel Xeon)

3. iperf3 itself is single-threaded per test stream, try using -P option:
e.g. iperf3 -c -u -b 10G -t 30 -P 6

(6 streams)

4. Check both forward and reverse stream (-R)

iperf3 -c 192.168.1.1 -t 10 -b 10G -u -R

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

Hi,
Thank you for your feedback,
applying your configuration:
cpufreq-set -g performance

sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'

Now i'm able to achieve 10Gbps some times 8,8 or 9,8Gbps on emitter side only and with only iperf and not iperf3, on the receiver side i can only achieve 5.59Gbps with 36% of loss frames, could you help to resolve this issue

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

it will be a minor version upgrade to 6.12.49

BTW, you may also want to consider DPDK or AF_XDP for better throughput?

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

=> TCP vs UDP send performance

When you are using iperf3 to send TCP packets, each TCP packet is 128KB, and the packet will be fragmented by the LSO feature of ENETC Hardware. So you see higher TCP transfer performance.

iperf3 does not enable UDP_SEGMENT when creating UDP sockets. Therefore:
- Each UDP packet is approximately MTU-sized (≈1500 bytes).
- For the same data size, UDP requires to send many more packets than TCP.
More packets → more kernel processing → lower performance compared to TCP with LSO.

=> Why receive is much lower in comparison to send for TCP?

- The TX and RX paths are not symmetrical in the Linux kernel, so the processing time for each RX packet and each TX packet in the kernel is different. Also TCP is using LSO offload in send.
- RSC is not enabled by default in kernel; We need to disable TCP timestamp so that the RSC of ENETC can work properly. Currently, the RSC of i.MX95 is disabled by default.
a) Enable RSC of i.MX95 (receiver side):
ethtool -K eth1 large-receive-offload on
b) Disabled TCP timestamp (sender side) :
sysctl -w net.ipv4.tcp_timestamps=0
sysctl -p /etc/sysctl.conf

After enabling RSC, you will see the TCP performance at receiver side will be increased.

Additionally you can try using Jumbo frames to get higher throughput. (Hope you are running latest release).

# Change MTU to 9000 on both side
ip link set dev eth1 mtu 9000

# Change RX buffer length in the ethernet driver.
ethtool -G eth1 rx-buf-len 16384

We can see between 8 to 10 Gbps UDP RX/TX with multi-stream mode.

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

Hi,

I'm interested only on UDP for the moment and not TCP and i think that the offload
mecanisms are not applicable for UDP
I tried to activate the jumbo frames by setting mtu to 9000 but i get an error that i exceeded the limits which is 1500 (im using the kernel version 6.12.3)

Regards,
Abdelmonaem

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

Ok i will try this and told you what i get,
Could you precise the linux kernel version for LF-Q4?? As i told you i'm on 6.12.3 and i cannot go to a superior version because i have a A1 silicon revision, i need to know if i have to upgrade to B0 revision or not

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

The RSC settings will also improve UDP performance.

The changes/fixes for Jumbo frames in ENETC should be available in upcoming LF-Q4 release in two weeks.

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

With i don't have an issue i can go to 10Gbps sometime 9, 8Bps with iperf in parallel flows with only one flow i'm at 5Gbps , but my big issue is the Udp, in mono flow i'm at 2Gbps with no lost frame it's ok, but in parallel flows i'm at 5,5Gbps with 42% of loss

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

i don't think that AF_XDP or DPDK will help to reduce a 46% of loss to 0%, i verified also the IRQ affinty and i can see that on the 10G interface i have 6 IRQ each one is affected to a CPU, and i don't see any CPU load issue during the test, the maximum CPU load for 1 CPU is 40%, i still don't understand why i keep loosing frames , maybe the fact that the kernel version used does not support jumbo frames ,
did you have any performance tests on the 10G interface in your Side (NXP) i think you should test in your side and see if you have the same issue than mine

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

AF_XDP or DPDK do not use kernel networking stack.

DPDK specially have a different driver and work in userspace only. It is highly optimized for networking and packet processing. It can provide really fast performance for all IP packets. you may check it out at;

Chapter 10: https://www.nxp.com/docs/en/reference-manual/RM00293.pdf

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

this test is performed on which silicon revision A1 or B0??
do you confirm that the kernel version used for this test 6.12.49 is applicable only for B0 or could be applicable on A1???
i'm not seeing this kernel version in your bsp delivery, the last one is 6.12.34 https://www.nxp.com/pages/alpha-beta-bsps-for-microprocessors:IMXPRERELEASES

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

The new kernel tree and changes are available at:

https://github.com/nxp-imx/linux-imx/commits/lf-6.12.49-2.2.0

A1 support has been dropped from newer LF release.

You have following options;

1. Just build the kernel independently and replace the kernel only in your build. (It may work)

2. Ask your marketing contact to replace your boards with B0 - so that you can run LF-Q4'2025 release.

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

Ok got it, but just to be sure, could you do the test with MTU = 1500 at reception side i need to know if you have the same result than me on this new version and also i see on your test you put TX at 1500 and reception at 9000 but this does not change anything it's like you are in 1500 in both side if you need to test jumbo frames you need to be at 9000 in both sides , please redo the tests with following configuration:
1- RX/TX with MTU = 1500 in both sides with mono and muti streams
2- RX/TX with MTU = 9000 in both sides with mono and muti streams

thank you in advance

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

i don't understand that frome those lines:
Single UDP Stream send (1500 MTU): 2 Gbps(this seems at tx with MTU 1500 in single stream you are at 2Gbps)
Multi UDP stream send (1500 MTU): 10 Gbps (this seems at tx with MTU 1500 in multi streams you are at 10Gbps)
Single UDP Stream send (9000 MTU): 8.2 Gbps(this seems at tx with MTU 900 in single stream you are at 8.2Gbps)
Single UDP Stream receive(9000 MTU): 3.9 Gbps(this seems at Rx with MTU 9000 in single stream you are at 3.9Gbps)
Multi UDP Stream receive(9000 MTU): 10Gbps(this seems at Rx with MTU 9000 in Multi streams you are at 2Gbps)

for me i don't see :
Single UDP Stream receive(1500 MTU)
Multi UDP Streams receive(1500 MTU)

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

yes, this how the results were tested.

MTU was same on both side in all cases (either both 1500 or both 9000)

Re: Imx95 verdin EVK, Aquantia10gbps interface limited on 1.2gbps on udp

see attached,

Also note that I am also running it on A1 SOC

UBOOT logs: -

U-Boot 2025.04-g8c1de2e1deca (May 09 2025 - 15:29:26 +0000)

CPU: i.MX95 rev1.1 at 1800MHz
CPU: Automotive temperature grade (-40C to 125C) at 30C
LM Boot reason: sw, origin: 2, errid: 1
LM shutdown reason: sw, origin: 2, errid: 1
Model: NXP i.MX95 19X19 board
DRAM: 15.8 GiB

--- SM logs

>$ info
SM Version = Build 633, Commit c37b26da
SM Config = mx95evk, mSel=0
Board = i.MX95 EVK, attr=0x00000000
Silicon = i.MX95 A1
Boot mode = normal
Boot device = MMC1
Boot stage = primary
Boot set = 1
ECID = 0x6E5F04BA0000000500041D0899123F81
PMIC 0 (0x08) = 0x20, 0x09, 0x10, 0x00, 0x01
PMIC 1 (0x2A) = 0x54, 0x22, 0x00, 0x0B
PMIC 2 (0x29) = 0x55, 0x22, 0x00, 0x0A
Compiler = gcc 14.2.1 20241119