Poor Ethernet performance

chevereau — Fri, 15 Dec 2017 09:09:48 GMT

Hi,

On different design based on IMX6Q processors, we have rather poor ethernet performance without us being able to explain it.

This example is with a module Apalis iMX6 Quad (Toradex Apalis iMX6 Quad 2GB IT V1.1B) and an Ixora Carrier Board (v1.0a), but we have the same problem with all our design based on IMX6, different rootfs (yocto, embedian...), different kernel release (3.10.17, 4.1...).

To resume :

Command	Ethernet link 1Gb (Mo/s)	Local loop (Mo/s)
scp	10	14
nt & netcat	37	45 Note : with a Celeron(x86), kernel 2.6.33, we have 112 Mo/s
iperf3	51	290

Command

Ethernet link 1Gb (Mo/s)

Local loop (Mo/s)

scp

nt & netcat

Note : with a Celeron(x86), kernel 2.6.33, we have 112 Mo/s

iperf3

290

How to explain these differences ? With a 1G Ethernet link, the theoretical maximum speed is 125 MB/s (We know there is a limit on the Ethernet of the IMX6, ~ 400 Mbits/s) . And we do not have a high CPU load.

On local loop, 290 Mo/s with iperf3, only 45 Mo/s with netcat ! where is the problem ?

We performed 3 tests, with an Ethernet link 1Gb and internal local loop :

Transfert test with scp
Transfert test with nc and netcat
iperf3 test

Transfert test - 500 Mo with scp
With another equipment - Ethernet link 1Gb : time scp /tmp/500M root@192.168.1.2:/dev/null The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established. RSA key fingerprint is SHA256:H7v+MGV7dPRPSa8FqE/7GfvTi4TTCfQaPaWED0g+aC0. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.1.2' (RSA) to the list of known hosts. root@192.168.1.2's password: 500M 100% 500MB 11.8MB/s 00:42 real 0m46.451s user 0m31.240s sys 0m9.440s Result : ~10Mo/s with Ethernet link 1Gb
With local loop : time scp /tmp/500M root@127.0.0.1:/dev/null 500M 100% 500MB 14.3MB/s 00:34 real 0m35.209s user 0m29.260s sys 0m7.510s Result : ~14Mo/s with local loop

Transfert test - 500 Mo with scp

With another equipment - Ethernet link 1Gb :

time scp /tmp/500M root@192.168.1.2:/dev/null

The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.

RSA key fingerprint is SHA256:H7v+MGV7dPRPSa8FqE/7GfvTi4TTCfQaPaWED0g+aC0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.1.2' (RSA) to the list of known hosts.

root@192.168.1.2's password:

500M 100% 500MB 11.8MB/s 00:42

real 0m46.451s

user 0m31.240s

sys 0m9.440s

Result : ~10Mo/s with Ethernet link 1Gb

With local loop :

time scp /tmp/500M root@127.0.0.1:/dev/null

500M 100% 500MB 14.3MB/s 00:34

real 0m35.209s

user 0m29.260s

sys 0m7.510s

Result : ~14Mo/s with local loop

With scp, we have encryption. Another way with nc and netcat.

Transfert test - 500 Mo with nc and netcat
With another equipment - Ethernet link 1Gb : netcat –l –p 1234 192.168.1.1 >/tmp/test time nc 192.168.1.2 1234 </tmp/500M real 0m13.522s user 0m0.070s sys 0m6.030s Result : ~37Mo/s with Ethernet link 1Gb
With local loop : netcat -l -p 1234 127.0.0.1 >/tmp/test time nc 127.0.0.1 1234 </tmp/500M real 0m11.049s user 0m0.110s sys 0m2.670s Result : ~45Mo/s with local loop

Transfert test - 500 Mo with nc and netcat

With another equipment - Ethernet link 1Gb :

netcat –l –p 1234 192.168.1.1 >/tmp/test

time nc 192.168.1.2 1234 </tmp/500M

real 0m13.522s

user 0m0.070s

sys 0m6.030s

Result : ~37Mo/s with Ethernet link 1Gb

With local loop :

netcat -l -p 1234 127.0.0.1 >/tmp/test

time nc 127.0.0.1 1234 </tmp/500M

real 0m11.049s

user 0m0.110s

sys 0m2.670s

Result : ~45Mo/s with local loop

Another test, with iperf3.

iperf3 test
With another equipment - Ethernet link 1Gb : iperf3 -s iperf3 -c 192.168.1.2 -M 1500 Connecting to host 192.168.1.2, port 5201 [ 4] local 192.168.1.1 port 47717 connected to 192.168.1.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 49.3 MBytes 414 Mbits/sec 0 246 KBytes [ 4] 1.00-2.00 sec 49.0 MBytes 411 Mbits/sec 0 246 KBytes [ 4] 2.00-3.00 sec 49.0 MBytes 411 Mbits/sec 0 257 KBytes [ 4] 3.00-4.00 sec 48.8 MBytes 410 Mbits/sec 0 288 KBytes [ 4] 4.00-5.00 sec 48.9 MBytes 410 Mbits/sec 0 288 KBytes [ 4] 5.00-6.00 sec 48.8 MBytes 409 Mbits/sec 0 304 KBytes [ 4] 6.00-7.00 sec 48.9 MBytes 410 Mbits/sec 0 327 KBytes [ 4] 7.00-8.00 sec 49.0 MBytes 411 Mbits/sec 0 327 KBytes [ 4] 8.00-9.00 sec 48.9 MBytes 410 Mbits/sec 0 327 KBytes [ 4] 9.00-10.00 sec 49.0 MBytes 411 Mbits/sec 0 495 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 490 MBytes 411 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 489 MBytes 410 Mbits/sec receiver iperf Done. Result : ~51Mo/s with Ethernet link 1Gb
With local loop : iperf3 -s iperf3 -c 127.0.0.1 -M 1500 Connecting to host 127.0.0.1, port 5201 [ 4] local 127.0.0.1 port 45121 connected to 127.0.0.1 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 277 MBytes 2.32 Gbits/sec 0 485 KBytes [ 4] 1.00-2.00 sec 277 MBytes 2.33 Gbits/sec 0 557 KBytes [ 4] 2.00-3.00 sec 278 MBytes 2.33 Gbits/sec 0 557 KBytes [ 4] 3.00-4.00 sec 277 MBytes 2.33 Gbits/sec 0 618 KBytes [ 4] 4.00-5.00 sec 278 MBytes 2.33 Gbits/sec 0 618 KBytes [ 4] 5.00-6.00 sec 277 MBytes 2.32 Gbits/sec 0 618 KBytes [ 4] 6.00-7.00 sec 278 MBytes 2.32 Gbits/sec 0 618 KBytes [ 4] 7.00-8.00 sec 277 MBytes 2.33 Gbits/sec 0 618 KBytes [ 4] 8.00-9.00 sec 276 MBytes 2.32 Gbits/sec 0 618 KBytes [ 4] 9.00-10.00 sec 277 MBytes 2.32 Gbits/sec 0 618 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 2.71 GBytes 2.32 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiver iperf Done. Result : ~290Mo/s with local loop

iperf3 test

With another equipment - Ethernet link 1Gb :

iperf3 -s

iperf3 -c 192.168.1.2 -M 1500

Connecting to host 192.168.1.2, port 5201

[ 4] local 192.168.1.1 port 47717 connected to 192.168.1.2 port 5201

[ ID] Interval Transfer Bandwidth Retr Cwnd

[ 4] 0.00-1.00 sec 49.3 MBytes 414 Mbits/sec 0 246 KBytes

[ 4] 1.00-2.00 sec 49.0 MBytes 411 Mbits/sec 0 246 KBytes

[ 4] 2.00-3.00 sec 49.0 MBytes 411 Mbits/sec 0 257 KBytes

[ 4] 3.00-4.00 sec 48.8 MBytes 410 Mbits/sec 0 288 KBytes

[ 4] 4.00-5.00 sec 48.9 MBytes 410 Mbits/sec 0 288 KBytes

[ 4] 5.00-6.00 sec 48.8 MBytes 409 Mbits/sec 0 304 KBytes

[ 4] 6.00-7.00 sec 48.9 MBytes 410 Mbits/sec 0 327 KBytes

[ 4] 7.00-8.00 sec 49.0 MBytes 411 Mbits/sec 0 327 KBytes

[ 4] 8.00-9.00 sec 48.9 MBytes 410 Mbits/sec 0 327 KBytes

[ 4] 9.00-10.00 sec 49.0 MBytes 411 Mbits/sec 0 495 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-10.00 sec 490 MBytes 411 Mbits/sec 0 sender

[ 4] 0.00-10.00 sec 489 MBytes 410 Mbits/sec receiver

iperf Done.

Result : ~51Mo/s with Ethernet link 1Gb

With local loop :

iperf3 -s

iperf3 -c 127.0.0.1 -M 1500

Connecting to host 127.0.0.1, port 5201

[ 4] local 127.0.0.1 port 45121 connected to 127.0.0.1 port 5201

[ ID] Interval Transfer Bandwidth Retr Cwnd

[ 4] 0.00-1.00 sec 277 MBytes 2.32 Gbits/sec 0 485 KBytes

[ 4] 1.00-2.00 sec 277 MBytes 2.33 Gbits/sec 0 557 KBytes

[ 4] 2.00-3.00 sec 278 MBytes 2.33 Gbits/sec 0 557 KBytes

[ 4] 3.00-4.00 sec 277 MBytes 2.33 Gbits/sec 0 618 KBytes

[ 4] 4.00-5.00 sec 278 MBytes 2.33 Gbits/sec 0 618 KBytes

[ 4] 5.00-6.00 sec 277 MBytes 2.32 Gbits/sec 0 618 KBytes

[ 4] 6.00-7.00 sec 278 MBytes 2.32 Gbits/sec 0 618 KBytes

[ 4] 7.00-8.00 sec 277 MBytes 2.33 Gbits/sec 0 618 KBytes

[ 4] 8.00-9.00 sec 276 MBytes 2.32 Gbits/sec 0 618 KBytes

[ 4] 9.00-10.00 sec 277 MBytes 2.32 Gbits/sec 0 618 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-10.00 sec 2.71 GBytes 2.32 Gbits/sec 0 sender

[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiver

iperf Done.

Result : ~290Mo/s with local loop

Re: Poor Ethernet performance

gabrielgrange — Thu, 21 Dec 2017 07:19:39 GMT

It is strange to have this problem only with iMX6 architecture. I don't know why for the moment.

Re: Poor Ethernet performance

b36401 — Tue, 23 Jan 2018 04:10:26 GMT

Please note that in case of scp there is a lot of encryption overhead that slows the transfer.
And only 1 core of 4 is loaded with this overhead at one moment. So the load may be only 25%.

Have a great day,
Victor

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: Poor Ethernet performance

rans — Tue, 06 Mar 2018 17:13:20 GMT

Hello CHEVEREAU PASCAL,

Is this issue resolved ?

Is there any advatage in using the PCIe ethernet adapter ?

Thanks,

ranran

Re: Poor Ethernet performance

chevereau — Wed, 07 Mar 2018 08:17:54 GMT

Hello,

The answer was given by NXP (Victor Linnik).

For this function, only one core is used. And with one core we are at the limit of performance.

Using another component does not change anything (that's what we do to get around the 400 Mbps limit on native interfaces).

Pascal Chevereau

Re: Poor Ethernet performance

rans — Wed, 07 Mar 2018 12:19:34 GMT

Hi,

"Another component" is the PCIe ethernet adapter ?

So, there is no advantage in using it and the performance is still poor with or without the external adapter , Right ?

Is there a way to increase performance, or is it that Ethernet performance is limited to ~50MB/sec ?

Thank you,

ranran

Re: Poor Ethernet performance

gabrielgrange — Wed, 07 Mar 2018 13:43:47 GMT

We use i210 with PCI ethernet interface with iMX6 and as the result, the performance was not good as well.
It is ok with iperf or gstreamer but standard commands (like scp, nt & netcat) use one cpu core and provide poor performance with iMX6 IT (800MHz).

Regards

G.Grange

i.MX ProcessorsのトピックRe: Poor Ethernet performance

Poor Ethernet performance

Re: Poor Ethernet performance

Re: Poor Ethernet performance

Re: Poor Ethernet performance

Re: Poor Ethernet performance

Re: Poor Ethernet performance

Re: Poor Ethernet performance