i.MX8MM Ethernet TX Bandwidth Fluctuations

[friederschrempf]

Hi,

we have a custom board with i.MX8M-Mini and ethernet PHY Microsemi VSC8531 connected via RGMII. While the interface works fine in general for link speeds of 10, 100 and 1000 MBit/s, there's an issue with the bandwith sometimes dropping to 50% of the expected/nominal value.

This issue can be seen with link speeds of 100 or 1000 MBit/s. In both cases the TX bandwidth is sometimes as expected (~93 MBit/s for link speed 100, ~900 MBit/s for link speed 1000) and sometimes it drops to much lower values like in the examples below (~45 MBit/s for link speed 100, ~600 MBit/s for link speed 1000).

The same behavior can be seen using a simple test with netcat, so it's not an iperf issue.

Looking at the statistics of the interface in Linux, there's nothing conspicuous. No packets are dropped and no errors reported.

So far we have tested and reproduced this behavior with Linux Kernel 5.4 and 5.10.

If anyone has any idea of what might be the cause of this, that would be great!

Thanks for your help!

Frieder

root@kontron-mx8mm:~# iperf3 -c 192.168.1.10 --bidir
Connecting to host 192.168.1.10, port 5201
[  5] local 192.168.1.20 port 53574 connected to 192.168.1.10 port 5201
[  7] local 192.168.1.20 port 53576 connected to 192.168.1.10 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr Cwnd
[  5][TX-C]   0.00-1.00   sec  5.79 MBytes  48.6 Mbits/sec    0 80.6 KBytes
[  7][RX-C]   0.00-1.00   sec  10.9 MBytes  91.8 Mbits/sec
[  5][TX-C]   1.00-2.00   sec  5.28 MBytes  44.3 Mbits/sec    0 89.1 KBytes
[  7][RX-C]   1.00-2.00   sec  11.1 MBytes  92.9 Mbits/sec
[  5][TX-C]   2.00-3.00   sec  5.34 MBytes  44.8 Mbits/sec    0 93.3 KBytes
[  7][RX-C]   2.00-3.00   sec  11.1 MBytes  92.8 Mbits/sec
[  5][TX-C]   3.00-4.00   sec  5.16 MBytes  43.3 Mbits/sec    0 93.3 KBytes
[  7][RX-C]   3.00-4.00   sec  11.1 MBytes  92.9 Mbits/sec
[  5][TX-C]   4.00-5.00   sec  5.34 MBytes  44.8 Mbits/sec    0 93.3 KBytes
[  7][RX-C]   4.00-5.00   sec  11.1 MBytes  92.8 Mbits/sec
[  5][TX-C]   5.00-6.00   sec  5.41 MBytes  45.4 Mbits/sec    0 93.3 KBytes
[  7][RX-C]   5.00-6.00   sec  11.1 MBytes  92.9 Mbits/sec
[  5][TX-C]   6.00-7.00   sec  5.53 MBytes  46.4 Mbits/sec    0 130 KBytes
[  7][RX-C]   6.00-7.00   sec  11.1 MBytes  92.8 Mbits/sec
[  5][TX-C]   7.00-8.00   sec  5.28 MBytes  44.3 Mbits/sec    0 130 KBytes
[  7][RX-C]   7.00-8.00   sec  11.1 MBytes  92.8 Mbits/sec
[  5][TX-C]   8.00-9.00   sec  5.28 MBytes  44.3 Mbits/sec    0 130 KBytes
[  7][RX-C]   8.00-9.00   sec  11.1 MBytes  92.8 Mbits/sec
[  5][TX-C]   9.00-10.00  sec  5.28 MBytes  44.3 Mbits/sec    0 130 KBytes
[  7][RX-C]   9.00-10.00  sec  11.1 MBytes  92.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  53.7 MBytes  45.0 Mbits/sec 0             sender
[  5][TX-C]   0.00-10.01  sec  53.2 MBytes  44.6 Mbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec   112 MBytes  94.1 Mbits/sec 0             sender
[  7][RX-C]   0.00-10.01  sec   111 MBytes  92.7 Mbits/sec                  receiver

iperf Done.

root@kontron-mx8mm:~# iperf3 -c 192.168.1.10 --bidir
Connecting to host 192.168.1.10, port 5201
[  5] local 192.168.1.20 port 53568 connected to 192.168.1.10 port 5201
[  7] local 192.168.1.20 port 53570 connected to 192.168.1.10 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr Cwnd
[  5][TX-C]   0.00-1.00   sec  74.4 MBytes   624 Mbits/sec    2 438 KBytes
[  7][RX-C]   0.00-1.00   sec   106 MBytes   885 Mbits/sec
[  5][TX-C]   1.00-2.00   sec  70.5 MBytes   591 Mbits/sec   24 321 KBytes
[  7][RX-C]   1.00-2.00   sec   108 MBytes   905 Mbits/sec
[  5][TX-C]   2.00-3.00   sec  69.3 MBytes   582 Mbits/sec    0 389 KBytes
[  7][RX-C]   2.00-3.00   sec   107 MBytes   895 Mbits/sec
[  5][TX-C]   3.00-4.00   sec  74.2 MBytes   622 Mbits/sec   14 385 KBytes
[  7][RX-C]   3.00-4.00   sec   108 MBytes   906 Mbits/sec
[  5][TX-C]   4.00-5.00   sec  69.2 MBytes   581 Mbits/sec    0 434 KBytes
[  7][RX-C]   4.00-5.00   sec   107 MBytes   897 Mbits/sec
[  5][TX-C]   5.00-6.00   sec  70.3 MBytes   590 Mbits/sec   14 191 KBytes
[  7][RX-C]   5.00-6.00   sec   108 MBytes   902 Mbits/sec
[  5][TX-C]   6.00-7.00   sec  68.0 MBytes   571 Mbits/sec    0 354 KBytes
[  7][RX-C]   6.00-7.00   sec   107 MBytes   900 Mbits/sec
[  5][TX-C]   7.00-8.00   sec  73.9 MBytes   620 Mbits/sec    1 356 KBytes
[  7][RX-C]   7.00-8.00   sec   106 MBytes   892 Mbits/sec
[  5][TX-C]   8.00-9.00   sec  72.9 MBytes   611 Mbits/sec    2 310 KBytes
[  7][RX-C]   8.00-9.00   sec   108 MBytes   905 Mbits/sec
[  5][TX-C]   9.00-10.00  sec  74.4 MBytes   624 Mbits/sec    0 404 KBytes
[  7][RX-C]   9.00-10.00  sec   108 MBytes   903 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec   717 MBytes   602 Mbits/sec 57             sender
[  5][TX-C]   0.00-10.00  sec   715 MBytes   599 Mbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  1.05 GBytes   901 Mbits/sec 0             sender
[  7][RX-C]   0.00-10.00  sec  1.05 GBytes   899 Mbits/sec                  receiver

iperf Done.  

[friederschrempf · Accepted solution]

So once more the kernel community turned out to be of much more avail than the NXP community. After I had posted my issue on the mailing lists it quickly turned out, that other people with different hardware were seeing the same problem.

Finally Joakim Zhang was kind enough to have a closer look and came up with the information that this is a known issue in relation to AVB support. There are three TX queues, but only the first one provides the full bandwidth. Queue 1 and 2 are limited to 50% of the nominal bandwidth. As the driver "randomly" chooses the queue, in about two thirds of the cases it selects one of the limited queues.

NXP has a patch in their downstream kernel, that solves this by preferring queue 0 for untagged, non-AVB traffic, but apparently nobody ever cared to upstream this so far.

Another quick workaround is to just disable the other queues by setting fsl,num-tx-queues = <1> in the devicetree, which obviously has the downside of not being able to handle the AVB traffic as intended.

[jimmychan]

Do you have the EVK and try the same test?

[friederschrempf]

We have the EVK in the office. When I have some time I will try to test on that board.

[friederschrempf]

I couldn't test on the NXP EVK so far, but we have some other hardware that uses the same Kontron i.MX8MM SoM, but with a KSZ8081 PHY connected via RMII (instead of RGMII) and I can see the same behavior there.