10G Port speed on LS1046ARDB

[__OTL__]

Hi 

We are testing 10G port speed on LS1046ARDB and the speed did not meet 10G.

• We conducted an iperf measurement using the software downloaded from the NXP server for the 10GLAN, but the speed wasn't very high.
• When we switched from LS1046ARDB to a Windows Desktop in the same environment, we got over 9Gbps, so the environment doesn't seem to be the problem.
• Disabling CONFIG_NETFILTER and CONFIG_CPU_FREQ led to some improvement, but the speed only increased by about 0.2Gbps.

Details of the measurement results:

• Hardware: LS1046ARDB
• Software: The LSDK-2108 prebuild binary downloaded with the following command: $flex-installer -i download -m ls1046ardb -f firmware_ls1046ardb_qspiboot.img

Measurement environment:

• Server: LS1046ARDB (copper 10GLAN)
  • iperf 3.6 (cJSON 1.5.2)
  • command: $iperf3 -s
• Client: Ubuntu Desktop
  • iperf 3.7 (cJSON 1.5.2)
  • command:
    • $iperf3 -c XX.XX.XX.XX --get-server-output
    • $iperf3 -c XX.XX.XX.XX --get-server-output -R
    • $iperf3 -c XX.XX.XX.XX --get-server-output -u -b 0
    • $iperf3 -c XX.XX.XX.XX --get-server-output -u -b 0 -R

Only TCP tx reached about 7Gbps, while rx and UDP were about 2Gbps.

Attached the logs.

[stadium_aquino]

I was able to get 10G speed with 9000 MTU and 97% of 10G speed with 1500 MTU with the following setup:

I used two PCs with ixgbe cards to generate and receive traffic. I set up the LS1046 to route packets between the PCs. I tested with a direct connection between the PCs first and verified that I got 10G throughput.

On one PC I used the following command. You can find this in the Linux kernel source:

samples/pktgen/pktgen_sample06_numa_awared_queue_irq_affinity.sh -i eth0 -d <ip_address> -m <mac_address> -n 0 -t $(nproc) -s <frame_size>

 The IP address and MAC address should be for the LS1046. On the other PC I ran the following command:

tcpstat -i eth0 -p -o 'Time:%S\tpps=%p\tavg=%a\tstddev=%d\tbps=%b\n' 5

This will print out ingress traffic statistics every 5 seconds. Note that "bps" only includes data bytes. That is, it does not include L1/L2 overhead.

Although I didn't get full 10G throughput at 1500 MTU, I suspect that it is possible. I didn't configure any kind of hardware offloading on the LS1046.

[__OTL__]

Hi @yipingwang, thanks for your quick comment!
I tested again along with your comment(fmc and iperf option), then TCP RX result is significant improved to 9.06Gbps from about 2.6Gbps, TCP TX keeps 7Gbps but UDP TX/RX is still lower than half ot 10Gbps.
Is this your expected result? or should be more high throughput result?

• Hardware: LS1046ARDB
• Software: The LSDK-2108 prebuild binary downloaded with the following command: $flex-installer -i download -m ls1046ardb -f firmware_ls1046ardb_qspiboot.img

Measurement environment:

• Server: LS1046ARDB (copper 10GLAN)
  • iperf 3.6 (cJSON 1.5.2)
  • command:
    • $fmc -c /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/config.xml -p /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/policy_ipv4.xml -a
      $iperf3 -s //cannot use "u" option for server side
• Client: Ubuntu Desktop
  • iperf 3.7 (cJSON 1.5.2)
  • command:
    • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30
    • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30 -R
    • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30 -u -b 10G
    • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30 -u -b 10G -R
• result result_RDB_20230728.zip
  • TCP tx keeps 7Gbps, rx improved to about 9Gbps
  • UDP tx keeps about 2Gbps, rx improved to about 4Gbps

[yipingwang]

Please deploy ubuntu main rootfs to SD card, then boot LS1046ARDB Linux and rootfs from SD card and try whether the iperf performance would be improved.

On your host PC:

$ wget https://www.nxp.com/lgfiles/sdk/lsdk2108/flex-installer && chmod +x flex-installer && sudo
mv flex-installer /usr/bin

$ flex-installer -i pf -d /dev/sdx
$ flex-installer -i auto -m ls1046ardb -d /dev/sdx

Then plugin SD card to LS1046ARDB, go to u-boot prompt and run "boot" command.

=>boot

[__OTL__]

Hi @yipingwang, I tested along with your comment, but looks no improvement...
Could you please check bellow result?

• HW：LS1046ARDB
• SW : LSDK21.08 ubntu main images(downloaded by $flex-installer -i auto -m ls1046ardb -d /dev/sdx)
• Env
  • server：LS1046ARDB(copper 10GLAN) 
    • iperf 3.6 (cJSON 1.5.2) 
    • cmd：$iperf3 -s
    • boot from SD card by changing DIP switch
  • client：Ubuntu Desktop
    • iperf 3.7 (cJSON 1.5.2)
    • cmd
      • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30
      • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30 -R
      • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30 -u -b 10G
      • $iperf3 -c XX.XX.XX.XX --get-server-output -P 10 -t 30 -u -b 10G -R
• result & log attacthed
  • TCP tx 6.3Gbps, rx 8.8Gbps
  • UDP tx 1.6Gbps, rx 2.6Gbps

[yipingwang]

Please use iperf command not iperf3, and use those commands which I provided in my first post to do the test.

Please apply fmc policy before running iperf.

[__OTL__]

Hi @yipingwang, I re-tested with "iperf", "-u" option in server side and fmc cmd, tcp rx/tx result achieves almost 10Gbps but udp rx/tx result looks a little strange.
Could you please check?
In parallel, we will check our network env again.

• HW：LS1046ARDB
• SW : LSDK21.08 ubntu main images(downloaded by $flex-installer -i auto -m ls1046ardb -d /dev/sdx)
• Env
  • LS1046ARDB(copper 10GLAN) 
    • iperf 2.0.13a
      • $ iperf -v
        iperf version 2.0.13a (17 Oct 2018) pthreads
    • cmd
      • $ sudo fmc -c /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/config.xml -p /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/policy_ipv4.xml -a 
      • $ iperf -c 10.84.118.216 -P 10 -t 30 // TCP TX
      • $ iperf -s // TCP RX
      • $ iperf -c 10.84.118.216 -P 10 -t 30 -u -b 10G // UDP TX
      • $ iperf -s -u // UDP RX
  • ubuntu Desktop
    • iperf 2.0.13
      • $ iperf -v
        iperf version 2.0.13 (21 Jan 2019) pthreads
    • cmd
      • $ iperf -s
      • $ iperf -c 10.84.118.227 -P 10 -t 30
      • $ iperf -s -u
      • $ iperf -c 10.84.118.227 -P 10 -t 30 -u -b 10G
• result & log attached 
  • TCP tx 9.34Gbps, rx 9.42Gbps
  • UDP tx 570Mbps??, rx 640Mbps??
    • cannot measure correctly? "XXX datagrams received out-of-order" logs are there.

[yipingwang]

On LS1046ARDB:

$ iperf -s -u

On your PC

$ iperf -c <ip address> -P 10 -t 30 -u -b 10G

Then check the result on LS1046ARDB, please check the result in "[SUM]" item.

[__OTL__]

Hi @yipingwang, Thanks for your kindly advice!
Our using "iperf" doesn't support "-R"(Reverse) option, but we can get great results on both tcp tx/rx and udp tx/rx on ubuntu main and yocto tiny!
Could you please check following results? They are expected result or should be more high throughput?

• HW：LS1046ARDB
• SW
  • LSDK21.08 ubntu main images(downloaded by $flex-installer -i auto -m ls1046ardb -d /dev/sdx)
  • LSDK21.08 yocto tiny images(downloaded from https://www.nxp.com/lgfiles/sdk/lsdk2108/firmware_ls1046ardb_qspiboot.img)
• Env
  • LS1046ARDB(copper 10GLAN) 
    • iperf 2.0.13a
    • fmc cmd applied
      • $ sudo fmc -c /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/config.xml -p /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/policy_ipv4.xml -a 
  • ubuntu Desktop
    • iperf 2.0.13
  • cmd
    • tcp Tx/udp Tx
      • LS1046ARDB : $ iperf -c 10.84.118.216 -P 10 -t 30 / $ iperf -c 10.84.118.216 -P 10 -t 30 -u -b 10G 
      • Desktop : $ iperf -s / $ iperf -s -u
    • tcp Rx/udp Rx
      • LS1046ARDB : $ iperf -s / $ iperf -s -u
      • Desktop : $ iperf -c 10.84.118.227 -P 10 -t 30 / $ iperf -c 10.84.118.227 -P 10 -t 30 -u -b 10G
  • result attached
    • ubuntu main
    • • tcp Tx 9.34Gbps, Rx 9.41Gbps
      • udp Tx 7.06Gbps, Rx 7.24Gbps
    • yocto tiny(we plan to use)
      • tcp Tx 9.36Gbps, Rx 9.41Gbps
      • udp Tx 9.00Gbps, Rx 8.12Gbps

[yipingwang]

The test result with Yocto tiny rootfs seems OK.

[__OTL__]

Hi @yipingwang, OK, thanks for your support.
So, we understand that our results improved by these three factors(paralell option in iperf/fmc cmd/using iperf v2)
Finally we have some questions, could you please answer them?
We would like to confirm how they affect the results just in case.

• About improvement by paralell option(-P option) in iperf
  This means single CPU core performance is bottoleneck?

• About improvement by fmc cmd
  How does this cmd change DPAA behavior from default behavior?
  I will check documents for the detail but please let us know the overview.

• About improvement by using iperf v2
  Can you get 10Gbps throughput with iperf3 in your environment?
  As previous comments we can't get good result with iperf3, but can with iperf v2, we are confused because no idea about how to understand this result,

[yipingwang · Accepted solution]

1. Yes, The number of CPU cores is bottleneck.

2. This is because in DPAA1, each interface used by default one pool channel across all software portals and also the dedicated channel of each CPU. In Linux Kernel, PCD Frame queues in use dedicated channels. You could refer to
the section "5. Dedicated and Pool Channels Usage in Linux Kernel" in Using QMAN Dedicated and Pool Channels in USDPAA and Linux Kernel .

3. It seems that there is problem with your iperf3 command itself.

In Ubuntu main rootfs, you could use "apt-get install iperf3" command to install iperf3 command online.

[__OTL__]

Hi @yipingwang, thanks for your answers!
I checked iperf3 then it's re-designed and supports only single thread, on the other hand iperf v2 series supports multi thread execution, so seems that measurement with iperf3 is limited by single core performance.
ref link : https://software.es.net/iperf/faq.html#:~:text=iperf3%20parallel%20stream%20performance%20is%20much%...

In fact, I tested roughly by creating four iperf3 processes, then I can get total 9Gbps for tcp tx/rx and 7Gbps for udp tx/rx, it's significantly improved from previous commented result.

Anyway I will use iperf v2 series for LS1046ARDB and our custom board with fmc command.
Thanks for your kindly support!

[yipingwang]

OK, thanks for your information.

[yipingwang]

1. Please boot up the target board with Ubuntu rootfs filesystem or apply fmc policy as the following.

fmc -c /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/config.xml -p /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/policy_ipv4.xml -a

2. Please run the following iperf command son the server and client sides.

Sever: iperf  -s

Client: iperf -c 10.10.10.2 -P 10 -t 30

 For UDP iperf test, you need to specify bandwidth with option "-b".

Sever: iperf -s -u

Client: iperf -c 100.1.1.99 -P 10 -t 30 -u -b 10G

If your problem persists, please provide the console log for the target board.