I have been working on the T2080RDB to try and get 10Gb Ethernet throughput and a few questions came up. First, here is my setup:
In the kernel menuconfig I have enabled:
- 802.1d Ethernet bridging
- Mellanox Technologies Ethernet Support (I am using a Mellanox ConnectX-2 Ethernet Adapter)
- Optimize for jumbo frames
- I have also set the "Maximum L2 frame size" to 9600.
*The last three were suggestions found online
I then create a bridge:
brctl addbr br0
brctl addif br0 fm1-mac10 fm1-mac9
ip link set fm1-mac10 up
ip link set fm1-mac9 up
ip link set br0 up
ip addr add 192.168.1.100/24 brd + dev br0
I now set the mtu of fm1-mac10 and fm1-mac9 to 9000 for jumbo packets:
ip link set dev fm1-mac10 mtu 9000
ip link set dev fm1-mac9 mtu 9000
I then set up iperf on both servers:
iperf -s -w 1000KB
iperf -c 22.214.171.124 -w 1000KB
Both computers have jumbo packets set to 9000
The max I have been getting is around 6 Gb/sec, which is not what I expect to be getting (when testing the throughput between the two computers, without the t2080 as the switch, I get ~9 Gb/sec). It also seems to be maxing out the CPU (each core is running at about 12 % totaling close to 100%) which is also unexpected.
My questions are:
How can I further increase my throughput; somewhere near 9 Gb/sec would be ideal?
Why am I using so much CPU and is that the expected result with this type of test?
It seems that the T2080 is processing each packet, which is why I suspect so much CPU is being used. Does the board have any way of avoiding this in order to use small amount of CPU when transferring data through Ethernet?
Any help would be much appreciated!