About Gigabit Ethernet performance gap between Android&Linux

[#1]

    Gigabit Ethernet should be one of most beautiful features in our imx6 platform which will bring more colorful dreams to many customers.

But recently,many people responsed that there were great performance gaps between using Android and Linux. Now let me give an exploration here.

    Same hardware, same kernel, different performance,why?

    In linux, its data throughput can reach 400Mbps.In JB, it can only get to 200Mbps.

    From the below info, we can see it should be related with frames dropping.

     root@android:/ # busybox ifconfig eth0

     eth0      Link encap:Ethernet  HWaddr 00:04:9F:02:6C:E1

               inet addr:192.168.0.100  Bcast:192.168.0.255  Mask:255.255.255.0

               inet6 addr: fe80::204:9fff:fe02:6ce1/64 Scope:Link

               UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

               RX packets:7382672 errors:71828 dropped:789 overruns:71828 frame:71828

               TX packets:4147006 errors:0 dropped:0 overruns:0 carrier:0

               collisions:0 txqueuelen:1000

               RX bytes:2568845018 (2.3 GiB)  TX bytes:284789020 (271.5 MiB)

     In TCP stack, there are three buffers involved in iperf test case:tcp_mem; tcp_rmem; tcp_wmem.

     All of them are described by three variables which will influence a lot for iperf test result.

     In linux,I got a snapshot for them:

     root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_mem

     18240      24320     36480

     root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_rmem

     4096       87380     778240

     root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_wmem

     4096       16384     778240

     In Android,I also got them to compare to:

     root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_mem

     9285      12380     18570

     root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_rmem

     4096       87380     396160

     root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_wmem

     4096       16384     396160

     The tcp_mem varibles define how the TCP stack should behave in kernel memory management.The first value tells the kernel the low threshold.

The second value tells the kernel at which point to start pressuing memory usage down. The third one tells the kernel how many memory pages it may use maximally.

If it is reached,TCP streams and packets start geting dropped until to a safe level.

     In tcp_rmem, the first value defines the minimum receive buffer for each TCP connection and this buffer is always allocated to a TCP socket.The second one defines

the default receive buffer size. The third one specifies the maximum receive buffer that can be allocated for a TCP socket.

     In tcp_wmem, three varibles also be given to describle the TCP send buffer for each TCP socket.

     We can check how these values come from in kernel code.There is an algorithm in kernel_imx/net/ipv4/sysctl_net_ipv4.c +450.

    limit = nr_free_buffer_pages() / 8;

    limit = max(limit, 128UL);

    sysctl_tcp_mem[0] = limit / 4 * 3;

    sysctl_tcp_mem[1] = limit;

    sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2;

    /* Set per-socket limits to no more than 1/128 the pressure threshold */

    limit = ((unsigned long)sysctl_tcp_mem[1]) << (PAGE_SHIFT - 7);

    max_wshare = min(4UL*1024*1024, limit);

    max_rshare = min(6UL*1024*1024, limit);

    sysctl_tcp_wmem[0] = SK_MEM_QUANTUM;

    sysctl_tcp_wmem[1] = 16*1024;

    sysctl_tcp_wmem[2] = max(64*1024, max_wshare);

    sysctl_tcp_rmem[0] = SK_MEM_QUANTUM;

    sysctl_tcp_rmem[1] = 87380;

    sysctl_tcp_rmem[2] = max(87380, max_rshare);

     From the above algorithm, we can see tcp_mem,tcp_wmem[2],tcp_rmem[2] all related with nr_free_buffer_pages() which stands for

amount of free RAM allocatable within ZONE_DMA and ZONE_NORMAL.

     So here, we can find the root cause of performance gap between Android and Linux. There is big gaps in free RAM while running different OS.

In fact, in android, Google has introduced one mechanism to tune these values through propertity.

Now we are using default AOSP's values, you can refer to them in device/fsl/imx6/etc/init.rc.For wifi and Ethernet, they are both using net.tcp.buffersize.wifi.

# Define TCP buffer sizes for various networks

#   ReadMin, ReadInitial, ReadMax, WriteMin, WriteInitial, WriteMax,

    setprop net.tcp.buffersize.default 4096,87380,110208,4096,16384,110208

    setprop net.tcp.buffersize.wifi    524288,1048576,2097152,262144,524288,1048576

    setprop net.tcp.buffersize.lte     524288,1048576,2097152,262144,524288,1048576

    setprop net.tcp.buffersize.umts    4094,87380,110208,4096,16384,110208

    setprop net.tcp.buffersize.hspa    4094,87380,262144,4096,16384,262144

    setprop net.tcp.buffersize.hsupa   4094,87380,262144,4096,16384,262144

    setprop net.tcp.buffersize.hsdpa   4094,87380,262144,4096,16384,262144

    setprop net.tcp.buffersize.hspap   4094,87380,1220608,4096,16384,1220608

    setprop net.tcp.buffersize.edge    4093,26280,35040,4096,16384,35040

    setprop net.tcp.buffersize.gprs    4092,8760,11680,4096,8760,11680

    setprop net.tcp.buffersize.evdo    4094,87380,262144,4096,16384,262144

I tried to change the above values but unfortunately got no obvious improvement.Hi,why????

so another topic,how tcp_mem and tcp_rmem cowork in kernel? In android, we only have way to tuning tcp_rmem or tcp_wmem settings but not tc_mem.

Take "iperf -c" for example, tcp_rmem will be filled up according to the frequency of Gigabit ethernet clock. And then it will be

repacked acoording to the size of tcp_mem.If tcp_mem is smaller, more times will be triggered and if it has exceeds the max value

dropping frames will be triggered. Then retransport will be launched in TCP. At last, performance will downgrade. It is just like go surfing using Gigabit but with a rubbish notebook.

You still can't enjoy good performance of Gigabit ethernet.

Why kernel calculate tcp_mem like this in ipv4? Maybe they consider the balance between single high-bandwidth and multiple connections.

You can imagine if we change the tcp_mem to use a solid big value, it may cause the board deny connections because of a lack of memory allocation

in tcp init. Here I will give out several method to improve our android ethernet performance.

• Enlarge your memory size in board design phase.

     I have double checked it by testing in our SabreAuto board whose memory is 2G whose download speed can reach to 270 Mbps about 50Mbps over Sabresd.

• Try to use older version android, if you can use ICS, you can abandon JB4.3. Compared with newer android version, old version will take less memory and there will leave more free memory to use. Using ICS, we can reach 380 Mbps downloading while in JB4.3, it can only get to 210 Mbps.
• If you are using sabresd's Gigabit ethernet for a very important case, you can balance it if you can throw other memory eaters like GPU. I have checked it if we disable GPU, the performance can reached to 340 Mbps in JB4.3 with about 50% improvement.
• Change tcp_mem algorithm to enlarge its max value threshold. Like you can change  "sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2" to "sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 3" above. You can see there won't be framedropping any more. Or you can also refer to How To: Network / TCP / UDP Tuning to hard code it. But like its author said in it, it is not recommended for those support multiple users or multiple connections. for it maybe cause the board to deny connections because of a lack of memory allocation.
• Tune tcp_rmem and tcp_wmem through the following patches in android. you will get bidirectional 320Mbps. But if you use ifconfig tool to set static ip you will not get these parameters set. For AOSP's framework only support DHCP now. For this case, you can manually echo these parameters in console before doing test.

               Gerrit Code Review

               Gerrit Code Review

• Change kernel's scheduler policy config.

     Disable CONFIG_FAIR_GROUP_SCHED and only enable CONFIG_RT_GROUP_SCHED will contribute some enhancement.

     With the above changes, I have tested on Sabresd RevC1 using android4.3GA, the bidirection speed can both reach 390~400Mbps.

[#2]

I suggest what you do is "@mention" your reviewers in a comment to the document (like this one), and ask them to be your designated reviewers and refer them to the main page of this staging area for instructions for reviewers.  Basically, all they have to do is review the doc, provide you some feedback, and then after you make needed changes, they re-review and submit a comment that says "I approve".  Pretty simple.

[ajithpv]

Hi,

We are working on i.MX6Q with Android ICS (3.0.35 kernel) and have following doubts.

1) You mentioned,by using Android ICS, you have achieved 380 Mbps downloading (iperf -c from board). I would like to know does it was stable (with small amount of difference)? or does it come like fluctuation (within 200~400Mbps range) during downloading time?

2) If it gives as fluctuation results range from 200~400Mbps, Do you have any idea on how to resolve this instability?

Thanks

Ajith P V

[waterzhou]

Can you try to change the tcp_rmem and tcp_wmem size like the patch I give out in this article?

[#5]

Hi Jianzheng,

Can you please leave this document in this review area (not your private space) until the review process has completed?  When the document is in your own space, and viewing has not been restricted, then everyone in the community can see the document.  We will want to remove all of our review comments from the document before it gets moved to a public community.

Thanks,

Grant

[maheshmahadeva1]

I would like to point customers to this thread as well so that they can share their results.

Best regards,

-Mahesh