About Gigabit Ethernet performance gap between Android&Linux Gigabit Ethernet should be one of most beautiful features in our imx6 platform which will bring more colorful dreams to many customers. But recently,many people responsed that there were great performance gaps between using Android and Linux. Now let me give an exploration here. Same hardware, same kernel, different performance,why? In linux, its data throughput can reach 400Mbps.In JB, it can only get to 200Mbps. From the below info, we can see it should be related with frames dropping. root@android:/ # busybox ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:04:9F:02:6C:E1 inet addr:192.168.0.100 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::204:9fff:fe02:6ce1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7382672 errors:71828 dropped:789 overruns:71828 frame:71828 TX packets:4147006 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2568845018 (2.3 GiB) TX bytes:284789020 (271.5 MiB) In TCP stack, there are three buffers involved in iperf test case:tcp_mem; tcp_rmem; tcp_wmem. All of them are described by three variables which will influence a lot for iperf test result. In linux,I got a snapshot for them: root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_mem 18240 24320 36480 root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 778240 root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 778240 In Android,I also got them to compare to: root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_mem 9285 12380 18570 root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 396160 root@sabresd_6dq:/# cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 396160 The tcp_mem varibles define how the TCP stack should behave in kernel memory management.The first value tells the kernel the low threshold. The second value tells the kernel at which point to start pressuing memory usage down. The third one tells the kernel how many memory pages it may use maximally. If it is reached,TCP streams and packets start geting dropped until to a safe level. In tcp_rmem, the first value defines the minimum receive buffer for each TCP connection and this buffer is always allocated to a TCP socket.The second one defines the default receive buffer size. The third one specifies the maximum receive buffer that can be allocated for a TCP socket. In tcp_wmem, three varibles also be given to describle the TCP send buffer for each TCP socket. We can check how these values come from in kernel code.There is an algorithm in kernel_imx/net/ipv4/sysctl_net_ipv4.c +450. limit = nr_free_buffer_pages() / 8; limit = max(limit, 128UL); sysctl_tcp_mem[0] = limit / 4 * 3; sysctl_tcp_mem[1] = limit; sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2; /* Set per-socket limits to no more than 1/128 the pressure threshold */ limit = ((unsigned long)sysctl_tcp_mem[1]) << (PAGE_SHIFT - 7); max_wshare = min(4UL*1024*1024, limit); max_rshare = min(6UL*1024*1024, limit); sysctl_tcp_wmem[0] = SK_MEM_QUANTUM; sysctl_tcp_wmem[1] = 16*1024; sysctl_tcp_wmem[2] = max(64*1024, max_wshare); sysctl_tcp_rmem[0] = SK_MEM_QUANTUM; sysctl_tcp_rmem[1] = 87380; sysctl_tcp_rmem[2] = max(87380, max_rshare); From the above algorithm, we can see tcp_mem,tcp_wmem[2],tcp_rmem[2] all related with nr_free_buffer_pages() which stands for amount of free RAM allocatable within ZONE_DMA and ZONE_NORMAL. So here, we can find the root cause of performance gap between Android and Linux. There is big gaps in free RAM while running different OS. In fact, in android, Google has introduced one mechanism to tune these values through propertity. Now we are using default AOSP's values, you can refer to them in device/fsl/imx6/etc/init.rc.For wifi and Ethernet, they are both using net.tcp.buffersize.wifi. # Define TCP buffer sizes for various networks # ReadMin, ReadInitial, ReadMax, WriteMin, WriteInitial, WriteMax, setprop net.tcp.buffersize.default 4096,87380,110208,4096,16384,110208 setprop net.tcp.buffersize.wifi 524288,1048576,2097152,262144,524288,1048576 setprop net.tcp.buffersize.lte 524288,1048576,2097152,262144,524288,1048576 setprop net.tcp.buffersize.umts 4094,87380,110208,4096,16384,110208 setprop net.tcp.buffersize.hspa 4094,87380,262144,4096,16384,262144 setprop net.tcp.buffersize.hsupa 4094,87380,262144,4096,16384,262144 setprop net.tcp.buffersize.hsdpa 4094,87380,262144,4096,16384,262144 setprop net.tcp.buffersize.hspap 4094,87380,1220608,4096,16384,1220608 setprop net.tcp.buffersize.edge 4093,26280,35040,4096,16384,35040 setprop net.tcp.buffersize.gprs 4092,8760,11680,4096,8760,11680 setprop net.tcp.buffersize.evdo 4094,87380,262144,4096,16384,262144 I tried to change the above values but unfortunately got no obvious improvement.Hi,why???? so another topic,how tcp_mem and tcp_rmem cowork in kernel? In android, we only have way to tuning tcp_rmem or tcp_wmem settings but not tc_mem. Take "iperf -c" for example, tcp_rmem will be filled up according to the frequency of Gigabit ethernet clock. And then it will be repacked acoording to the size of tcp_mem.If tcp_mem is smaller, more times will be triggered and if it has exceeds the max value dropping frames will be triggered. Then retransport will be launched in TCP. At last, performance will downgrade. It is just like go surfing using Gigabit but with a rubbish notebook. You still can't enjoy good performance of Gigabit ethernet. Why kernel calculate tcp_mem like this in ipv4? Maybe they consider the balance between single high-bandwidth and multiple connections. You can imagine if we change the tcp_mem to use a solid big value, it may cause the board deny connections because of a lack of memory allocation in tcp init. Here I will give out several method to improve our android ethernet performance. Enlarge your memory size in board design phase. I have double checked it by testing in our SabreAuto board whose memory is 2G whose download speed can reach to 270 Mbps about 50Mbps over Sabresd. Try to use older version android, if you can use ICS, you can abandon JB4.3. Compared with newer android version, old version will take less memory and there will leave more free memory to use. Using ICS, we can reach 380 Mbps downloading while in JB4.3, it can only get to 210 Mbps. If you are using sabresd's Gigabit ethernet for a very important case, you can balance it if you can throw other memory eaters like GPU. I have checked it if we disable GPU, the performance can reached to 340 Mbps in JB4.3 with about 50% improvement. Change tcp_mem algorithm to enlarge its max value threshold. Like you can change "sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2" to "sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 3" above. You can see there won't be framedropping any more. Or you can also refer to How To: Network / TCP / UDP Tuning to hard code it. But like its author said in it, it is not recommended for those support multiple users or multiple connections. for it maybe cause the board to deny connections because of a lack of memory allocation. Tune tcp_rmem and tcp_wmem through the following patches in android. you will get bidirectional 320Mbps. But if you use ifconfig tool to set static ip you will not get these parameters set. For AOSP's framework only support DHCP now. For this case, you can manually echo these parameters in console before doing test. Gerrit Code Review Gerrit Code Review Change kernel's scheduler policy config. Disable CONFIG_FAIR_GROUP_SCHED and only enable CONFIG_RT_GROUP_SCHED will contribute some enhancement. With the above changes, I have tested on Sabresd RevC1 using android4.3GA, the bidirection speed can both reach 390~400Mbps. Re: About Gigabit Ethernet performance gap between Android&Linux I would like to point customers to this thread as well so that they can share their results. Best regards, -Mahesh Re: About Gigabit Ethernet performance gap between Android&Linux Hi Jianzheng, Can you please leave this document in this review area (not your private space) until the review process has completed? When the document is in your own space, and viewing has not been restricted, then everyone in the community can see the document. We will want to remove all of our review comments from the document before it gets moved to a public community. Thanks, Grant Re: About Gigabit Ethernet performance gap between Android&Linux Can you try to change the tcp_rmem and tcp_wmem size like the patch I give out in this article? Re: About Gigabit Ethernet performance gap between Android&Linux Hi, We are working on i.MX6Q with Android ICS (3.0.35 kernel) and have following doubts. 1) You mentioned,by using Android ICS, you have achieved 380 Mbps downloading (iperf -c from board). I would like to know does it was stable (with small amount of difference)? or does it come like fluctuation (within 200~400Mbps range) during downloading time? 2) If it gives as fluctuation results range from 200~400Mbps, Do you have any idea on how to resolve this instability? Thanks Ajith P V Re: About Gigabit Ethernet performance gap between Android&Linux I suggest what you do is "@mention" your reviewers in a comment to the document (like this one), and ask them to be your designated reviewers and refer them to the main page of this staging area for instructions for reviewers. Basically, all they have to do is review the doc, provide you some feedback, and then after you make needed changes, they re-review and submit a comment that says "I approve". Pretty simple.
View full article
