topic Re: g_ncm driver blocks CPU for small USB packets in i.MX Processors

g_ncm driver blocks CPU for small USB packets

sebastianpancea — Wed, 15 Jun 2016 13:33:26 GMT

Hello,

I did some performance tests for the g_ncm driver with the iperf tool.

The tests were done between a board with an IMX6DL SoC + 1GB of RAM and a laptop.

Used kernel version is 3.14.38 and Android L.

I observed that there is a very high CPU load when I set the sent USB packet length of the laptop to 1800 bytes.

In order to execute the test, on the board I run an iperf server:

~# iperf-armv7 -V -s -u &

and on the laptop I run the the iperf client after I set the packet length to 1800 bytes:

~# echo 1800 > /sys/class/net/usb0/cdc_ncm/tx_max # this sets the USB packet length over the NCM interface to 1800 bytes

~# iperf -V -u -c $(BOARD_IPV6_ADDR)%usb0 -b 50M # this starts the test for 50Mb/s bandwidth

When I run the test with the above commands I can see that on the board the CPU consumption raises much with the following 2 processes:

1321 1 31% S 4 16564K 480K fg root iperf-armv7

3 0 20% R 1 0K 0K fg root ksoftirqd/0

If I raise the speed of the test to >70Mb/s the board becomes totally unresponsive during the test.

When I leave the packet length to default value instead of changing it to 1800, I observed that the length of the sent packets is ~9k bytes and the ksoftirqd isn't trigger anymore and the system behaves better.

I observed the length of the packets with a USB protocol analyzer.

The problem is that the board needs to communicate with a phone that sends data over the NCM interface at 50Mb/s and 100Mb/s with the USB packet length being 1500 bytes.

This affects our performance badly.

I also observed in "/proc/interrupts" that many more USB interrupts are triggered when the package size is 1500 bytes vs. when it is 9000 bytes.

Does anyone know what could cause this performance problem and if there is any fix for it?

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Wed, 22 Jun 2016 02:03:43 GMT

Hi,

Can you test it with 2*imx6 board?

Also, as you know, UDP protocol has max MTU 1500 Bytes Size. So, actually, UDP packet has max 1472Bytes data field. In my opinion, when you set it to 1800, it will decrease UDP performance and then influence the CPU loading.

Re: g_ncm driver blocks CPU for small USB packets

sebastianpancea — Wed, 22 Jun 2016 08:45:09 GMT

Hi Tao Zheng,

When I give on the laptop the "echo 1800 > /sys/class/net/usb0/cdc_ncm/tx_max" command, I modify the size of the USB packets and not the size of the UDP packets.

The size of the UDP packets remains 1448 bytes no mater what value I write in the "tx_max" file.

I checked this by capturing the packets with wireshark.

So, as I was saying the size of the USB packets affects us.

As we increase on the laptop the value of "/sys/class/net/usb0/cdc_ncm/tx_max" the USB packets will be bigger and the performance will be better.

Unfortunately we need to have a phone connected to our board that sends USB packets with ~1500 bytes length and the performance is affected badly.

What we are doing is to simulate the behavior of the phone with the laptop by limiting the USB packet length to 1800 bytes and doing the iperf test.

Also what do you mean by "2*imx6 board"?

I have a SABRE SD board at hand and I can use it to do the same test if needed.

I'm pretty sure the results will be the same.

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Thu, 23 Jun 2016 02:08:58 GMT

Because you are using PC and imx6. So, we can't make sure which end has caused the issue. In my opinion, we can take 2 imx6 boards for testing: one imx6 is client, the other is server.

Re: g_ncm driver blocks CPU for small USB packets

sebastianpancea — Thu, 23 Jun 2016 12:22:52 GMT

Hi Tao Zheng,

I did the same test between two IMX6 boards.

The kernel version from our IMX6 boards is 3.14.38 and you can't set the USB packet size by writing the /sys/class/net/usb0/cdc_ncm/tx_max file because the file doesn't exist.

My laptop runs the 3.16.0 kernel version and this file is exposed by the driver.

In order to force the size of the USB packets on IMX6 board to 1800 I applied the lines bellow:

diff --git a/drivers/net/usb/cdc_ncm.c b/drivers/net/usb/cdc_ncm.c

index 75d7d9d..6cd35de 100644

--- a/drivers/net/usb/cdc_ncm.c

+++ b/drivers/net/usb/cdc_ncm.c

@@ -90,6 +90,7 @@ static int cdc_ncm_setup(struct usbnet *dev)

/* read correct set of parameters according to device mode */

ctx->rx_max = le32_to_cpu(ctx->ncm_parm.dwNtbInMaxSize);

+ ctx->ncm_parm.dwNtbOutMaxSize = 1800;

ctx->tx_max = le32_to_cpu(ctx->ncm_parm.dwNtbOutMaxSize);

ctx->tx_remainder = le16_to_cpu(ctx->ncm_parm.wNdpOutPayloadRemainder);

ctx->tx_modulus = le16_to_cpu(ctx->ncm_parm.wNdpOutDivisor);

I measured with the USB protocol analyzer and indeed the transmitted USB packet size is 1800 bytes during the iperf test between the two IMX6 boards.

The problem of blocking CPU during the test reproduces between the two IMX6 boards the same way it reproduces between:

- phone and board

- laptop and board

This takes us to the conclusion that there is a problem for IMX6 at receiving USB packets of small sizes at big speeds.

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Fri, 08 Jul 2016 03:09:54 GMT

Hi,

I will try to reproduce and verify it on my platform next week.

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Thu, 21 Jul 2016 08:19:06 GMT

iperf -V -s -u -i 5 &

iperf -t 1000000 -V -c fe80::641a:1eff:fe70:6f1b%usb0 -i 5 -u -b 50M

Hi,

I found that when bandwidth is 50Mbits/s, the cpu loading is about 12%. Do you think it can't meet your requirement?

Re: g_ncm driver blocks CPU for small USB packets

sebastianpancea — Fri, 22 Jul 2016 09:31:07 GMT

Hi Tao Zheng,

As I was saying above we have a phone connected to the device that sends packets at 100Mb/s with the USB packet size of 1500 bytes.

Please retry your test for speeds > 70Mb/s after setting on your computer USB packet size to 1800 bytes with this command:

"echo 1800 > /sys/class/net/usb0/cdc_ncm/tx_max"

This way you can observe how your IMX board will freeze too.

We are trying to reach the speeds of at least 100Mb/s and USB packet size of 1500 bytes without freezing our board.

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Mon, 25 Jul 2016 02:31:35 GMT

Hi,

My platform is Linux 3.14.52 and imx6q-sabresd.

This is my test steps:

First,

One imx6 is for client, one imx6 is for server, set imx6 g_ncm tx_max to be 1800Bytes as follows:

ctx->ncm_parm.dwNtbOutMaxSize = 1800;

ctx->tx_max = le32_to_cpu(ctx->ncm_parm.dwNtbOutMaxSize);

Second:

Power on 2 boards and connect them with USB cable, and run commands as follows:

server: iperf -V -s -u -i 5 &

client: iperf -t 1000000 -V -c fe80::481a:5aff:fe56:108a%usb0 -i 5 -u -b 100M

Third:

Run top at the server and you can see the cpu loading as follows:

root 20 0 29888 844 724 S 21.5 0.1 1:41.30 iperf

It is 21.5%.

I don't find the phenomenon that you have described and it is not freezed.

Do you mean that 21.5% cpu loading can't meet your requirements?

Re: g_ncm driver blocks CPU for small USB packets

sebastianpancea — Mon, 25 Jul 2016 13:29:17 GMT

Hi,

At 21,5% cpu load it could probably meet our requirements but our board is totally freezing during the test execution.

The board we are using has a iMX6DL processor instead of iMX6Q in your case.

I tried to reproduce the problem with a SABRE AUTO board with quad core cpu and I can still reproduce it.

Other differences between our setups are:

- we are using 3.14.38 kernel vs 3.14.52 in your case

- we are building kernel config for android and you are building for linux

I will try to see if I can reproduce it on linux config.

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

sebastianpancea — Tue, 26 Jul 2016 11:53:48 GMT

Hi Tao Zheng,

I did the same test on the 3.14.52 kernel built with "imx_v7_defconfig" kernel config, in order to be aligned with you.

I can still reproduce the problem using SABRE AUTO board with IMX6Q.

If I set the size of the USB packets to 1800 on my laptop, and I do the iperf test at 100Mb/s the board freezes.

What I mean by board freezes is that I cannot type anything in the serial console until the test is over.

Can you please send me your kernel image to try and see if it works on my SABRE AUTO board?

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Wed, 27 Jul 2016 01:48:00 GMT

server cpu loading: 21.5%

client cpu loading: 65%

( server and client are all imx6 platform)

But there is no freeze phenomenon.

Re: g_ncm driver blocks CPU for small USB packets

sebastianpancea — Wed, 27 Jul 2016 12:54:39 GMT

Hi,

I tested with your kernel between two imx6 boards:

- server on SABRE AUTO with IMX6Q

- client on SABRE SD with IMX6DL

I can still reproduce the freeze problem.

On the server during the test run, don't run the top application.

Just try to type characters in the console during test run.

On my board, the characters that you type in the console don't appear.

After the test is finished the typed characters will appear in the console.

If you type them quick you can also get an error like this one:

"imx-uart 21f0000.serial: Rx FIFO overrun"

This is what I mean by board freeze.

Can you please try and reproduce this behavior?

Thank you!

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Thu, 28 Jul 2016 07:06:36 GMT

sorry. I can't reproduce your phenomeon.

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Thu, 11 Aug 2016 05:56:54 GMT

Hi, Sock

I can't download your file

Error 102: Can not get information about the file keyword: mjiq5986A

Error -507: You are not authorized to access data with personal secure keyword 'mjiq5986A'.

Re: g_ncm driver blocks CPU for small USB packets

tonyzheng — Mon, 29 Aug 2016 08:23:04 GMT

Hi, Sock

Can you reproduce it on your platform?

In addition, it seems it is not usb issue, but uart issue. The USB transaction has occupied too much bandwidth and lead to less bandwidth for uart. Can you help customer to reproduce it on our reference board? Thanks.