IMX6 Ethernet packet loss with SATA write operation

azucarillo · ‎08-01-2016

Hello,

We have an issue with Ethernet connection when performing write operation on external SATA drive.

Here's our configuration:

- Freescale IMX6 QUAD (MCIMX6Q7CVT08AD)

- Linux kernel 3.16.7

- Embedded module TQMa6x

- Linux BSP with YOCTO dizzy toolchain (kernel patch from Linux BSP for TQMa6x rev 0105 with OSELAS toolchain)

Our system has a constant incoming traffic on ethernet interface of raw frames at 30Mbit/s.

This works fine except when we ha a write operation on SATA. In that case we experience a packet loss up to 14%.

We can reproduce the problem using iperf with UDP packet.

Here's the test:

- On the IMX6 target board we start iperf:

$ iperf -u -s

- On the host we start iperf:

$ iperf -u -c 192.168.2.1 -t28800 -b 30M -i10

Results:

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams

UDP buffer size: 160 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.1 port 5001 connected with 192.168.2.88 port 47808

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 3] 0.0-10.0 sec 35.8 MBytes 30.0 Mbits/sec 0.006 ms 0/25513 (0%)

[ 3] 10.0-20.0 sec 35.8 MBytes 30.0 Mbits/sec 0.007 ms 0/25512 (0%)

[ 3] 20.0-30.0 sec 35.8 MBytes 30.0 Mbits/sec 0.017 ms 0/25512 (0%)

[ 3] 30.0-40.0 sec 35.8 MBytes 30.0 Mbits/sec 0.008 ms 0/25512 (0%)

[ 3] 40.0-50.0 sec 35.8 MBytes 30.0 Mbits/sec 0.013 ms 0/25512 (0%)

[ 3] 50.0-60.0 sec 35.8 MBytes 30.0 Mbits/sec 0.011 ms 0/25512 (0%)

[ 3] 60.0-70.0 sec 35.8 MBytes 30.0 Mbits/sec 0.008 ms 0/25512 (0%)

When performing a write operation on SATA disk (e.g. writing a 1GB sample file with dd)

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams

UDP buffer size: 160 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.1 port 5001 connected with 192.168.2.88 port 57920

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 3] 0.0-10.0 sec 33.3 MBytes 27.9 Mbits/sec 0.425 ms 1777/25511 (7%)

[ 3] 10.0-20.0 sec 30.8 MBytes 25.9 Mbits/sec 0.044 ms 3524/25515 (14%)

[ 3] 20.0-30.0 sec 31.9 MBytes 26.8 Mbits/sec 0.008 ms 2734/25512 (11%)

[ 3] 30.0-40.0 sec 33.5 MBytes 28.1 Mbits/sec 0.009 ms 1619/25512 (6.3%)

We tried to change in kernel source SATA DMA mode to:

mdma0

mdma1

mdma2

udma0

udma1

udma2

udma3

udma4

udma5

udma6 (was the default)

With no improvements.

If we try the same test writing on MMC we experience no packet loss at all.

Thanks in advance!

Giovanni Malfarà

igorpadykov · ‎08-01-2016

Hi Giovanni

one can try to change priorities, for example the ENET priority to 1 :

/unit_tests/memtool -32 0x00247100=0x1

/unit_tests/memtool -32 0x00247104=0x1

please check In iMX6DQ reference manual Chapter 45

Network Interconnect Bus System (NIC-301) and Table 45-5

http://cache.freescale.com/files/32bit/doc/ref_manual/IMX6DQRM.pdf

or try to add kernel boot parameter "enable_wait_mode=off".

Also note that Linux kernel 3.16.7 is not supported by nxp, recommended to

try on Sabre reference board with nxp kernels found on i.MX6Q product page

i.MX 6 Series Software and Development Tool|NXP

Best regards

igor

-----------------------------------------------------------------------------------------------------------------------

Note: If this post answers your question, please click the Correct Answer button. Thank you!

-----------------------------------------------------------------------------------------------------------------------

azucarillo · ‎08-02-2016

Thank you so much Igor!

We just tried with no improvements, the result is exactly the same.

Do you have any other suggestions before trying the new Linux Kernel?

Since it's not easy to move to the newer version if there were other

things to try it would be worth.

Thank you very much.

Best Regards,

Giovanni Malfarà

igorpadykov · ‎08-02-2016

Hi Giovanni

one can try to analyze ddr usage with mmdc profiling

MMDC Profiling Tool

Best regards

igor

azucarillo · ‎08-02-2016

Hi Igor. In the page you linked, there are two more links but if I click

on them I get Unauthorized Access.

How can I be authorized?

Thank you very much

Giovanni

igorpadykov · ‎08-02-2016

Hi Giovanni

please look at last post from Erez Steinberg

with the same material. Also it is included

in unit tests (imx-test/..test/mmdc), seems for kernel 3.16.7

most close

http://patches.openembedded.org/patch/110753/

Best regards

igor

azucarillo · ‎08-22-2016

Hi Igor,

I installed the MMDC profiling tools.

Here's the result:

============================================

Iperf with UDP packets and without SATA write

============================================

MMDC new Profiling results:

***********************

Measure time: 60000ms

Total cycles count: 1615387008

Busy cycles count: 1166556102

Read accesses count: 10610905

Write accesses count: 16389026

Read bytes count: 301019844

Write bytes count: 255752552

Avg. Read burst size: 28

Avg. Write burst size: 15

Read: -199.36 MB/s / Write: 4.07 MB/s Total: 8.85 MB/s

Utilization: 2%

Bus Load: 72%

Bytes Access: 20

===========================================

Iperf with UDP packets, with SATA write and packet loss

===========================================

MMDC new Profiling results:

***********************

Measure time: 60001ms

Total cycles count: 1615389160

Busy cycles count: 2033804176

Read accesses count: 98887352

Write accesses count: 90426003

Read bytes count: 3596292658

Write bytes count: 2638088496

Avg. Read burst size: 36

Avg. Write burst size: 29

Read: -2383.39 MB/s / Write: 41.93 MB/s Total: 30.83 MB/s

Utilization: 19%

Bus Load: 125%

Bytes Access: 32

What do you think? Do you think we have a bus-priority issue?

Thank you

Best Regards,

GIovanni Malfarà

martin_maurer · ‎08-22-2016

Just as note: In my opinion 60s ("Measure time: 60001ms") is too long for MMDC tool. You get overflows.

Look here:

"Read: -2383.39 MB/s / Write: 41.93 MB/s Total: 30.83 MB/s"

R+W is not more around total.

Try e.g. 1 to 2 second and check if results are better.

azucarillo · ‎08-22-2016

Hello Martin,

Here's the result with 2 seconds:

Without SATA write:

MMDC new Profiling results:

***********************

Measure time: 2000ms

Total cycles count: 1056162912

Busy cycles count: 113214627

Read accesses count: 1050883

Write accesses count: 1719010

Read bytes count: 29686892

Write bytes count: 26642448

Avg. Read burst size: 28

Avg. Write burst size: 15

Read: 14.16 MB/s / Write: 12.70 MB/s Total: 26.86 MB/s

Utilization: 3%

Bus Load: 10%

Bytes Access: 20

With SATA WRITE

MMDC new Profiling results:

***********************

Measure time: 2001ms

Total cycles count: 1056311448

Busy cycles count: 345828336

Read accesses count: 6562724

Write accesses count: 5125583

Read bytes count: 252684162

Write bytes count: 143900744

Avg. Read burst size: 38

Avg. Write burst size: 28

Read: 120.43 MB/s / Write: 68.58 MB/s Total: 189.01 MB/s

Utilization: 7%

Bus Load: 32%

Bytes Access: 33

Thank you so much

Best Regards,

Giovanni

azucarillo · ‎08-31-2016

Hello,

we tried with a new kernel. This is our acttual configuration:

- Freescale IMX6 QUAD (MCIMX6Q7CVT08AD)

- Linux kernel 4.1.6-TQMa6-linux-v4.1.6-20151009

- Embedded module TQMa6x

- Linux BSP with OSELAS.Toolchain-2013.12.2 linaro-4.8-2013.11

And running the same test we get exactly the same results (up to 14% packet loss).

Any suggestion?

Thank you so much

Best Regards,

Giovanni Malfarà

Yuri · ‎10-10-2016

Hello,

You may try to increase data buffers for both Ethernet and SATA data

in Your application. Also double buffering approach can help.

Regards,

Yuri.

IMX6 Ethernet packet loss with SATA write operation

IMX6 Ethernet packet loss with SATA write operation

i.MX6Quad