Hello,
We have an issue with Ethernet connection when performing write operation on external SATA drive.
Here's our configuration:
- Freescale IMX6 QUAD (MCIMX6Q7CVT08AD)
- Linux kernel 3.16.7
- Embedded module TQMa6x
- Linux BSP with YOCTO dizzy toolchain (kernel patch from Linux BSP for TQMa6x rev 0105 with OSELAS toolchain)
Our system has a constant incoming traffic on ethernet interface of raw frames at 30Mbit/s.
This works fine except when we ha a write operation on SATA. In that case we experience a packet loss up to 14%.
We can reproduce the problem using iperf with UDP packet.
Here's the test:
- On the IMX6 target board we start iperf:
$ iperf -u -s
- On the host we start iperf:
$ iperf -u -c 192.168.2.1 -t28800 -b 30M -i10
Results:
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.1 port 5001 connected with 192.168.2.88 port 47808
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 35.8 MBytes 30.0 Mbits/sec 0.006 ms 0/25513 (0%)
[ 3] 10.0-20.0 sec 35.8 MBytes 30.0 Mbits/sec 0.007 ms 0/25512 (0%)
[ 3] 20.0-30.0 sec 35.8 MBytes 30.0 Mbits/sec 0.017 ms 0/25512 (0%)
[ 3] 30.0-40.0 sec 35.8 MBytes 30.0 Mbits/sec 0.008 ms 0/25512 (0%)
[ 3] 40.0-50.0 sec 35.8 MBytes 30.0 Mbits/sec 0.013 ms 0/25512 (0%)
[ 3] 50.0-60.0 sec 35.8 MBytes 30.0 Mbits/sec 0.011 ms 0/25512 (0%)
[ 3] 60.0-70.0 sec 35.8 MBytes 30.0 Mbits/sec 0.008 ms 0/25512 (0%)
When performing a write operation on SATA disk (e.g. writing a 1GB sample file with dd)
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.1 port 5001 connected with 192.168.2.88 port 57920
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 33.3 MBytes 27.9 Mbits/sec 0.425 ms 1777/25511 (7%)
[ 3] 10.0-20.0 sec 30.8 MBytes 25.9 Mbits/sec 0.044 ms 3524/25515 (14%)
[ 3] 20.0-30.0 sec 31.9 MBytes 26.8 Mbits/sec 0.008 ms 2734/25512 (11%)
[ 3] 30.0-40.0 sec 33.5 MBytes 28.1 Mbits/sec 0.009 ms 1619/25512 (6.3%)
We tried to change in kernel source SATA DMA mode to:
mdma0
mdma1
mdma2
udma0
udma1
udma2
udma3
udma4
udma5
udma6 (was the default)
With no improvements.
If we try the same test writing on MMC we experience no packet loss at all.
Thanks in advance!
Giovanni Malfarà
Hi Giovanni
one can try to change priorities, for example the ENET priority to 1 :
/unit_tests/memtool -32 0x00247100=0x1
/unit_tests/memtool -32 0x00247104=0x1
please check In iMX6DQ reference manual Chapter 45
Network Interconnect Bus System (NIC-301) and Table 45-5
http://cache.freescale.com/files/32bit/doc/ref_manual/IMX6DQRM.pdf
or try to add kernel boot parameter "enable_wait_mode=off".
Also note that Linux kernel 3.16.7 is not supported by nxp, recommended to
try on Sabre reference board with nxp kernels found on i.MX6Q product page
i.MX 6 Series Software and Development Tool|NXP
Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------
Thank you so much Igor!
We just tried with no improvements, the result is exactly the same.
Do you have any other suggestions before trying the new Linux Kernel?
Since it's not easy to move to the newer version if there were other
things to try it would be worth.
Thank you very much.
Best Regards,
Giovanni Malfarà
Hi Igor. In the page you linked, there are two more links but if I click
on them I get Unauthorized Access.
How can I be authorized?
Thank you very much
Giovanni
Hi Giovanni
please look at last post from Erez Steinberg
with the same material. Also it is included
in unit tests (imx-test/..test/mmdc), seems for kernel 3.16.7
most close
http://patches.openembedded.org/patch/110753/
Best regards
igor
Hi Igor,
I installed the MMDC profiling tools.
Here's the result:
============================================
Iperf with UDP packets and without SATA write
============================================
MMDC new Profiling results:
***********************
Measure time: 60000ms
Total cycles count: 1615387008
Busy cycles count: 1166556102
Read accesses count: 10610905
Write accesses count: 16389026
Read bytes count: 301019844
Write bytes count: 255752552
Avg. Read burst size: 28
Avg. Write burst size: 15
Read: -199.36 MB/s / Write: 4.07 MB/s Total: 8.85 MB/s
Utilization: 2%
Bus Load: 72%
Bytes Access: 20
===========================================
Iperf with UDP packets, with SATA write and packet loss
===========================================
MMDC new Profiling results:
***********************
Measure time: 60001ms
Total cycles count: 1615389160
Busy cycles count: 2033804176
Read accesses count: 98887352
Write accesses count: 90426003
Read bytes count: 3596292658
Write bytes count: 2638088496
Avg. Read burst size: 36
Avg. Write burst size: 29
Read: -2383.39 MB/s / Write: 41.93 MB/s Total: 30.83 MB/s
Utilization: 19%
Bus Load: 125%
Bytes Access: 32
What do you think? Do you think we have a bus-priority issue?
Thank you
Best Regards,
GIovanni Malfarà
Just as note: In my opinion 60s ("Measure time: 60001ms") is too long for MMDC tool. You get overflows.
Look here:
"Read: -2383.39 MB/s / Write: 41.93 MB/s Total: 30.83 MB/s"
R+W is not more around total.
Try e.g. 1 to 2 second and check if results are better.
Hello Martin,
Here's the result with 2 seconds:
Without SATA write:
MMDC new Profiling results:
***********************
Measure time: 2000ms
Total cycles count: 1056162912
Busy cycles count: 113214627
Read accesses count: 1050883
Write accesses count: 1719010
Read bytes count: 29686892
Write bytes count: 26642448
Avg. Read burst size: 28
Avg. Write burst size: 15
Read: 14.16 MB/s / Write: 12.70 MB/s Total: 26.86 MB/s
Utilization: 3%
Bus Load: 10%
Bytes Access: 20
With SATA WRITE
MMDC new Profiling results:
***********************
Measure time: 2001ms
Total cycles count: 1056311448
Busy cycles count: 345828336
Read accesses count: 6562724
Write accesses count: 5125583
Read bytes count: 252684162
Write bytes count: 143900744
Avg. Read burst size: 38
Avg. Write burst size: 28
Read: 120.43 MB/s / Write: 68.58 MB/s Total: 189.01 MB/s
Utilization: 7%
Bus Load: 32%
Bytes Access: 33
Thank you so much
Best Regards,
Giovanni
Hello,
we tried with a new kernel. This is our acttual configuration:
- Freescale IMX6 QUAD (MCIMX6Q7CVT08AD)
- Linux kernel 4.1.6-TQMa6-linux-v4.1.6-20151009
- Embedded module TQMa6x
- Linux BSP with OSELAS.Toolchain-2013.12.2 linaro-4.8-2013.11
And running the same test we get exactly the same results (up to 14% packet loss).
Any suggestion?
Thank you so much
Best Regards,
Giovanni Malfarà
Hello,
You may try to increase data buffers for both Ethernet and SATA data
in Your application. Also double buffering approach can help.
Regards,
Yuri.