i.MX Processors中的主题 Re: FEC ethernet packetloss

FEC ethernet packetloss

MarekVasut — Thu, 05 Dec 2013 12:09:38 GMT

Hello!

we have an issue here when communicating over FEC ethernet on the i.MX6. We would expect that if we run continuous stream of packets, the MX6 would be able to transfer them without dropping any of them. The problem is that we observe sporadic packet loss when both ends of the link operate in 1000 Mbps / full duplex mode. The packets are occasionally not transmitted FROM the FEC ethernet TOWARDS the host PC. We see first few dropped packets after roughly 5 hours of continuous transfer. After that, we see a few dropped packets every 2-3 hours. We produced exact steps to reproduce these packetloss issues, maybe someone has an idea? Thank you !

Steps:

HOSTPC: We have Intel i7 820QM with Intel i82577LM ethernet (e1000e driver) && Intel i7 3970X with i82579LM (e1000e driver) ethernet.

TARGET: We have MX6Q SabreAuto , MX6Q SabreLite and two custom boards, one with MX6Solo and other with MX6Dual. All use FEC ethernet for this test.

Any combination of TARGET and HOSTPC above have these symptoms. For your convenience, you can try with SabreAuto as the TARGET platform.

1) Connect TARGET directly through a 50cm CAT6 ethernet cable with a HOSTPC.

2) Boot Freescale Linux 3.0.35-4.1.0 (in default imx6_defconfig configuration for sabreauto, in slightly modified configuration for the custom mx6dual and mx6solo boards) on TARGET.

3) Boot the TARGET into userland on SD card, install "iperf" tool.

4) Make sure the link is in 1000/FD mode on HOST:

$ ethtool -s eth0 speed 1000 duplex full

5) Make sure the link is in 1000/FD mode on TARGET:

$ ethtool -s eth0 speed 1000 duplex full

6) Disable any possibly interfering network managers etc. on both ends:

$ /etc/init.d/networking stop

$ /etc/init.d/network-manager stop

7) Bring up network interface on HOSTPC:

$ ifconfig eth0 192.168.1.1 netmask 255.255.255.0

8) Bring up network interface on TARGET:

$ ifconfig eth0 192.168.1.2 netmask 255.255.255.0

9) Start "iperf" on HOSTPC in UDP server mode:

$ iperf -u -s -l 4M -i 60

10) Start "iperf" on TARGET in UDP client mode:

$ iperf -u -c 192.168.1.1 -t 28800 -b 1000M -i 60

Re: FEC ethernet packetloss

DuanFugang — Fri, 06 Dec 2013 01:56:42 GMT

hi,

It cannot avoid UDP drop packets.

And, since imx6 ethernet hw bandwidth have limitation (400-700Mbps), your hostpc netcard bandwidth may be better than it. Use iperf to do udp bandwidth test with 1000M throughput, the drop packet will occur.

Thanks,

Andy

Re: FEC ethernet packetloss

detlevzundel — Mon, 09 Dec 2013 09:14:47 GMT

Hi Andy,

can you please elaborate why you think we "cannot avoid to drop UDP packets"? Your reference to the bandwidth limitation

(that we of course know about) is not relevant, as our scenario is not a congestion problem at all.

If you read the report closely, you will find that (as far as we can see) these packets have been passed to the FEC by the driver

successfully but are _not_ transmitted to the other side. We follow this because the receiving side is faster and we have a full

duplex point to point connection so packet collissions cannot happen.

We fully acknowledge that in certain scenarios packet losses can happen at various points outside the MAC, but we consider

a system that cannot even put a packet "on the wire" to be not correct and look desperately for help on how to set this straight.

We also know that not using the FEC but an external PCIe network card does also _not solve_ the problem. Using the same network

cards in a regular PC without problems shows that they are not the source of the problem.

We really would like to use the i.MX6 in networking applications, but without a solution to this (and the PCIe problem)

we see serious problems here.

So can you please elaborate why you think this behaviour is accaptable?

Re: FEC ethernet packetloss

TeleLaci — Mon, 09 Dec 2013 18:18:25 GMT

Hi Detlev,

I could be wrong because I'm not a huge network guru, but I don't understand what is the problem at all.

UDP protocoll doesn't guarantee that your packet will be arrived. A packet might be lost anywhere on the route to the target. (On the internet or on your local network, wherever)

Your proggy (which one uses UDP) has to solve this dilemma.

TCP protocoll does guarantee that your packet will be arrived. It repeats lost packets.

Marek says that: "few dropped packets every 2-3 hours"

If those packets would go with TCP, they would be repeated. Few dropped (therefore repeated) of many thousands of succeeded packets. Simply it has no impact on performance, even 0.001 % neither, this controller error (yes its probably a controller or linux driver error) is totally negligible.

If you use UDP protocol you should not expect those packets will be arrived, anyway. Doesn't matter why did they get lost. UDP protocol doesn't check if a packet has arrived to the target. But TCP does.

I could be wrong, I'm sorry for that, I just don't understand what are you talking about, because this is so tiny negligible error, it has no impact on your ethernet speed at all.

Re: FEC ethernet packetloss

EricNelson — Mon, 09 Dec 2013 18:30:14 GMT

Hi Marek,

Have you looked at the performance counters (ENET_IEEE_x) to see if any reported failures correlate with the dropped packets?

I assume you've seen our results at http://boundarydevices.com/i-mx6-ethernet/, and I assume you know about the bandwidth limitations on the receive side.

We haven't done much testing with constant transmission, since the bigger issues tend to be on the receive side.

Re: FEC ethernet packetloss

EricNelson — Mon, 09 Dec 2013 18:33:01 GMT

Hi Detlev,

Do you have any information about the type of failure from the host side?

In essence, I'm wondering if the problem indicates that the packet was received but rejected or not received at all.

Your PC side driver should report packets that fail a CRC check and it would be useful to know if that statistic

increments when you detect packet loss.

Re: FEC ethernet packetloss

MarekVasut — Tue, 10 Dec 2013 13:22:44 GMT

Hi,

the problem is that the FEC ethernet swallows the packet, the packet is not emitted on the ethernet link. We are trying to figure out where this comes from. Is it a bug in the FEC or in the software ?

Re: FEC ethernet packetloss

MarekVasut — Tue, 10 Dec 2013 13:23:18 GMT

The packet is not received at all. The packet is not emitted on the ethernet link.

Re: FEC ethernet packetloss

TeleLaci — Tue, 10 Dec 2013 15:25:47 GMT

Hi Marek,

Yes, I understand that. I didn't understand why did you make problem of that. Because its probability soooo tiny, a real network program resends it, (if it was an important packet, at all) and case closed.

A packet might get lost for many reason:

- dog is chewing the cat6 network cable

- network plug is oxidized, point of contact is very bad

- bad weather, its lightens and transient pulses make trouble in your computer

- ethernet chip is over-heated, it starts to make errors

- sw bug in low level net drivers and/or op. system

- so on...I can list it forever, without end

Hardwares, (and softwares) make errors, this is their nature. If your FEC sends millions of packets without problem, and then it makes an error, that is not so bad at all, its error rate seems very low, actually its very good in my opinion. (You know what? Do calculate its error rate and compare that to the official FEC catalogue datas, you will be surprised). And its still absolutely not sure what is the origin of the error. It might be a faulty power-supply (PS errors are very sly), for example, not the FEC. Your real-life (not this artifical testing) network will have much more and bigger error sources I bet. And still, it will work, without any problem, with minor or negligible performance drops. I think you have nothing to worry about. If I were you I would test the whole networking application what I want to realize, and I would try to sift the weakest link out. Your FEC wont be the weakest link with this 1E-10 error probability, I'm perfectly sure.

I could be wrong tho :smileyhappy:

Re: FEC ethernet packetloss

detlevzundel — Tue, 17 Dec 2013 08:35:23 GMT

Hi Tele Laci,

actually in one application we don't care at all about UDP but about Ethercat Master frames being sent out from the i.MX6. We use the UDP setup to allow everybody to duplicate the problem without "exotic" Ethercat Slaves. In the Ethercat Master setup we really do care about every individual packet as it triggers one read cycle from all the slaves.

To state it explicitely again - the problem we want to solve is that the FEC controller receives packets from the Linux driver, but does _not_ put them on the wire without any indication to the contrary. Of course packet loss can occur somewhere "further down" the data path, but our test setup precludes that.

To make an analogy to our problem, think about an I2C controller that you instruct to send out a message, but it simply refuses to do this without signalling an error. I would consider this to be a hardware bug and I hope you would too :smileywink:

Re: FEC ethernet packetloss

TeleLaci — Tue, 17 Dec 2013 10:23:17 GMT

Hi Detlev,

I think I understand the problem, but maybe you you don't understand my point. Probably the FEC has some kind of hidden problem. We donno what. You want to fix it. In consequence of this error you lose 1 packet from 1 million, therefore, the error has impact on the speed (because of repeated packet) about 10E-6 ... 0.0001%. (I donno the exact numbers I just said an example, but its very rare, "...first few dropped packets after roughly 5 hours of continuous transfer..." ) Sooner or later, you are gonna fix this bug I'm sure. What will be the outcome? You will improve the average long term speed by 0.0001% !!! Are you sure this is an important bug to fix? You want to spend precious hours, to hunt this error down? That is my only doubt, I think you ignore the proportions, I agree with you anyway, I respect when someone is so precise like you.

Re: FEC ethernet packetloss

detlevzundel — Tue, 17 Dec 2013 10:51:41 GMT

Hi Tele Laci,

you are right, I _understand_ but I cannot follow your point. We have a deterministic CPU and deterministic functional blocks and I thus expect deterministic behaviour of the system as a whole. When we see a problem here, we need to diagnose it in order to be even sure how it will influence a final product. I believe your whole line of argument regarding the speed is "very problematic" to say the least. All we currently know is that we see a certain error rate in a certain test case. We have _no_ guarantee whatsoever that this is already the worst case as we do not even understand the problem. At this point it is perfectly feasible that somebody else will find a usage scenario where the failure rate will be much higher. Maybe it will then even surpass your current "subjective threshold" that you are obviously ready to accept?

Moreover, if I would accept such an "acceptable failure threshold" then where is the limit? Do we then also accept that the CPU may miscalculate once every billion operations? Do you accept other IP blocks to expose such (measurable) failure rates (e.g. SATA swallowing write operations)? I really do not want to go down that route of probabilistic computing and honestly, I have not read anything about that in the specification of the i.MX6 chip :smileywink:

And yes, I know that all electronic circuitry can malfunction because of radiation and other _external_ factors that we cannot control, but the chip as such still has to be deterministic in absence of such things.

Re: FEC ethernet packetloss

TeleLaci — Tue, 17 Dec 2013 11:38:57 GMT

Hi Detlev,

You forgot something. Computer networks are always fault tolerant systems, don't grab my thoughts out from that context. I have no idea, where are the "just acceptable thresholds" and barriers, but I'm absolutely sure that 10E-6 network speed is totally negligible, especially after I accepted imx6 FEC errata about 50% speed deficiency, it can deliver about 400Mbit instead of 800Mbit.

Do I accept CPU miscalculations with 1E-9 probability? Of course I do, I'm not stupid. If CPU were fault tolerant too, and it would fix it's own errors automatically, why not. What I would see on the BLACK BOX, its speed is slower by 1E-7 percent, but it works perfectly anyway. And who cares?

Of course if you discover something more serious with that shady-looking FEC, or he is just acting suspiciously, makes a bad move, that must be investigated, I agree with that. But if only 1E-7 speed deficiency, then let him go please. :smileywink:

Re: FEC ethernet packetloss

detlevzundel — Tue, 17 Dec 2013 12:09:02 GMT

Hi Tele Laci,

our support request was (and still is) "FEC packet loss" and not about the speed of the network.

Thanks

Detlev

Re: FEC ethernet packetloss

TeleLaci — Tue, 17 Dec 2013 12:25:46 GMT

Ok, I will do tests on my machine, as described above, then I will report. I must help if I can. I always wanted to test my network.

It seems long thing, I will run it while I sleep, then I'll be back

Laci

Re: FEC ethernet packetloss

YixingKong — Fri, 21 Feb 2014 07:06:22 GMT

Marek

Had your issue got resolved? If yes, we are going to close the discussion in 3 days. If you still need help, please feel free to reply with an update to this discussion.

Thanks,
Yixing

Re: FEC ethernet packetloss

MarekVasut — Fri, 21 Feb 2014 23:55:45 GMT

No, it was not :-(

Re: FEC ethernet packetloss

YixingKong — Mon, 24 Feb 2014 03:34:12 GMT

Marek

Ok, I will branch your issue to an internal groupa and assign an engineer to work on it.

Regards

Yixing

Re: FEC ethernet packetloss

YixingKong — Mon, 24 Feb 2014 05:11:38 GMT

Here is the response from internal engineer:

You use HostPC as the slave (receiver), imx6 as the host (transmit), the packet loss may be caused by your HostPC.

And for UDP communication, it cannot make sure there have no packet loss.

Re: FEC ethernet packetloss

cxh — Thu, 27 Feb 2014 09:43:43 GMT

I get the same error.

When I use http to download large files, there's many packets lost.

stability of iMX6 Ethernet

We configured the fec as RMII mode.