High ethernet packet loss (iperf3)

[rans]

Hello,

We are struglling for at least a week now, with performance issues related to Ethernet packets and gstreamer: we are sending video over rtp (udp) from PC to imx6 ( lec-imx6 from adlink) , and observed a very bad performance even with small resolutions and bandwidth: starting from ~50Mbits/sec there is a very bad video playback over the imx6 station.
We have been investigating this issue, for couple of days, and seems now that we found the root cause of this performance issue.

It seems that there is high loss of packets between PC sender and the lec-imx6 receiver. we see this with a simple iperf3 test between this station. with 40Mbit/sec there is already over 85% packet loss !!

We see that the issue been discussed here:
https://community.nxp.com/thread/444707

But can't see it was resolved yet.

I would appreciate your assistance on this. This is a critical issue.
This is based on yocto build supplied by adlink from web site (jethro yocto)

Best Regards,
Ran

[chevereau]

Global problem in Ethernet management ?

To have the correct performance in video we had to activate the jumbo frame on a Gigabit controller.

Continuing our investigation, we found very large differences between iperf3 and netcat.

And that on the Gigabit Ethernet controller and on the local loop !

This problem is detailed on this request : https://community.nxp.com/thread/466054

Regards,

Pascal

[rans]

One more...
If jumbo frame resolved the issue:

Can you please add how you configured jumbo frame with imx6 ?

Thank you for your time,

ranran

[rans]

Hello Chevereau,

Thank you very much for the comment.

Wwe plan to get back with this issue, also with ethernet ( SMARC 2.0 Carrier  connect tech).

I have followed your link https://community.nxp.com/thread/466054

I see that you used Ixora (is it a PCIe adapter?),

yet your bandwidth was not good enough.

You wrote that you have used jumbo frames. 

What bandwidth did it give you finally ?  Is this issue resolved ? Which tool should we use for testing (iper3, iper2, netcat) ?

Thank you for the time,

ranran

[b36401]

Possibly the issue is caused with bad connection, weak cable or something like this.
Please try to use another ethernet cable, to use or do not use ethernet swith between i.MX and PC, etc.

Have a great day,
Victor

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rans]

Hello Victor,

We're struglling with this issue for some 2 weeks now.

We first thought is it gpu issue, but doing same pipeline without streaming works fine.

The problem starts only with streaming.

So we made iperf tests.

iperf shows very low packet drop, but iperf3 shows very large.

It change from 30%-60% already in low bandwidth of 50Mbps.

Doing the exact same test, between PC to PC shows much lower packet loss in same bandwidth.

Changing cable, switch, etc - did not help.

I also see that this issue been discussed here before:

1.

https://community.nxp.com/thread/444707

2.

i.MX6Q Ethernet: Low RX performance 

3.

Also see last comments in github:

Extremely high packet loss with UDP test · Issue #296 · esnet/iperf · GitHub 

As you can see from clemens comments, it is a real problem with imx6.

I am not sure if it is related to imx, drivers, or board (adlink lec-imx6) , but it really made us stuck.

I hope someone can test this issue with iperf3, and give us a good solution, because currently we are considering that imx6 is too weak with its ethernet high packet drop.

Regards,

Ran

[anjojohn]

while sending video,does ifconfig gives any errors like RX  errors, dropped, overruns and frame. Also ethernet is in 100M or 1G ? 

[rans]

Hello John,

We are using 1Gbps, I don't remember that the counters in ifconfig show any errors or drops.

But if iperf3 is a reliable tool (and I think it is), than we must understand why it show such high rate of lost packets.

The ethernet loss packets performance as seen with iperf3 explain the degraded performance with gstreamer rtp.

Regards,

Ran

[anjojohn]

Could you check in 100M bandwidth. We had similar issue in 1G bandwidth, so we switched to 100M. 

[rans]

Hi John,

Thanks a lot for the ideas.

I can check that, but it won't help me too much, becuase we must use 1Gbps.

We  need to stream video over ethernet in bandwidth ~450Mbits/sec), so we need to understand if imx6 can handle that, or maybe it is too weak, or buggy.

We made a lot of effort with gstreamer till now, not understanding why PC sender-PC reciever (pentium) with same gstreamer rtp pipeline works well, but PC sender-imx6 reciever performs very bad.

After investigation on this, which got strengthen when we used iperf3 for testing, we got to the conclusion that it is an ethernet issue which cause a lot of packet drops, starting from 50Mbit/sec (!!). Note that iperf tool shows much lower values. Seems that iperf3 catch much more problems than simple iperf (was discussed here too: Extremely high packet loss with UDP test · Issue #296 · esnet/iperf · GitHub  )

Thank you for any ideas,

Ran

[anjojohn]

This can also help

i.MX6 Gigabit Ethernet - Boundary Devices 

[rans]

Hi John,

Thanks a lot for the feedback.

Seems that the key here is "The actual measured performance in an optimized environment is up to 400 Mbps"

For some reason, our performance in Ethernet is very low, this is shown with iperf3.

I am not sure if it is a driver version, board issue, or imx6 revision(?), but we are seeing bad performance even in 50Mbps.

Not sure how we can continue from that. When you check with iperf3 (please try more than once):

in target:

iperf3 -s

in PC (for example the target has ip 192.168.0.1):

iperf -uZVc 192.168.0.1 -b100m 

What is the loss rate you get on your board ?

How can we make our configuration an optimized environment ?

Regards,

Ran

[anjojohn]

Hi Ranshalit,

We had similar video streaming requirement. How ever stable IMX6 bandwidth is around 200 to 300Mbps. Below is the quote from IMX6Dualite/Solo errata

"The theoretical maximum performance of 1 Gbps ENET is limited to 470 Mbps (total for Tx and Rx). The actual measured performance in an optimized environment is up to 400 Mbps"

https://www.nxp.com/docs/en/errata/IMX6SDLCE.pdf