UDP packets oder on LS1046

[zwg132883]

Hi,

My cpu is ls1046,that all the ethernets work on  DPAA1. When the MAC10 receive the UDP packets, it has some out-order packets which can’t be corrected.

Our board use MAC10 run in 10G had integrated FMC tool, and when run FMC tool command there were no errors. Also RCW is set to RR_FFSSPPPH_1133_5559.

fmc -c /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/config.xml -p /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/policy_ipv4.xml -a

 For the 10G MAC10, using fmc tool can reduce the out order packets to 3%, comparing with no using fmc tool. But the out-order packets can't be corrected to zero. I don't know why FMC tool can’t  correct all of the out-order packets? thanks.

[yipingwang]

Please refer to the update from the AE team.

Did customer also execute fmc commands both server and client?
If there are ls1046ardb both server and client, please execute fmc commands on the two boards.

I have a trial w/o fmc.
no fmc, UDP out-of-order can be reproduced, after used fmc tool to execute fmc commands, the issue is gone.

[yipingwang]

I cannot reproduce your issue on LS1046ARDB.

I need to discuss with the AE team for the probable cause on your custom board.

[zwg132883]

Hi, Yiping

  FMC commands are executed on both server and client. And both server and client are ls1046 cards. Do you have any logs for the out-of-order packets statics in DPAA? Or is there some bug addressing strategy to support this? In this way, we can find the root cause faster. This issue has blocked for so long time.

[yipingwang]

Please refer to the following update from the AE team.

Actually, FMC only keeps order in rx side. That's mean if there are out-of-order packets arrived in MAC, the FMC can't keep the out-of-order packets to order.
In other words, if the packets is already out-of-order from TX side, the RX side may show out-of-order.
So, please suggest customer to observe how may cores handle iperf3 tool in client, so there are multiple cores handled iperf3, please use single core with taskset or other tool.

[zwg132883]

Hi,Yiping

We still suspect there are some problems for the DPAA MAC when receive the packets. I summary all of our application scenarios, flow chat, test results and problem-descriptions. Pls find them in the attachment. You can send it to your AE, and help us to analysis. 

[yipingwang]

MAC counters:

You can be found here:

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@2/net/eth2/mac_tx_stats

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@2/net/eth2/mac_rx_stats

Note in case the sysfs entries differ from above, identify the counters by:

find / -name "mac_rx_stats"

[yipingwang]

FMC received packets that match the PCD to same queue.
Also, there is no statistics for out-of-order in dpaa. The fragments may appear out-of-order in the S/G list.
Did customer observe same issue with un-fragment packet size?

[zwg132883]

Hi,yiping

1.How can I check if the out-of-order in the S/G list? It means is there a method to verify it? 

2.Due to all of the packets send/receive are in the kernel, how can I distinguish which are un-fragment packets or fragment packets?

[yipingwang]

Please refer to the following update from the AE team.

I suspect both server and client are ls1046ardb, is it right?

Please confirm if there are all cores or single core handle the iperf3 process. If all cores handle the traffic, it maybe cause out-of-order.

use "fmc -x && fmc -c  /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/config.xml -p /etc/fmc/config/private/ls1046ardb/RR_FFSSPPPH_1133_5559/policy_ipv4.xml -a" and confirm single core to handle the packets.

If the iperf3 traffic on fm1-mac10 had been handled by same queue, the out-of-order issue should be gone.

// after executed fmc

07:36:43 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle

07:36:44 PM  all    1.79    0.00    8.67    0.00    0.26    6.89    0.00    0.00    0.00   82.40

07:36:44 PM    0    0.00    0.00    0.00    0.00    0.00    0.99    0.00    0.00    0.00   99.01

07:36:44 PM    1    7.69    0.00   37.36    0.00    1.10   28.57    0.00    0.00    0.00   25.27

07:36:44 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

07:36:44 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

root@localhost:~# iperf3 -s

-----------------------------------------------------------

Server listening on 5201

-----------------------------------------------------------

Accepted connection from 192.85.1.1, port 36100 [  5] local 192.85.1.10 port 5201 connected to 192.85.1.1 port 38293

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-1.00   sec   119 MBytes   999 Mbits/sec  0.014 ms  0/86262 (0%)

[  5]   1.00-2.00   sec   119 MBytes  1.00 Gbits/sec  0.012 ms  0/86354 (0%)

[  5]   2.00-3.00   sec   119 MBytes  1000 Mbits/sec  0.012 ms  0/86294 (0%)

[  5]   3.00-4.00   sec   119 MBytes   999 Mbits/sec  0.009 ms  126/86330 (0.15%)

[  5]   4.00-5.00   sec   119 MBytes  1000 Mbits/sec  0.008 ms  0/86325 (0%)

[  5]   5.00-6.00   sec   119 MBytes  1.00 Gbits/sec  0.011 ms  0/86351 (0%)

[  5]   6.00-7.00   sec   119 MBytes  1.00 Gbits/sec  0.008 ms  0/86331 (0%)

[  5]   7.00-8.00   sec   119 MBytes   999 Mbits/sec  0.012 ms  0/86279 (0%)

[  5]   8.00-9.00   sec   119 MBytes  1.00 Gbits/sec  0.014 ms  0/86352 (0%)

[  5]   9.00-10.00  sec   119 MBytes  1.00 Gbits/sec  0.012 ms  0/86351 (0%)

[  5]  10.00-10.00  sec  35.4 KBytes   872 Mbits/sec  0.009 ms  0/25 (0%)

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  0.009 ms  126/863254 (0.015%)  receiver

-----------------------------------------------------------

Server listening on 5201

### before execute  fmc

07:37:04 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle

07:37:05 PM  all    1.52    0.00   10.15    0.00    1.52   11.17    0.00    0.00    0.00   75.63

07:37:05 PM    0    1.02    0.00    8.16    0.00    2.04   11.22    0.00    0.00    0.00   77.55

07:37:05 PM    1    1.02    0.00   10.20    0.00    1.02   11.22    0.00    0.00    0.00   76.53

07:37:05 PM    2    2.02    0.00   11.11    0.00    1.01   11.11    0.00    0.00    0.00   74.75

07:37:05 PM    3    2.02    0.00   11.11    0.00    2.02   11.11    0.00    0.00    0.00   73.74

root@localhost:~# iperf3 -s

-----------------------------------------------------------

Server listening on 5201

-----------------------------------------------------------

Accepted connection from 192.85.1.1, port 36098 [  5] local 192.85.1.10 port 5201 connected to 192.85.1.1 port 32952

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-1.00   sec   119 MBytes   999 Mbits/sec  0.040 ms  0/86285 (0%)

[  5]   1.00-2.00   sec   119 MBytes  1000 Mbits/sec  0.039 ms  0/86288 (0%)

[  5]   2.00-3.00   sec   119 MBytes  1.00 Gbits/sec  0.042 ms  3/86386 (0.0035%)

[  5]   3.00-4.00   sec   119 MBytes   999 Mbits/sec  0.041 ms  -3/86281 (-0.0035%)

[  5]   4.00-5.00   sec   119 MBytes  1000 Mbits/sec  0.037 ms  4/86302 (0.0046%)

[  5]   5.00-6.00   sec   119 MBytes  1.00 Gbits/sec  0.041 ms  -3/86363 (-0.0035%)

[  5]   6.00-7.00   sec   119 MBytes  1000 Mbits/sec  0.048 ms  -1/86309 (-0.0012%)

[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec  0.050 ms  1/86365 (0.0012%)

[  5]   8.00-9.00   sec   119 MBytes  1000 Mbits/sec  0.040 ms  3/86326 (0.0035%)

[  5]   9.00-10.00  sec   119 MBytes  1000 Mbits/sec  0.039 ms  1/86325 (0.0012%)

[  5]  10.00-10.00  sec  38.2 KBytes   980 Mbits/sec  0.036 ms  -1/26 (-3.8%)

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[SUM]  0.0-10.0 sec  447407 datagrams received out-of-order

[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  0.036 ms  4/863256 (0.00046%)  receiver

-----------------------------------------------------------

Server listening on 5201

[zwg132883]

Hi Yiping 

I have descripted my cards’ application scenario in the doc of “ls1046 dpaa issue”. Ls1046 mac10 connected to the BCM switch. I hope your AE helps us analysis our problem, why after integrating the FMC tool, most of the out-of-order packets ( receive side) are gone?  CPU PPC(T1042) hasn’t this issue when they have the same scenarios. Only a few out-of-order packets still exist. Last time, AE suspected the un-fragment packets, is it true? Could you give me some suggestions based on my scenario?

 In addition, iperf3 I had tested many times, after integrating the FMC there are still some out-of-order packets. Let the iperf3 bind a single core is not match my scenario, because this is Tx side. But the out-of-order packets in the Rx side for our application scenario. In all, please focus our scenario, rather than binding the iperf to single core. Thanks.

[yipingwang]

Please look at the iperf test result on your custom board, the performance test data is very low.

We suspect whether there is packets losing problem on the MAC layer of your custom board.

We suggest you check MAC counters on your custom board.

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@9/net/eth5/mac_rx_stats

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@9/net/eth5/mac_tx_stats

Note in case the sysfs entries differ from above, identify the counters by:

find / -name "mac_rx_stats"

[zwg132883]

Hi,Yiping

 When it had out-of-order packets, this is the statistic date of client side. Pls help me to analysis.

[zwg132883]

Hi,Yiping,

Is there some progress for this issue? Or do you need more logs about it? thanks.

[yipingwang]

Please run the following commands on both Server and Client sides, then send the results to me.

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@9/net/eth5/mac_rx_stats

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@9/net/eth5/mac_tx_stats

cat /sys/devices/platform/soc/1a00000.fman/1a91000.port/statistics/*

[zwg132883]

Hi, Yiping

   Eth5 is connected to BCM Switch in our cards, and eth5 the server side, the client side is our application process. So I try to use eth2(1G,MAC6) which is connected to the mate card to have this test using iperf3. When I get “/sys/devices/platform/soc/1a00000.fman/1a91000.port/statistics/*”, I can’t confirm which fman port is eth2, and I collected all the port of fman.

[yipingwang]

On the server side, why the following value is 0?

cat ./sys/devices/platform/soc/1a00000.fman/1a91000.port/statistics
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0
fm0-port-rx7 counter: 0

[zwg132883]

Hi,Yiping

  I had said that I did the iperf3 test by using eth2(MAC6) because you needed both server and client side. For eth5(MAC10), it hasn’t client side date because of it connected to BCM switch. eth2 is the MAC6, I can’t confirm which fman port is the MAC6. I had provided you all the fman port logs I can get. Please find the right fman port date which is corresponding to the MAC6.

I saw 0x1aad000.port is not zero. Is it the right fman port to the eth2(MAC6)? I had collected the logs again.

root@odin:/sys/devices/platform/soc/1a00000.fman# cat 1aad000.port/statistics/*

       fm0-port-tx5 counter: 65

       fm0-port-tx5 counter: 53

       fm0-port-tx5 counter: 0

       fm0-port-tx5 counter: 53

       fm0-port-tx5 counter: 0

       fm0-port-tx5 counter: 53

       fm0-port-tx5 counter: 53

       fm0-port-tx5 counter: 0

       fm0-port-tx5 counter: 0

[yipingwang]

In your server side log, all fman rx port statistics are zero.

[zwg132883]

 Hi,Yiping 

I think you hadn’t answered my question. I had said I used eth2(MAC6 – FM1 mEMAC6: 1AE_A000h) for the iperf3 test. I also tested it again, and discovered that when I used MAC6 (– FM1 mEMAC6: 1AE_A000h), “1aad000.port/statistics/*” have data, it’s not zero. Is MAC6’s fman port is 1aad000.port? You can see this in the logs.

[yipingwang]

In your server log, I only see fm0-port-tx5 for 1aad000.port is not zero, however I didn't see fm0-port-rx5 result for 1aad000.port in your server log.

[zwg132883]

Hi,Yiping 

 I got the logs again. The fman port rx is not zero. Pls help to analysis for it. thanks.