Out-of-Order packets and high software-interrupt-rate on T2080 when receiving jumbo packets

[clemenseisserer]

Hello,

We are using one of our T2080RDB (running SDK-1.9 / linux-3.12.37) to receive a continous stream of 9k jumbo packets (UDP) on the 10G link (fman1-mac9).

Compared to SDK-1.6 where packets were lost, SD-1.9 behaves a lot better.

However we are experiencing the following problems:

• UDP packets are received out-of-order (using recvfrom in userspace). 
  In /proc/interrupts it can be seen that QMan-Portal 0 -> 7 are active and handle interrupts each on a dedicated core, so maybe distributing the packets to multiple QMan portals could be an issue?

• The system experiences a quite high software interrupt rate during this kind of load, top reports:Cpu(s):  2.0%us,  0.8%sy,  0.0%ni, 48.2%id,  0.0%wa,  1.8%hi, 47.2%si,  0.0%st

Is there any way how we can receive the packets in order?

Are there tuning possibilities for our kind of workloads to get the software interrupt rate down?

Thank you in advance, Clemens

[clemenseisserer]

Some further information: For 63,5TB data (consisting of 9k jumbo frames), 4,35G interrupts were generated - which translates to 1 interrupt per ~15kb of data.

Is this to be expected?

root@t2080rdb:~# cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7

124:          0          0          0          0          0          0          0  546099313   OpenPIC   124 Level     QMan portal 7

125:          0          0          0          0          0          0          0          0   OpenPIC   125 Level     BMan portal 7

126:          0          0          0          0          0          0  555860282          0   OpenPIC   126 Level     QMan portal 6

127:          0          0          0          0          0          0          0          0   OpenPIC   127 Level     BMan portal 6

128:          0          0          0          0          0  545426042          0          0   OpenPIC   128 Level     QMan portal 5

129:          0          0          0          0          0          0          0          0   OpenPIC   129 Level     BMan portal 5

130:          0          0          0          0  554352922          0          0          0   OpenPIC   130 Level     QMan portal 4

131:          0          0          0          0          0          0          0          0   OpenPIC   131 Level     BMan portal 4

132:          0          0          0  545187734          0          0          0          0   OpenPIC   132 Level     QMan portal 3

133:          0          0          0          0          0          0          0          0   OpenPIC   133 Level     BMan portal 3

134:          0          0  554957818          0          0          0          0          0   OpenPIC   134 Level     QMan portal 2

135:          0          0          0          0          0          0          0          0   OpenPIC   135 Level     BMan portal 2

136:          0  546313678          0          0          0          0          0          0   OpenPIC   136 Level     QMan portal 1

137:          0          0          0          0          0          0          0          0   OpenPIC   137 Level     BMan portal 1

138:  556060144          0          0          0          0          0          0          0   OpenPIC   138 Level     QMan portal 0

139:          0          0          0          0          0          0          0          0   OpenPIC   139 Level     BMan portal 0

[clemenseisserer]

*bump*