USDPAA order restoration does not work

[omer_er]

Hi,

I have a TCP/UDP performance problem while using T4240RDB and SDK 2.0 linux kernel. I am using the RDB board as a router, basicly the board is routing the traffic  received from eth0 to eth1.

There is one iperf client and one server on windows Host connecting over T4240RDB.

The performance of iperf is nearly 700Mbps.

Then I found that the main reason behind this low performance value is the packet ordering. Since T4240 is a multicore cpu the ingress packet order and the egress packet order is not same.

The first solution that we found is using fmc to affine tcp/udp flows to a single core for order preservation.

In this solution the performance for a single  tcp flow could not reach even 2 gbps which is not enough for us.

After some research we found that the order restoration mechanism can be used for sorting reordered packets.We asked NXP how to enable this mechanism in Linux kernel, they told us it is not supported in linux kernel instead you have to use USDPAA for this mechanism.

We started to use USDPAA ipsecfwd_app application in order to understand how order preservation and restoration mechanisms are work We  measured the TCP performance with and without order preservation/restoration.

The results was not as expected.  The TCP performance for ipsecfwd_app is 2.4Gbps with or without using order restoration mechanism. Also we are seeing out of order packets even enabling  order restoration mechanism in USDPAA.

Below you can find what we have done for enabling order preservation/restoration in usdpaa. If there is an error we will be appreciated if you tell us.

//ORDER RESTORATİON

1 - I've changed ppac.h file for order restoration as stated  in SDK document

   #undef PPAC_ORDER_RESTORATION
   to;
   #define PPAC_ORDER_RESTORATION

2 -  I built  usdpaa apps. I use SDK2.0 on T4240RDB.

I got 2.4Gbps TCP performance , and there are out of order packets.

//ORDER PRESERVATİON

I also build the usdpaa-apps for PPAC_ORDER_PRESERVATION. In SDK document it says:

1 - change ppac.h as stated in SDK document.

#undef PPAC_HOLDACTIVE /* Process each FQ on one portal at a time */
#undef PPAC_ORDER_PRESERVATION /* HOLDACTIVE + enqueue-DCAs */
Change the above to
#define PPAC_HOLDACTIVE /* Process each FQ on one portal at a time */
#define PPAC_ORDER_PRESERVATION /* HOLDACTIVE + enqueue-DCAs */

2 -  I built  usdpaa apps. I use SDK2.0 on T4240RDB.

I got 3 Gbps TCP performance  and there is no out of order packets.

If I am doing right for enabling order restoration in USDPAA why I am seeing out of order packets and why the TCP performance of order restoration  is is lower than order preservation value even pool channels are using in order restoration mechanism.

Why ORDER_PRESERVATION way better than ORDER_RESTORATION in speed and packet ordering ?

Any help will be appreciated.

Thanks.

[yipingwang]

Hello ömer faruk er,

For PPAC Order Preservation, please define "PPAC_HOLDACTIVE", "PPAC_ORDER_PRESERVATION" and undef "PPAC_AVOIDBLOCK".

For PPAC Order Restoration, please define "PPAC_ORDER_RESTORATION" , "PPAC_AVOIDBLOCK" and undef "PPAC_HOLDACTIVE".

Order Preservation or restoration is regarding one stream, not a Ethernet port, for one stream, if sending packets 1,2,3, 4, 5 and receiving packets order for this stream is 1, 2, 3, 4, 5, then we consider there is no out of order.

Order Preservation is that one stream is handle by a specific CPU at the same time. Order Restoration is that one stream is handled by multiple CPUs, then addition restoration step is needed, if 2, 3, 4, 5 packets have arrived, they are held off to wait for packet 1 to do restoration.

So for multiple streams application, please use PPAC Order Preservation. For one stream application, please use PPAC Order Restoration.

Thanks,

Yiping

[omer_er]

Thanks for your answer. I did exactly what you said. I use Order Restoration for one stream application; but it is worse then Order Preservation for one stream(2.4Gbps vs. 2.9Gbps). When I compare default application (which is just AVOIDBLOCK is defined) and ORDER_RESTORATION (AVOIDBLOCK and ORDER_RESTORATION) performance, there are no difference. I thought the low speed performance of default because of packet ordering, but when I open the ORDER_PRESERVATION define I saw the same performance(which is 2.4Gbps). Why am I getting the same performance for default application and ORDER_PRESERVATION ?

Thanks.

[yipingwang]

Hello ömer faruk er,

Order Restoration is that one stream is handled by multiple CPUs, Order Preservation is that one stream is handle by a specific CPU. So as normal, for one stream application, the performance of Order Restoration should be better than Order Preservation.

Please define "PPAC_ORDER_RESTORATION" , "PPAC_AVOIDBLOCK" and undef "PPAC_HOLDACTIVE".

On the target board, please use "cat /proc/interrupts" and "top" command to check whether CPU is the bottleneck of the performance.

Thanks,

Yiping

[omer_er]

Hi yipingwang‌,

Thanks for your answers. I got another question for you. In order to enable order restoration in USDPAA I changed the code like this:

#define PPAC_ORDER_RESTORATION

#define PPAC_AVOIDBLOCK

When I check ppac_send_frame function in ppac.h which is in root/include/ppac.h location at usdpaa-apps, after I examined the call stack I found out that qman_enqueue_orp function is not called. As I know in order to enable order restoration this function has to be called. I found out that inside the function ppac_send_frame local_orp_fq is allways NULL. As expected qman_enqueue_orp is not called because it depends on either local_orp_fq is NULL. When I trace the call stack, I found that local_orp_fq is set to NULL in function macro POST_ORP().

In summary, I could not enable order restoration in USDPAA.

Any help will be appreciated.

Thanks.