LwIP FreeRTOS throughput for S32G399

[rizwan_at_disruptdotcom]

Hello,

I am concerned regarding iPerf results I got with 1G gmac speed, using the available Lwip freertos example.

RTD: S32 Real-Time Drivers Version 4.0.2 HF03

Example lwip_FreeRTOS_S32G399A_M7

Board: https://www.nxp.com/design/design-center/development-boards-and-designs/GOLDBOX-3

Default example iperf result: 

iperf -c 192.168.0.200 -p 5001 -P 1 -i 1 -t 20

------------------------------------------------------------

Client connecting to 192.168.0.200, TCP port 5001

TCP window size: 2.50 MByte (default)

------------------------------------------------------------

[  1] local 192.168.0.10 port 38578 connected with 192.168.0.200 port 5001

[ ID] Interval       Transfer     Bandwidth

[  1] 0.0000-1.0000 sec  20.6 MBytes   173 Mbits/sec

[  1] 1.0000-2.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 2.0000-3.0000 sec  15.2 MBytes   128 Mbits/sec

[  1] 3.0000-4.0000 sec  15.1 MBytes   127 Mbits/sec

[  1] 4.0000-5.0000 sec  14.0 MBytes   117 Mbits/sec

[  1] 5.0000-6.0000 sec  15.1 MBytes   127 Mbits/sec

[  1] 6.0000-7.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 7.0000-8.0000 sec  15.1 MBytes   127 Mbits/sec

[  1] 8.0000-9.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 9.0000-10.0000 sec  14.0 MBytes   117 Mbits/sec

[  1] 10.0000-11.0000 sec  15.1 MBytes   127 Mbits/sec

[  1] 11.0000-12.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 12.0000-13.0000 sec  15.1 MBytes   127 Mbits/sec

[  1] 13.0000-14.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 14.0000-15.0000 sec  15.2 MBytes   128 Mbits/sec

[  1] 15.0000-16.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 16.0000-17.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 17.0000-18.0000 sec  15.1 MBytes   127 Mbits/sec

[  1] 18.0000-19.0000 sec  13.9 MBytes   116 Mbits/sec

[  1] 19.0000-20.0000 sec  14.0 MBytes   117 Mbits/sec

[  1] 20.0000-20.3825 sec   128 KBytes  2.74 Mbits/sec

[  1] 0.0000-20.3825 sec   295 MBytes   121 Mbits/sec

iPerf results with tcpip_thread prio set to 2 instead of 1, and increased tcpip mbox to 80, No of TCP segments PCB set to 80 instead of 20 and tcpip msg api/inpkt set to 40 instead of 20:

iperf -c 192.168.0.200 -p 5001 -P 1 -i 1 -t 20

------------------------------------------------------------

Client connecting to 192.168.0.200, TCP port 5001

TCP window size: 2.50 MByte (default)

------------------------------------------------------------

[  1] local 192.168.0.10 port 38134 connected with 192.168.0.200 port 5001

[ ID] Interval       Transfer     Bandwidth

[  1] 0.0000-1.0000 sec  22.0 MBytes   185 Mbits/sec

[  1] 1.0000-2.0000 sec  17.0 MBytes   143 Mbits/sec

[  1] 2.0000-3.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 3.0000-4.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 4.0000-5.0000 sec  17.0 MBytes   143 Mbits/sec

[  1] 5.0000-6.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 6.0000-7.0000 sec  15.5 MBytes   130 Mbits/sec

[  1] 7.0000-8.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 8.0000-9.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 9.0000-10.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 10.0000-11.0000 sec  17.0 MBytes   143 Mbits/sec

[  1] 11.0000-12.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 12.0000-13.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 13.0000-14.0000 sec  17.0 MBytes   143 Mbits/sec

[  1] 14.0000-15.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 15.0000-16.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 16.0000-17.0000 sec  16.6 MBytes   139 Mbits/sec

[  1] 17.0000-18.0000 sec  16.0 MBytes   134 Mbits/sec

[  1] 18.0000-19.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 19.0000-20.0000 sec  15.6 MBytes   131 Mbits/sec

[  1] 20.0000-20.3902 sec   128 KBytes  2.69 Mbits/sec

[  1] 0.0000-20.3902 sec   326 MBytes   134 Mbits/sec

I want to know what is the tested limit for lwip stack on s32g3 cortex m7 with freertos?

I expect around 8-900Mbits/second throughput on iPerf with no other task other than the iperf application and the tcpip_thread.

Any help/insights would be appreciated.

[rizwan_at_disruptdotcom]

@alejandro_e 
One more question: 

If we use PFE but with same stack: Lwip, EthIf, Eth, Gmac_Ip with FreeRTOS. Will the iperf results be better? Can you please provide numbers if this experiment is documented already.

Thanks!

[alejandro_e]

Hello @rizwan_at_disruptdotcom,

Thanks for reaching out to us. Regarding your question, the command you shared measures the throughput of the bytes in the payload in the TCP frame, it does not show the raw throughput. Even if the line is transmitting at 100% capacity, you will never see  a value close to 1000Mbits/s, the TCP frames and the protocol itself includes quite a heavy overhead that needs to be processed in the M7 core. The value you see does not include the framing of the payload, nor the handshakes, ACKs and retransmissions. It only shows the effective payload transmitted from the host to the board. Therefore the bottleneck is in the M7 core. You could try to run the same test using the A53 cores running Linux, those are more powerful cores and should be affected noticeably less than the M7 cores. 

You can also use the PFE network accelerator which offloads the handling process of the application cores. However, this depends on your network requirements.

Let me know if you have more questions.

[rizwan_at_disruptdotcom]

@alejandro_e  Can you confirm that this is the ceiling limit for the cortex-m7/lwip/freertos or is there room for optimization in the integration. Does NXP has its own numbers for this stack? The documentation that came with this example does not talk about throughput numbers at all.

[alejandro_e]

Hello @rizwan_at_disruptdotcom,

Please find my answers below: 

> Can you confirm that this is the ceiling limit for the cortex-m7/lwip/freertos or is there room for optimization in the integration. Does NXP has its own numbers for this stack? The documentation that came with this example does not talk about throughput numbers at all.

We do not offer official test results, although checking internally your results are about what others are getting. I got a slight improvement after chaning MEMP_NUM_UDP_PCB from 8 to 32, I got around a maximum of 134Mbits/s, while before I was getting a maximum of 123. Note that the lwip configuration in the RTD4.0.2 is the fastest version we have delivered, improvements were added in previous verions.

Since lwip is not an NXP product, we do not offer official results or advertise any kind of expected performance, we offer the example "as is". You can check the lwip documentation to get information on how to improve the performance.

> If we use PFE but with same stack: Lwip, EthIf, Eth, Gmac_Ip with FreeRTOS. Will the iperf results be better? Can you please provide numbers if this experiment is documented already.

I have very little knowledge on what exactly the iperf tests does, however, for what I have seen, the performance may even be worse, I mentioned the PFE with the idea of accelerating your final application. Basically the PFE can route, shape and classify packages automatically, this is, without the application core spending time on that, the configuration is highly dependent on your application and thus, a general test like iperf may not benefit from it. if you prefer, you can share some general details of your final application and I can help you identifying the best options.

In the PFE example application you can generate a configuration with lwip and check if it is appropriate for your requirements. Regarding the documentation of the results, we do not offer those results, just as in the GMAC example.

As a heads up, the PFE can be quite tricky to configure from the M7 side, so if you have any question please let me know.

Regards

[rizwan_at_disruptdotcom]

@alejandro_e 
thank you for the detailed reply.

I have more questions and numbers which, unfortunately, they are even more concerning:

Setup:

• gPTP task firing every 3ms as required by gPTP manual and of prio lower than tcpip_thread
• tcpip_thread at max prio in system
• LwIP port in gmacif.c updated to use EthIf instead so that bot gPTP and LwIP use one interface to speak to driver. Which required me to remove zero-copy support in RX and TX path because of EthIf and Eth APIs. Eth requires ownership of DMA buffers so we can not do pbuf_alloced_custom and let lwIP decide when to return DMA buffers to Eth. Similarly, TX path does not allow zero copy because we have to use EthIf API to get txBuf from driver and lwIP can not directly map its pbuf to txBuf.

Results: Iperf dropped down to 40.3 Mbits/sec 

• IMHO this is because of EthIf dependency in gPTP stack provided by NXP that we had to update LwIP porting layer to use EthIf/Eth API instead of directly using Gmac_Ip API. Can you please confirm these numbers or are they different from your expectation. 
• Is there a plan for updating gPTP implementation to use Gmac_Ip instead of EthIf?
• What is the recommended way to integrate gPTP along with lwIP in one system. We want to keep zero-copy for RX and TX path to keep throughput at max possible rate.

Any help here is much appreciated,
Rizwan.

[alejandro_e]

Hello again @rizwan_at_disruptdotcom,

Please find my answers below:

- We do not have official test results when combining gPTP and LWIP, however, chaining from the Gmac_Ip API to the EthIf is not expected to slow down the throughput in such a magnitude. My conclusion is that the gPTP is using more core processing time and therefore the throughput gets affected so much.

- There is no plan to update gPTP from EthIf to Gmac_Ip.

- We have any document on how to integrate both components, furthermore, my expertise on both is quite limited and therefore I cannot give you any recommendation.

As I mentioned above, my recommendations are to use the A53 cores or use the PFE accelerator, depending on your application.

Let me know if you have more questions.