RTCS running out of memory during TCP data transfer such as FTP

[paulgounot]

Dear Freescale supprt and community fellows,

I am facing since MQX 4.0, now MQX4.1.1 an issue with RAM footprint stability during TCP file transfer, FTP for example. Systems runs out of memory. There are various posts on the community forum on the topic, however most of the time the conclusions are to way for the next MQX release or to tweak TCP/IP stack parameters. We are freezing the code for a production candidate and can't take the risk of a new MQX upgrade at this stage.

After several days investigating the issue, I've been able to characterize the problem and isolate it to the way RTCS manages cloned TCB. A short report describing the issue more in details, steps to reproduce with MQX 4.1.1, and commented logs is enclosed.

Not being a TCP nor a RTCS expert, still I'm struggling to find a smart implementation to solve it which will be safe from a TCP protocol point of view and from a RTCS integrity point of view.

Please could Freescale developpers have a look to my report and advise about how to do so.

Any comment from the community fellows is welcome.

I'm using Kinetis devices (K60 and K64) with no external memory.

Regards,

Paul.

[karelm_]

Hi Paul,

Thank you very much for your detailed and readable analysis. I used a workaround to improve RTCS behavior. I set socket option OPT_TIMEWAIT_TIMEOUT to limit a time for which TCB is stored. In FTP server (MQX 4.1.2 and newer I think) it is set to 10ms. However this violates TCP RFC. Earlier it was not possible to set this value lower than three seconds. If you do not mind, you might try to use it in both FTP and HTTP server. Also try to search for macro DEFAULT_TIMEWAIT_TIMEOUT in RTCS headers. This macro determines for how long shall be TCB stored as default for all sockets.

Best regard,

Karel

[paulgounot]

Hi Karel,

Thank you.

It is end of the day here, I will give it a try tomorrow morning, hopefully shortening the timeout will reduce the probability of a re-entry, however this violates the TCP RFC as you wrote and I wonder other consequences it could have in the field. Keep you posted on this test tomorrow.

The other idea I had and which would not violate the RFC was to release all TCBs (cloned or not) associated with what I call the “parent socket” when that socket is shut down with ABORT (during processing of PASV); the problem I have due to my limited knowledge of RTCS is how to list all such TCBs. I saw in the socket structure pointers to TCBs, in TCB structure there are also pointers to socket (forced to 0 for cloned TCB). But I haven’t been able so far to identify a relationship between all the values I can observe in the memory with JTAG. Do you have any idea on how to do this? Are you aware of any documentation about socket and TCB management in RTCS? We can pursue under NDA if required.

Regards,

Paul.

De : Karel Mozny

Envoyé : mardi 24 mars 2015 16:14

À : Paul GOUNOT

Objet : Re: - RTCS running out of memory during TCP data transfer such as FTP

<https://community.freescale.com/>

RTCS running out of memory during TCP data transfer such as FTP

reply from Karel Mozny <https://community.freescale.com/people/karelm.?et=watches.email.thread> in MQX Software Solutions - View the full discussion <https://community.freescale.com/message/497578?et=watches.email.thread#497578>

[karelm_]

Hi Paul,

the RFC is one thing, the real world often differs :smileywink: I would not be afraid to lower the timeout as this is a know issue with TCP in embedded and it is reasonable workaround. As far as I know there is no other documentation other than what is publicly available (I work on RTCS for last 3 years). I would not recommend to poke RTCS internals as this might have some unpredictable consequences like breaking interoperability, bad ACK generating etc.(already happened to me). You might try to compare the 4.1.1 and 4.2.0 code as soon as it is available. In MQX 4.2.0 there will be a major change in socket behavior. We implemented SO_LINGER socket option which allows you to change how socket is closing, there also is closesocket function emulating close() function with socket as a parameter. In 4.2.0 If TCB enters TIME_WAIT state RX and TX buffers are deallocated. This should save you a few KiB of RAM per connection.

Best regards,

Karel.

[paulgounot]

Hi Karel,

According to your suggestion, I changed DEFAULT_TIMEWAIT_TIMEOUT in rtcs_sock.h to 20ms. The issue disappears.

With my MQX device connected directly to my PC (round trip time rtt=2ms), I can’t observe any difference in term of TCP performances.

Short term, I will do more tests into the field to confirm, especially with setup where rtt>20ms.

Long term, it is likely the good option for us will be to rebase to MQX 4.2 RTCS version as the socket & TCB management looks smarter for the small memory devices.

May I ask you to follow up with the RTCS development team in Freescale so that they think about a regression test about TCB management memory footprint for the FTP server for the next releases such as the one I proposed in the doc (directories of small and big files with very fast transitions)?

Thanks a lot for your help.

Best regards,

Paul.

De : Karel Mozny

Envoyé : mercredi 25 mars 2015 09:43

À : Paul GOUNOT

Objet : Re: - RTCS running out of memory during TCP data transfer such as FTP

<https://community.freescale.com/>

RTCS running out of memory during TCP data transfer such as FTP

reply from Karel Mozny <https://community.freescale.com/people/karelm.?et=watches.email.thread> in MQX Software Solutions - View the full discussion <https://community.freescale.com/message/497855?et=watches.email.thread#497855>