ColdFire/68K Microcontrollers and ProcessorsのトピックRe: interniche

interniche

wkurzbauer — Mon, 25 Jun 2018 11:57:32 GMT

Hello,

Please can you tell me the most up-to-date version of Nichelite for coldfire processors.

I run a digibutler board (MFC52531) from the lektor magaize in the year 2008.

As I have problems with the TCP/IP stack (heap problems) I would like to know if

eventually a newer revision than mine (rev.3.2) exists.

If so please can you also provide me a link to it.

Thank you

Werner Kurzbauer

Re: interniche

TomE — Tue, 26 Jun 2018 14:07:58 GMT

3.2 is the latest version on Freescale's web site, dated Feb 2009.

I have a version, still 3.2, but with "LIST OF CHANGES MADE BY MARC VANDENHENDE, 2011-05-24".

Here's a link to a post in this forum with a link to a copy on Dropbox, dated November 2013

https://community.nxp.com/message/101060?commentID=101060#comment-101060

Not surprisingly, it isn't on Dropbox any more.

So I've attached it to this post. I don't know if it fixes your problem. From a quick look at the "Changes" document I couldn't see anything that obviously matched.

You may have to debug it, but at least you'll be starting from something with less bugs.

Watch out for the big and horrible "CW 7.2 ABI change" which might affect this code.

If you search this forum for "nichelite" you'll find about 56 posts. Have you looked through them, and did you find anything there?

Tom

Re: interniche

wkurzbauer — Tue, 26 Jun 2018 15:21:17 GMT

Hi,

Thank you for your quick response!!! Actually I did not expect that anyone is still responding on a "digibutler" topic...

I currently try to adapt the rev.3.2 to my digibutler board (as specifically the cpu definitions appear to have changed from

the previous Nichelite version).

As you are also using a Digibutler may I ask if you also noticed the problem that after some days the server stopped apparently having a heap allocation problem (calloc1.c and npalloc.c report problems via printf (RS232) and the ftp stack stop to work).

I tried to find the problem but only with little success as it might really be located deep in the guts of the system.

Maybe you found a solution...

My hope was this new revision but the migration of my own code and the adaption to my digibutler board seem to require much more effort than hoped.

Did you already adapt the cpu definitions of rev.3.2. to the digibutler board?

I use codewarrior 7.1 so I hope I do not have any problem from the compiler side (I do not really what is "CW 7.2 ABI change" !!??).

If you are still interested in fiddling around with the digibutler board

I really appreciate an exchange of knowledge and experience!!

Thanks a lot and regards

Werner

Re: interniche

TomE — Wed, 27 Jun 2018 00:02:08 GMT

I have never heard of the "Digibutler" until your post. I just read Marc Vandenhende's post back in 2013 and took a copy of the stack as I knew someone would be asking for it 5 years later, and you did :-).

> I do not really what is "CW 7.2 ABI change" !!??

If you're working with CodeWarrior it is important that you do. You're using 7.1, and I assume the "old system" that works with all the sample code released to that date and all the code in the App Notes. Then in 7.2 the ABI was changed and that broke all existing code (any code that used assembly). I assume that Marc's version of the code has been changed to work with the new one.

You should still compare Marc's sources with your original sources to find the bugs he's fixed in case he fixed your one. But you'll have to know how to recognise the changes that the 7.1 to 7.2 migration caused otherwise that may confuse you.

Here's some notes I keep on that:

https://community.freescale.com/message/401840#401840

Read my last post here:

https://community.freescale.com/message/356263#356263

This mentions that the parameter passing changed between CW7.1 and CW7.2:

Re: Updating CW 7.1 to 7.2, using MQX 3.3, missing librarys

https://community.freescale.com/message/66959#66959

Marc Vandenhende Jun 17, 2010 11:11 PM (in response to Mark Armbrust)

"CW7.2 uses a different way to pass parameters to functions (REG_ABI) compared to CW7.1, which uses STD_ABI by default. These methods are not compatible with each other."

Search (at top right of this page) for "REG_ABI".

Other posts said that prior to 7.2 you could select either option, and all the libraries and example code on Freescale's web site defaulted to STD_ABI. With CW7.2 that option was removed, and it changed to REG_ABI. So all existing Assembly code broke, and none of the sample code (in App Notes and so on) got updated. This traps everybody, but isn't documented anywhere that I can find. It might be in the documentation provided with CW somewhere, I've never had a copy to check.

7.12 was released in June 2nd, 2009.

7.2 released 22 Jan 2010.

> apparently having a heap allocation problem

Either the Nichelite code isn't correctly freeing memory that it has malloced, is using data after being freed, is double-freeing, or your code that interfaces to it isn't allocating or freeing properly. The "standard way" to diagnose this is to completely replace the memory allocator with one that has diagnostics and that can detect and report bad operations.

Google finds this. Try replacing your "malloc" with something like this and then run:

http://scottmcpeak.com/memory-errors/

> after some days the server stopped

It is very hard to take a "demo project" that runs once, and then turn it into something that can run for weeks or months. That usually takes a lot of work, sometimes even a rewrite.

> calloc1.c and npalloc.c

Those files don't exist in the Nichelite TCP/IP sources. There's only "src/common/alloc.c" which is worryingly simple and has a comment at the top that says "99% of this code stolen/borrowed from the K&R C examples". So your allocator must be someone else's. I'm guessing (from Google searches) it is the one in the OS you're using.

Have you checked here for updates? I suspect Coldfire won't have been supported for years though. Correct - I've just run Nichelite's "Configurator" and Coldfire is nowhere to be seen.

https://www.hcc-embedded.com/embedded-systems-software-products/tcp-stack-networking

p.s. (I don't know why this editor has made the rest of my text BLUE or how to fix it, I've tried three times already.

Tom

Re: interniche

vier_kuifjes — Wed, 27 Jun 2018 10:59:10 GMT

I know about the Digibutler but never used it.

It may well be that you can run the NicheLite stack on the Digibutler without modification. I had it running on both MCF52233 and MCF52235 MCUs without change. You do have to compile the stack on codewarrior 7.2 however, preferrably with all updates installed. Some of the modifications I did were to make it compatible with this version and have to do with the REG_ABI thingy. When REG_ABI is used, parameters are passed to subroutines via CPU registers instead of the stack. This is taken care of automatically by the compiler, but if the program also contains assembly language routines that assume parameters are being passed by the stack, these routines will no longer function properly. Interniche does contain some assembly language routines, which I modified to take parameters via the CPU registers instead of the stack.

Re: interniche

wkurzbauer — Wed, 27 Jun 2018 16:52:23 GMT

Hi,

Thanks for the responses. The digibutler board uses the MFC52231 and was originally supplied with CW 6.3, the freescale_HTTP_Web_Server by Eric Gregori and cpu definitions tailored to the board.

I used CW7.1 to compile my code and I hoped that this CW version can also be used with the rev.3.2 of nichelite as long as I use the MFC52231 (I do not know if rev.3.2 still supports MFC52231)?

I tried to install a CW7.2 version though...but it still denies to install on my 64-bit machine :smileysad:

@tom: I was a little sloppy concerning the functions reporting the error:

these are calloc1() in MEMIO.C and npalloc() in iutil.c

I will follow your suggestion to replace the memory allocator....

However as I used the freescale_HTTP_Web_Server by Eric Gregori (still available at NXP) I hoped that such severe bugs do not occur. Also in the forums the issue is only rarely discussed and in fact no solution provided,,,,

When I went through the code changes in rev.3.2 I noticed that FSL (I don't know for who the shortcut stands for) apparently modified and revised the freescale ("Gregori") version and not a "generic" interniche version. So maybe the bug is still not removed...

cheers

Werner

Re: interniche

TomE — Wed, 27 Jun 2018 23:48:22 GMT

> these are calloc1() in MEMIO.C and npalloc() in iutil.c

ColdFire_Lite_M52259EVB/src/projects/NicheLite/Source/cf_specific/iutil.c

ColdFire_Lite_M52259EVB/src/projects/NicheLite/Source/misclib/MEMIO.C

Enable "NPDEBUG" in MEMIO.C and see if it finds anything if you haven't done that already. Also "MEMIO_DEBUG".

Also note that "npalloc():" is documented as "Wrappers for heap calls, with memory clearing and counters", so write some code to print those counters. The only current way is to enable "HEAP_STATS" which prints on every alloc and free. That may overload your system, or it may be the way to debug this.

The most likely problem is a memory leak. Something is allocating memory and not freeing it. On a system with a lot of free memory you either have to actually TEST code that uses malloc() and free() to make sure it isn't leaking, or you have to wait a long time until all the memory is used up. Or you can change the heap allocation function (mheap_init()) to return an artificially small heap to force these problems to happen sooner. It is likely nobody has done that with some or all parts of this code, so that's now your job.

Also enable "HEAP_STATS" and periodically (once a minute should be enough) call "mh_stats()". I would suggest you improve the debugging by adding code to print the "npalloc()" wrapper counters at the same time. That should tell you if you're suffering from a "memory leak" where some code is allocating a block and then never freeing it. You should also be able to see if a "leak" corresponds to a particular operation on the stack or the web server.

Better still, why not generate a Web Page that gives all of the current memory statistics? Then you can monitor it in real time!

For these problems that don't happen all that often, it is likely the "leak" is in an error case, like a bad or aborted web page access, or a TCP socket that is aborted rather than properly closed. Windows has a habit of not closing TCP connections properly, which can cause embedded systems to run out of memory or sockets with all the connections in "Timeout" state.

> FSL (I don't know for who the shortcut stands for) apparently modified and revised the freescale

https://en.wikipedia.org/wiki/FSL

It's Freescale's New York Stock Exchange Ticker symbol too.

Tom

Re: interniche

wkurzbauer — Thu, 28 Jun 2018 15:18:44 GMT

Hi,

I enabled "HEAP_STATS" to get the heap information but strange enough, apart from the enabled functionality,

every time when a web-page is loaded from my freescale server the following message appears 3 times:

Access error -- PC = 0x00006F66

Error on operand write

Appartently the mcf5xxx_exception_handler (void *framep) in mcf5xxx.c is called...

The reported address is in the module TCPIN.C

in the function "tcp_xmit_timer(struct tcpcb * tp)" at address 0x00006F66:

;
; 1384:    TCPT_RANGESET(tp->t_rxtcur,
; 1385:        (short)(((tp->t_srtt >> 2) + tp->t_rttvar) >> 1),
; 1386:        TCPTV_MIN, TCPTV_REXMTMAX);
;
0x00006F4C 0x303C0080               move.w   #128,d0               ; '..'
0x00006F50 0x3F40000A               move.w   d0,10(a7)
0x00006F54 0x7002                   moveq    #2,d0
0x00006F56 0x3F400006               move.w   d0,6(a7)
0x00006F5A 0x202E0074               move.l   116(a6),d0
0x00006F5E 0xE480                   asr.l    #2,d0
0x00006F60 0xD0AE0078               add.l    120(a6),d0
0x00006F64 0xE280                   asr.l    #1,d0
0x00006F66 0x3F400002               move.w   d0,2(a7)        <<<<<<<<<<<<<<<<<<<<<<<<<<<<
0x00006F6A 0x4EB900005ECC           jsr      0x00005ecc            ; 0x00005ecc
0x00006F70 0x48C0                   ext.l    d0
0x00006F72 0x2D400020               move.l   d0,32(a6)
;
; 1387: }
;

I do not know why this exception is only shown when "HEAP_STATS" is enabled and I do not know wheter this

is a compiler problem or a code problem. However it might well be one step towards a solution of the problem...

May I ask you if you have any idea...

cheers

Werner

Re: interniche

TomE — Fri, 29 Jun 2018 09:02:37 GMT

This is all standard debugging. Do you have a debug pod? Can you single-step? Can you put a breakpoint on that instruction and dump the stack and the registers to see what's going on?

You haven't said if you have turned NPDEBUG and/or MEMIO_DEBUG on. Have you?

So this only happens with "HEAP_STATS".

I don't have the Nichelite code easily viewable, but you should do what I would. Which is to look at the sources of the multiple layers of the memory allocator, and then see what CHANGES in that code when you turn the debugs on.

I remember that one of the debugs added "poison" or a "pattern" to freed blocks. That is to see if something is writing to a block after it has been freed. It is likely that your code is READING from a block after it has been freed, and it getting upset because it is reading the "poison" value. This doesn't happen without the debug flags as it is probably reading "Stale, but legitimate" data.

The code is in "tcp_xmit_timer()". It is possible that the TCP connection creates a "Transmit Timer" and puts it on a "Timer Queue" to be activated. When the socket closes it is important/essential that that close cleanly removes all references to freed data, including anything it puts on a timer queue. Maybe it doesn't. You'll have to read the code and draw up a lot of flowcharts, or single-step (a lot) or add debug printing to find this.

The error you're getting is strange. The code is setting up to call a function, and is pushing four parameters onto the stack, as 128 -> 10(A7), 2 -> 6(A7) and the sum in the middle to 2(A7). There must be a fourth one you didn't include in the code you quoted. Anyway, if the first two worked, then I can't see how the third one would fail. They're all storing to addresses that are very close (on the stack). It isn't as if one can be a "wild pointer" different to the other ones.

See if you can make the Exception Handler tell you something more useful than what it is doing. Read through the "Exception Handling" chapter of the CPU manual. It would be REALLY useful if the handler could tell you WHICH exception happened. Just printing the "Format Vector Word" would help. If it doesn't to this you should be able to add code to make it do so.

And do all the other things I suggested, like printing statistics and counters so you can see leaks as they happen.

(Edit)

> Anyway, if the first two worked, then I can't see how the third one would fail.

To work that out you have to read the "Coldfire Core" Chapter in the Reference Manual, specifically the "Access Error Exceptions" section, as that would seem to be what you're getting. It said "Access Error" and there are 10 different codings in the Exception Word, four of which are valid, and one matches "Error on Operand Write". For the CFV2 though, the section says:

The V2 ColdFire processor uses an imprecise reporting mechanism for access errors on operand
writes. Because the actual write cycle may be decoupled from the processor’s issuing of the
operation, the signaling of an access error appears to be decoupled from the instruction that
generated the write. Accordingly, the PC contained in the exception stack frame merely represents
the location in the program when the access error was signaled.

So the one thing I can saw with some confidence is that the Write you have marked at "6F66" wasn't the one that failed.

Still, there were TWO previous writes to the stack (memory referenced by A7) and the CFv2 doesn't have a deep enough write pipeline for it to have blown up three or more writes earlier, so there must still be something wrong with the stack address.

Another problem is that it is very hard to get an "Access Error" on the Coldfire. All of memory is mapped, there's no Memory Management unit. You're (probably) not running with Supervisor and User mode, so there's no "memory you can't get to". It doesn't usually flag accesses to address spaces with no memory present either.

You need to drop into the debugger and see what A7 is. You also want to get it to give you a register dump at the time of the exception.

It is possible your system is configured with a small stack, and the extra printing that "HEAP_STATS" is doing is walking off the end of the stack, causing corruptions. This is another fault condition that CPUs with memory management catch for you, but simpler ones like the CFV2 let you get wrong without giving you any help at all.

Tom

Re: interniche

wkurzbauer — Fri, 06 Jul 2018 08:06:48 GMT

Hi Tom,

Thanks for your useful advices.

After some investigations if think I could localize the problem.

First of all the statics variable in memio.c (mh_totfree) is wrongly set in the mem_free() function and has a negative overflow after some time (the length of the header structure is not properly registered in the statictics in case of blocks are freed or allocated which are not at the end of the heap).

The heapstat monitor (mh_stats()) therefore reports false values.

However the heap as such seems to work properly.

My problem is the freescale HTTP server:

When my server is connected to the net it is sometimes "visited" by some unwanted guests (probably IP-scanners) and if this is the case the memory after the "attack" is not properly freed.

It seems that my bowser sents is able to send a response without getting an error fom m_send (M_SOCK...) but then the session is not properly terminated and I assume the socket is not properly closed or something like that?

So the available hep gets smaller and smaller and after some time the system collapses....

Now I have to dive into the freescale server code to really find the problem (I again hope that the TCP/IP stack operates properly). As the attacks are not predictable this can take some time.....

May I ask if this problem is known to you?

cheers

Werner

Re: interniche

TomE — Sat, 07 Jul 2018 04:07:22 GMT

> The heapstat monitor (mh_stats()) therefore reports false values.

Have you fixed it so you can trust it?

> My problem is the freescale HTTP server:

> When my server is connected to the net it is sometimes "visited" by some

> unwanted guests (probably IP-scanners) and if this is the case the memory

> after the "attack" is not properly freed.

Are you sure that's what is happening? I suspect there is no bug.

But what should you do if the server does have a bug that leads to a memory leak?

Guess what Apache does? The last time I looked at the code (10 years ago, so it might have changes), an Apache server spawns multiple threads forming a "pool". Each one is allocated to incoming client requests. After a client has responded to about 100 requests, IT IS KILLED AND RESTARTED! That is because it is assumed that it will leak, and rather than actually fix the bugs it is easier to simply kill it and clean up any mess. Of course the servers are running on top of an OS that keeps a separate memory pool per process and keeps a list of all file handles and sockets. So everything is cleanly released when the "kill" happens.

This is a lot harder to do on embedded systems!

You should be able to tell from the memory statistics how big each "leak" is. Then compare that "leak size" with all the "request sized" looking for a match. You could even log every allocation and free, logging the caller function address. That would let you know what code isn't freeing when it should.

A standard "problem" (actually design characteristic) of TCP/IP is that a connection is expected to stay open FOR EVER unless deliberately closed. And it will stay open without traffic and without sending any data. This is not a bug.

There is a "Feature" of standard (large, on Linux and Windows) TCP/IP systems called "Keepalive". There's a conflict between selfish programmers who would like "keepalives" every few seconds and the "Old Net Gods" who rightly said "that doesn't SCALE" and so required the MINIMUM time for the keepalives to be 2 HOURS:

Keepalive - Wikipedia

So I suspect your "attacks" are performing HTTP Opens which are first causing TCP Opens, and then either aren't sending any data, or are sending a bit and then go off to attack something else without closing the connection.

The only thing that will cause the connection to drop is either a deliberate timeout in the server, or something that makes the server want to send some data on the connection. The retries on the send will eventually close the connection, but NOTHING ELSE WILL.

The proper way to monitor this is to add a command to the system (maybe even a web page) that lists or counts all the open connections. Linux machines have "netstat -s". Windows has "netstat". You want to write an equivalent for your system. Especially if you can add a column listing how long the socket has been open for. Then you can see if the number of open sockets is increasing without limit (and taking up memory without limit).

Maybe modern servers have timeouts to handle this problem. Maybe the Freescale one is so old (and meant as a demo rather than a bulletproof on-the-internet server) that it doesn't have any sort of timeout, or it does, but it isn't turned on.

Is this the one you're running, or is it another one? Where's the source?

"Freescale\NicheliteColdfireLite\7.2 REG_ABI 20110524.zip\7.2 REG_ABI 20110524\CF_Lite_v3.2_MVDH_20110524_CW7.2\src\projects\example\freescale_HTTP_Web_Server"

You could rely on KEEPALIVEs to close sockets. But you might run out of them or memory way before two hours is up.

Here's how to configure it in the 2009 version, with a recommendation on dropping the timeout from 2 hours to as low as a minute (for embedded systems). It even includes source code and documents a bug in that version that means it won't work properly with Windows, but it also gives a patch to fix that.

https://www.nxp.com/docs/en/application-note/AN10775.pdf

Here's a post from 2010 saying it doesn't time out. But the original poster may not have turned it on. There's a post in there from Marc on his fixes to Nichelite too. I've searched the Freescale Nichelite documentation and didn't get a match on "keepalive" (or even "keep" or "timeout").

Here's what happens if you search the code for keywords:

$ find . -type f | xargs grep KEEPALIVE
./NicheLite/Source/h/msock.h:#define SO_KEEPALIVE 0x0008 /* keep connections alive */
./NicheLite/Source/mtcp/tcp_timr.c: if ((((M_SOCK)(tp->t_inpcb))->so_options & SO_KEEPALIVE) &&

$ find . -type f | xargs grep TCPT_KEEP
./NicheLite/Source/h/mtcp.h: * The TCPT_KEEP timer is used to keep connections alive. If an
./NicheLite/Source/h/mtcp.h: * an ack segment in response from the peer. If, despite the TCPT_KEEP
./NicheLite/Source/h/mtcp.h:#define     TCPT_KEEP      2     /* keep alive */
./NicheLite/Source/mtcp/TCPAPI.C:   tp->t_timer[TCPT_KEEP] = TCPTV_KEEP_INIT;   /* initial connect keep alive */
./NicheLite/Source/mtcp/TCPIN.C:   tp->t_timer[TCPT_KEEP] = tcp_keepidle;
./NicheLite/Source/mtcp/TCPIN.C:         tp->t_timer[TCPT_KEEP] = TCPTV_PERSMAX;                //FSL was TCPTV_KEEP_INIT;
./NicheLite/Source/mtcp/tcp_timr.c:   case TCPT_KEEP:   //FSL case 2
./NicheLite/Source/mtcp/tcp_timr.c:         tp->t_timer[TCPT_KEEP] = (short)tcp_keepintvl;
./NicheLite/Source/mtcp/tcp_timr.c:         tp->t_timer[TCPT_KEEP] = (short)tcp_keepidle;

./NicheLite/Source/h/mtcp.h:#define TCPTV_KEEP_INIT (75*PR_SLOWHZ) /* initial connect keep alive */

./NicheLite/Source/h/mtcp.h:#define     PR_SLOWHZ   2 /* TCP ticks per second */
./NicheLite/Source/h/mtcp.h:#define TCPTV_SRTTDFLT    (3*PR_SLOWHZ)        /* assumed RTT if no info */
./NicheLite/Source/h/mtcp.h:#define TCPTV_PERSMIN     (5*PR_SLOWHZ)        /* retransmit persistance */
./NicheLite/Source/h/mtcp.h://#define TCPTV_PERSMAX     (60*PR_SLOWHZ)       /* maximum persist interval */
./NicheLite/Source/h/mtcp.h:#define TCPTV_PERSMAX     (10*PR_SLOWHZ)       /* maximum persist interval */      //FSL lowered
./NicheLite/Source/h/mtcp.h:#define TCPTV_KEEP_INIT   (75*PR_SLOWHZ)       /* initial connect keep alive */
./NicheLite/Source/h/mtcp.h:#define TCPTV_KEEP_IDLE   (120*60*PR_SLOWHZ)   /* dflt time before probing */

There's no demo code that sets that socket option anywhere. Maybe you should add it to the HTTP server where it opens its sockets. And you might like to change the "two hours" definition above for TCPTV_KEEP_IDLE to something quicker. Except there's a "FSL" comment in there that may have changed how this worked.

The simplest way around all of these problems is to just reset the box periodically, or to have a monitor check memory and sockets and reset if it is about to run out. If it already crashes when it runs out of memory then you've already done that :-).

That's assuming it is a TCP problem. It may still be a bug as you've said. In which case it should be easy to find and fix.

Tom

Re: interniche

wkurzbauer — Sat, 07 Jul 2018 10:39:54 GMT

Hi,

I think I localized the problem:

In TCPIN.C in the function tcp_rcv (PACKET pkt)

the following code makes the problem :

if (tiflags & TH_RST)
   {
      switch (tp->t_state)
      {

      case TCPS_SYN_RECEIVED:
         so->error = ECONNREFUSED;
         goto close;

      case TCPS_ESTABLISHED:
         TCP_MIB_INC(tcpEstabResets);     /* keep MIB stats */
      case TCPS_FIN_WAIT_1:
      case TCPS_FIN_WAIT_2:
      case TCPS_CLOSE_WAIT:
         so->error = ECONNRESET;
         close:
         tp->t_state = TCPS_CLOSED;
         TCP_STAT_INC(tcps_drops);
         m_tcpclose(tp);
         if (so->callback)
            so->callback(M_CLOSED, so, NULL);
         GOTO_DROP;      <<<<<<<<<<<<<<<<<<<<<<<----------------------------------------------

      case TCPS_CLOSING:
      case TCPS_LAST_ACK:
      case TCPS_TIME_WAIT:
         m_tcpclose(tp);
         GOTO_DROP;
      }
   }

on the marked location the code jumps to the label "drop":

drop:
tcp_pktfree(pkt);

   /* destroy temporarily created socket */
   if (dropsocket)
      m_delsocket(so);
   return SUCCESS;

It seems that if dropsocket is 0 (and this apparently the case) the socket is not properly deleted. I assume that this is in fact the goal in the above location !?

I changed the code in that I do not jump to the label included

tcp_pktfree(pkt);

m_delsocket(so);

return (0);

directly in the above case option.

I do have to admitt that I do not fully understand what happens but I hope this fixes the problem...

Maybe you know more...

cheers

Werner

Re: interniche

TomE — Mon, 09 Jul 2018 06:10:03 GMT

That doesn't look right.

Looking to see what "dropsocket" is really for, at line 441:

      /*
       * Mark socket as temporary until we're
       * committed to keeping it.  The code at
       * ``drop'' and ``dropwithreset'' check the
       * flag dropsocket to see if the temporary
       * socket created here should be discarded.
       * We mark the socket as discardable until
       * we're committed to it below in TCPS_LISTEN.
       */

What that tells me is that the code is structured to get a socket, and then hand it off to the next level of code, making it IT'S RESPONSIBILITY to dispose of it at the right time. By calling "m_delsocket()".

So search for everywhere else in the code where "m_delsocket()" is called, and you'll find it being called in m_close().

If you dispose of it in "tcp_rcv()" when something else is still using it, you'll risk bad memory corruption when the socket is freed the second time.

So something, possible the HTTP Server isn't following the rules, and may not be calling through to m_close() properly.

It is likely that some simple code was written for Linux or Windows, and rather than closing the socket properly under all circumstances (normal close and error cases) it just bails or aborts or something. On Linux and Windows, when a program does that, the OS cleans up after the program so that a messy, but OK way to write code. There is no OS to do that here - the program has to follow the rules and shut down everything cleanly.

Looking again at the code you quoted:

if (tiflags & TH_RST)

That only happens when a socket is (or was) open and receives a RESET segment from the remote.

What is MEANT to happen is this triggers:

         if (so->callback)
            so->callback(M_CLOSED, so, NULL);
         GOTO_DROP;

The "callback" is meant to be there, and IT is meant to cleanly close the socket. If it is being called and it doesn't close the socket, then that's the bug. If there is no callback registered, and so there's nothing there to perform that operation, then the bug is that it got to here with no callback and nothing closing the socket.

I think a better change would be to change the above to catch this possible case:

         if (so->callback)
            so->callback(M_CLOSED, so, NULL);
         else
            dropsocket = 1;
         GOTO_DROP;

Also add some sort of debug printing code to the above added line (remembering to add braces), to see if that is actually happening. Then you can see if that fixes the problem.

Tom

Re: interniche

wkurzbauer — Tue, 10 Jul 2018 09:57:09 GMT

Hi,

I modified the code according to your suggestion (setting dropsocket = 1 in case of no callback).

However this again lead to a number of undeleted sockets and reducing the heap after each attack.

To my understanding the callback function freescale_http_cmdcb(int code, M_SOCK so, void * data) only closes and deletes the socket (by executing freescale_http_remove( M_SOCK so )) if it was a "VALID_EMG_SESSION".

If not - the callback function is called without closing the socket and the else-branch setting the variable "dropsocket" to 1 is not executed.

Therefore the socket is not deleted after jumping to the label "drop".

I enabled the debug variable netstat and after calling the command "sockets"

a huge number of sockets having no TCPCB were listed...(one can see that the

ip addresses coming from various attackers...):

In Nichlite rev.3.2 issuing the command socket also deletes all sockets having no TCPCB

for (so = (M_SOCK)msoq.q_head; so; so = so->next)
   {
      ns_printf(pio,"%lx, %u.%u.%u.%u, %u->%u, ", /*(long)*/so,
         PUSH_IPADDR(so->fhost), (so->lport), (so->fport));

ns_printf(pio,"0x%x, %u, %u", (unsigned)so->so_options,
(unsigned)so->rcvdq.sb_cc, (unsigned)so->sendq.sb_cc);

      tp = so->tp;
      if(tp)
         ns_printf(pio, ", %ld, %ld, %s\n", tp->snd_una, tp->snd_nxt, tcpstates[tp->t_state]);
      else
      {
         ns_printf(pio, " (no TCPCB)\n");
   m_delsocket(so);  //FSL No TCPCB then delete socket (so)...might want to implement this capability elsewhere    <<<--
      }
   }
   return 0;

Thus after issuing the command my heap again nearly growed to its original size (except one socket in FIN_WAIT_2 state which seems to linger in the socket list forever without being deleted..).

The problem is when can I (automatically) delete all those sockets having no TCPCB without causing any further troubles?

cheers

Werner

Re: interniche

TomE — Wed, 11 Jul 2018 05:21:40 GMT

> To my understanding the callback function freescale_http_cmdcb ...

Maybe it is wrong or wrongheaded. The TCP "closing" code (in its various flavours) has to signal back to the user that a "request to close" has happened or a close has completed. The socket interface to the "user code" (the http server) doesn't support asynchronous notification of this sort of thing, so the only "signal" is a return code to the "read()", "recv()", "send()" and other calls. The user code should then "close()" the socket.

The underlying way it has to work is documented here, and best shown in the diagram on page 23:

https://tools.ietf.org/html/rfc793

An easier to read version is here:

https://en.wikipedia.org/wiki/Transmission_Control_Protocol#/media/File:Tcp_state_diagram_fixed_new.svg

From the second of the above, if the other end terminates an "ESTABLISHED" connection by sending "FIN", the stack sends "ACK" and goes to "CLOSE WAIT". Somehow it signals this state to the user that then performs the "CLOSE", the stack send a "FIN" and waits for an "ACK". Note it will stay there "forever" according to the diagram and the original protocol. In practice there has to be a time out here. If the User Code sends a "CLOSE" to "ESTABLISHED", then you have to end up in "TIME_WAIT". If you ask Google "how long is TCP TIME WAIT timeout" it'll tell you "4 minutes".

If you're in "FIN WAIT 2" then it means your user code issued "CLOSE", the stack send FIN, received the ACK and from "FIN WAIT 2" is waiting for the other end to play nice and send its "FIN" to which this stack will send "ACK". The other end isn't playing nice. Its stack has signaled the event at its end, and is waiting for that user code to call "CLOSE", and it isn't. If it is an "attacker" there's no reason for it to play nice. There's another path not shown on that diagram. If your code has sent data or the FIN it will retry it until it gets the ACK or until there's a very long timeout, at which point it'll abort somehow. But it can legitimately stay in "FIN WAIT 2" "for ever". People have added timers to get out of this jail. You'll have to check the Nichelite code to see if it has one of these. For reference, here's something Microsoft did:

https://docs.microsoft.com/en-us/windows-hardware/drivers/network/fin-wait-2-timer

Here's the hooks under Linux:

https://serverfault.com/questions/7689/how-do-i-get-rid-of-sockets-in-fin-wait1-state

> The problem is when can I (automatically) delete all those sockets having no TCPCB without causing any further troubles?

The problem is "how did they get like that?". There's something wrong with the stack or user code that is doing that.

> m_delsocket(so);  //FSL No TCPCB then delete socket (so)...might want to implement this capability elsewhere

When Freescale worked on this code they added at least 704 "FSL" comments as they went about changing the code. So you "might want to implement this capability elsewhere".

This comment might be useful:

//FSL **have newly created so socket have pointer to newly created tcpcb
//FSL Note: there is always a socket/tcpcb pair and they have pointer to each other

That may mean if there's a socket with "so->tp" being NULL then it is safe to delete it. As long as it isn't on a timer queue or something. Make sure the "socket delete" function stops all timers and removes it from all queues.

Tom

Re: interniche

wkurzbauer — Wed, 11 Jul 2018 20:12:04 GMT

Hi,

After studying your hints and advices I finally decided to implement a socket purge function which is called by the taskmanager every xx seconds (I chose 20min).

The function removes all sockets without TCPCB and all sockets in FIN_WAIT_2 (whereby at the first call the

waiting sockets are only marked (i.e. put in a list) and finally deleted after CNT_DOWN times calling the purge function.

Up until now I was able to keep the heap.... :smileywink:

/* FUNCTION: purge sockets()
*
* PARAM1: void * pio
*
* RETURNS: nothing
*/
#define MAX_STUCK_SOCKETS 5
#define CNT_DOWN 2

typedef struct stuck_socket
   {
   M_SOCK socket;
   int timeout;
   } SOCKET_TIMEOUT_STRUCT;

SOCKET_TIMEOUT_STRUCT stuck_socket_list [MAX_STUCK_SOCKETS];

void purge_socket_list_init (void)
   {
   int i;
   for (i=0;i < MAX_STUCK_SOCKETS; i++ )
       {
       stuck_socket_list[i].timeout = 0;   //initialize list
       stuck_socket_list[i].socket = 0;
       }
   }

int purge_sockets (void * pio)
{
   M_SOCK so;
   struct tcpcb * tp;
   char i,found = 0;

   ns_printf(pio,"purging sockets...\n");
   if (msoq.q_len == 0)    //nothing to do
   {
      ns_printf(pio,"No TCP sockets...\n");
      return 0;
   }

   for (so = (M_SOCK)msoq.q_head; so; so = so->next)   // loop through the socket chain
       {
       tp = so->tp;
       if(tp)   // check TCBCB
           {
           if (tp->t_state == TCPS_FIN_WAIT_2)   //check wait state
               {
               found = 0;
               for (i=0;(stuck_socket_list[i].socket != so) && (i < MAX_STUCK_SOCKETS); i++)
                   {
                   if (stuck_socket_list[i].socket == so)

{

                                 if (stuck_socket_list[i].timeout == 0)
                                       {
                                       m_delsocket(so);
                                       ns_printf(pio,"Socket %lx, IP: %u.%u.%u.%u deleted (FIN_WAIT_2)\n",so,PUSH_IPADDR(so->fhost));
                                       stuck_socket_list[i].socket = NULL;

                                       }
                                 found = 1;
                                 break;

                                 }
                   }
               if (!found) // not yet in the list -> put stuck socket in the list
                   {
                   i=0;
                   while ((stuck_socket_list[i].socket != 0) && (i < MAX_STUCK_SOCKETS))
                       i++;           // find a free list entry
                   stuck_socket_list[i].socket = so;
                   stuck_socket_list[i].timeout = CNT_DOWN;
                   ns_printf(pio,"Socket %lx, IP: %u.%u.%u.%u pending (FIN_WAIT_2)\n",so,PUSH_IPADDR(so->fhost));
                   }
               }
           }
       else
           {
           m_delsocket(so);
           ns_printf(pio,"Socket %lx, IP: %u.%u.%u.%u deleted (no TCPCB)\n",so,PUSH_IPADDR(so->fhost));
           }
       }

   for (i=0;i < MAX_STUCK_SOCKETS; i++ )
               {
               if (stuck_socket_list[i].timeout == 0)   //delete all orphaned timed out sockets from list
                   stuck_socket_list[i].socket = 0;
               else
                   stuck_socket_list[i].timeout--;       // decrement "waiting" list elemets
               }

   return 0;
}

cheers

Werner

Re: interniche

TomE — Wed, 11 Jul 2018 23:09:53 GMT

Congratulation on finding a fix for this.

Could I also suggest that your function count the number of those sockets that should be deleted, and trigger a purge when the number gets too high as well? If your device comes under "sustained attack", then it could run out of memory in a matter of seconds.

Tom

Re: interniche

wkurzbauer — Tue, 24 Jul 2018 11:49:41 GMT

Hi,

I just would like to report that the solution appears to work well. Up until now no heap problem...

Again thank you for your help!

The sole problem I have now that as long as I am in my LAN everything works fine, but if I access the webserver

from outside it happens quite frequently that packets get lost and the javascript file or the HTML file which is sent by the webserver are not sent correctly (both are about 10kB).

I checked the received code and I found out that !within! the sent files pieces of 1400 bytes (exactly the

MAX_BYTES_TO _SEND in one packet are missing).

Maybe it be that the maximum segment life time (which is currently set

to 2 in main .c:

   /* Heap memory saving trick - reduce the time a TCP socket
    * will linger in CLOSE_WAIT state. For systems with limited
    * heap space and a busy web server, this makes a big difference.
    */
   TCPTV_MSL = 2;    /* set low max seg lifetime default */

is too short? Otherwise I do not understand how parts of the code can be missing within a transferred file in a TCP connection which should guarantee a correct file transfer...

Do you have any possible explanation how this can happen without any reported error neither on the client (browser) side nor on the server side?

Thank you and cheers

Werner

Re: interniche

TomE — Tue, 24 Jul 2018 23:52:07 GMT

That means the TCP code is thoroughly broken. You should ditch it and replace it with something else. If it can't get this right, then I wouldn't trust it at all. It may be "originally broken" or you may have made some changes to it that caused this problem. So first I'd suggest checking every change you've made to the code from the original that you started with, and also check against that "updated" version.

The tricky bit is needing to understand everything about how TCP is meant to work and how all the code works in order to understand the impact of any changes.

I'd suggest trying to change over to a different TCP/IP stack. Have a look at this one:

https://community.nxp.com/thread/57196

The link at the top of that post (there's 9 full pages in the thread) doesn't work - so look here:

https://sourceforge.net/projects/fnet/

There's an active forum on this stack there too.

Your code is doing something really bad. The whole POINT of TCP (when compared to something like UDP) is that transmission errors are expected and must be handled. Put simply, every packet sent has a sequence number. Put properly, every BYTE send has a sequence number and the packets have byte sequence ranges. When a packet goes missing somewhere, the missing range is sent again when the remote doesn't ack receipt.

Packets can go missing "in the wild" on the Internet, in intermediate routers or switches, or even in the Ethernet Driver code in the computer sending or receiving. TCP has to handle all of these.

The fact that you're getting a "valid data stream" with a full-sized Ethernet packet's worth of data going missing means that the part of the TCP stack itself that is meant to perform the retries is broken, and is probably retransmitting the wrong data.

I'm guessing that when the stack has to retransmit from "Sequence N" it wrongly skips forward in the data stream and sends the next lot.

Or it could be that for efficiency and working on small memory systems, the stack doesn't copy the data in order to retransmit it, but relies on the data buffer from the user code (the HTTP Server) remaining valid until it has been acknowledged. If it is doing that there has to be some form of "handshake" back to your HTTP stack to tell it when it can reuse the buffer. If that isn't working, your HTTP Server code might be overwriting the buffer with the next lot of data. So it could be the HTTP Server isn't using the TCP interface properly. I'd check that first. Maybe Nichelite requires a "blocking write" for TCP data transmission and you're using a non-blocking call or something?

Debugging this is HARD as you've got to have network errors in order to trigger this so you can debug it.

Debugging this is EASY as all you have to do is to add a small amount of debugging code to the low-level Ethernet driver (or between TCP and Ethernet) to corrupt every 10th packet. Just have it add one to a byte in the data stream so the checksum is now bad, or to make tracing the problem easier, for every TCP packet sent that is bigger than 1000 bytes (so it's in the middle of a data stream), write a string like "#################" at offset 100 or so. You'll be able to see that in a Wireshark dump. Then run Wireshark on the PC running the web browser, do the transfer, look for the "######" and then see what happens with the sequence numbers and data when the packet is retransmitted.

???? The board you're using is 10 years old. The Nichelite stack dates from 2008 or 2009. How come you're having these problems NOW instead of THEN? Have you just started using this? Why not something more modern?

Tom

Re: interniche

TomE — Mon, 30 Jul 2018 00:17:34 GMT

> Or it could be that for efficiency and working on small memory systems, the stack doesn't

> copy the data in order to retransmit it, but relies on the data buffer from the user code ...

The Nichelite TCP stack allows both methods according to the documentation. There's a "copy this data" standard socket interface" (m)send()) as well as a more complicated "manage the buffer" one (tcp_send()).

Are you using the web server source from here or the previous equivalent:

"7.2 REG_ABI 20110524/CF_Lite_v3.2_MVDH_20110524_CW7.2/src/projects/example/freescale_HTTP_Web_Server/"

if so, then it uses "m_send()" for everything.

BUT it looks like it sets the socket non-blocking:

freescale_http_server.c:  m_ioctl(so, SO_NONBLOCK, NULL); /* make socket non-blocking */
‍‍‍

Actually it sets SOME socket as non-blocking. I'm assuming the socket being set above is the one used in the following function, but that's a big assumption that you should check.

That's fine (to set non-blocking), as long as the sending code checks for that, which it does:

I'm guessing this is the code you're using, please check and see if it is:

"7.2 REG_ABI 20110524/CF_Lite_v3.2_MVDH_20110524_CW7.2/src/projects/example/freescale_HTTP_Web_Server/freescale_http.c":

void freescale_http_send_file( int session )
{
    ...
    length = emg_web_read( session, scratch_ram_for_send, MAX_BYTES_TO_SEND );      
    }

    if( length )    //FSL As long as greater than zero bytes, read then send data to client
    {       
        ...
        if( bytes_sent >= 0)
        {
            error_delay = 0;    //FSL Reset the delay counter
        }
        else
        {
            if(( freescale_http_sessions[session].socket->error == EWOULDBLOCK ) ||
               ( freescale_http_sessions[session].socket->error == ENP_RESOURCE ))
               //FSL added test for ENP_RESOURCE system error
            {
                error_delay++;          //FSL Increment delay counter
                if( error_delay > 80 )  //FSL If delay counter expired, change http state to CLOSE
                {
                    freescale_http_sessions[session].state = EMG_HTTP_STATE_CLOSE;                  
                    printf( "\n\nStack failed due to blocking" );
                }
            }
            
            // If socket reported a error, or would block,
            // Give some extra time to sleep.
            tk_sleep( 10 );     //FSL The sleep lets transmit buffers and heap space free up (hopefully)
        }       
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I think I can see a problem there.

"emg_web_read()" reads data from the file AND ADVANCES THE FILE POINTER. It then tries to send it, and if sent, so far so good. It will then go and send the next lot. If the socket fills up (with a slower-than-local connection), then it returns "EWOULDBLOCK" and the caller has to SAVE that data and send it the next time.

I can't find any code in there that could do that. It doesn't save the data for next time and it doesn't reset the file pointer on the data it just read. It should do the equivalent of an "lseek()", which is "eng_web_rewind()". This function is used in the "token replace" code, but it should also be used on EWOULDBLOCK recovery.

Without that, when the socket blocks it will throw away one whole read, wait a while and then try again with the next lot. It will throw that away too if the data hasn't been acked.

It looks like this code has only been tested on "fast local and perfect networks" and has never run on the "real internet" before. Congratulations for being the first person to try this code that way in about 12 years, or at least to notice that it doesn't work.

The good news in the above is that the Nichelite code seems to be fine. It is just the Freescale Demo code that has this problem.

Tom