MQX Software SolutionsのトピックRe: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 23 Jan 2014 20:20:57 GMT

Hi All,

I'm using the HTTP Server from RTCS 4.0.2 and have discovered a problem. When the client webpage sends a large volume of CGI POST messages the entire MQX system can be crashed. What I mean by "high volume" is I have a webpage with a gain slider on it and as it is moved it generates a CGI POST message that is about 100 bytes long. If I only allow the webpage to send one message per second there's no problem but, if I don't pace it, after a few (5, 10) seconds of continuous slider movement, MQX crashes. It's not a problem with my CGI function that process the incoming gain command (I've commented it out and just ignored these messages and the same thing happens).

The main CGI function that is called when any incoming message is received reads from the socket into a temp buffer (allocated for this session), calls the appropriate function to handle the message (based on it's content), frees the memory allocated for the buffer and exits. The problem seems to occur when a new CGI POST message is received while a previous message is still being processed. Has anybody else run into a problem similar to this? When the system crashes (under debug control) I frequently observe that MQX is running the SOCK_STREAM_shutdown() function. Any thoughts on troubleshooting something likes this? Thanks for any replies!

Best Regards,

Tim

_mqx_int CGI_ParseMsg(HTTPSRV_CGI_REQ_STRUCT *param)
     {
     uint_32 lTotalMsgBytes, lReadLength, lBytesRead, lReadval;
     _mqx_int lRetval = 0;
     char_ptr sRecvBuffer = NULL;
     char_ptr sBufferPtr = NULL;
     //
     //----------------------------------------------------------------------------
     if(param->request_method != HTTPSRV_REQ_POST)  
          return(0);  
     //Attempt to allocate a sRecvBuffer to store the received data
     //----------------------------------------------------------------------------
     lTotalMsgBytes = param->content_length;
     sRecvBuffer = _mem_alloc(lTotalMsgBytes);
     if(sRecvBuffer == NULL)
          {
         PSERVE_LogMsg(local->cOurSlot, "CGI_ParseMsg() - Unable To Allocate sRecvBuf", gcLOGENTRY_ERROR);
        return(0);
        }

     //Read in the data until all bytes (per lTotalRecvdMsgBytes value) are received
     //----------------------------------------------------------------------------
     sBufferPtr = sRecvBuffer; //Use sBufferPtr as a movable temp pointer so sRecvBuffer can always point to buffer start
     lBytesRead = 0;
     while(param->content_length > 0)  
        {
        lReadLength = HTTPSRV_cgi_read(param->ses_handle, sBufferPtr, (lTotalMsgBytes-lBytesRead));
        param->content_length -= lReadLength;
        sBufferPtr += lReadLength;
        lBytesRead += lReadLength;
       }

     //Parse / decode the msg here
     //----------------------------------------------------------------------------
     :
     :
     :
     _mem_free(sRecvBuffer);
     return(0);
     }

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 23 Jan 2014 21:52:32 GMT

Hi All,

I was able to observe a crash where the debugger continued to function and was able to view Task Summary and Stack Usage info. Looks like I have two tasks in distress (main and TCP/IP) with no stack overflows. I've tried increasing the stack size in both the RTCS init (_RTCSTASK_stacksize = 100000;) and HttpdServer init ( params.script_stack = 100000; ) but it seems to make no difference. A buffer overflow or a failure to release allocated memory would be my usual suspects, but that doesn't seem to be the case here.

~Tim

TASK SUMMARY

STACK USAGE

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

stevejanisch — Mon, 27 Jan 2014 16:45:13 GMT

Hello Tim:

I can say I've run into this issue in a past lifetime (i.e. a non-Freescale MQX job) about 8 years ago. At the time I was led to believe that this was a fairly common occurrence for several of the open source web servers. CGI messaging creates a new process handler for the request, and pounding the server with CGI messages crashed quite a few of them.

To get around this, a new protocol called FastCGI was developed. FastCGI differs from standard CGI in that the process was persistent, so it reduced overhead since the server did not have to create and then destroy the process handler over and over again.

I know that this doesn't really help you with MQX, but perhaps you may find the information helpful or see if Freescale plans on implementing FastCGI in the library (hopefully someone from Freescale will read this and let us know). It is also possible that you could look into the code of one of the open source servers and implement this in MQX (one in particular that was very good was the Hiawatha web server). At some point in time I plan on using this type of communications in my project, although I am far from the step where this would be necessary. Or at the worst case, perhaps it would be possible to limit the requests on the server... although that probably is much help in the long run aside from stopping the whole OS to crash.

I'm very interested in what you come up with... as I said at some point perhaps we could work on a FastCGI port together.

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 30 Jan 2014 17:08:40 GMT

Hi Steve,

Thanks for the idea. I will look at this problem from that angle and see if I can affect it. Right now I've just dialed back the cgi message frequency to no more then one every 100mS and that seems to work. Not a very satisfying fader experience for the user but it beats crashing the OS. I will post any insights or breakthroughs back to this thread. Thanks again for your reply.

Best,

Tim

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Luis_Garabo — Thu, 30 Jan 2014 20:56:37 GMT

Hi Tim,

Try to add a break point in the function _task_set_error(). Then take a screenshot to the task stack and share here so we can see where in the code these invalid pointers are coming up. Maybe we can figure out how to solve that.

Regards,

Garabo

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Wed, 12 Feb 2014 22:38:24 GMT

Will do Garabo, thanks for the tip.

Best,

Tim

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 13 Feb 2014 00:01:26 GMT

Hi Garabo,

I haven't been able to catch a crash that leaves the debugger operational yet but, setting the breakpoint you suggested is catching semi-regular errors (mostly during start-up) as MQX attempts to allocate a memory partition in support of processing a received TCP/IP packet. The task_set_error() function parameter "new_error_code" value is 2050 and the function _partition_alloc_internal() that called task_set_error() passed it the constant "PARTITION_OUT_OF_BLOCKS". This error condition seems to occur semi-regularly during operation of the system (especially during boot up) but doesn't normally seem to crash things.

Not sure that this is the task stack condition during a crash caused by heavy CGI activity but it may be a clue as to what's happening? I will keep attempting to catch a breakpoint hit that coincides with the CGI activity crash and post the stack if I catch it. Any thoughts on what I might do to prevent this error condition? The system (MQX 3.8.1.1 with RTCS 4.01) is running on a TWR-MPC-5125 module with 256MB of DDR memory and I have tons of working ram available to my application if I need it so I don' t think it's a shortage of ram, right? Any config changes I can make to try and make more memory available to MQK for this operation? Thanks for your help.

Best Regards,

Tim

Also, just caught this error during normal (non heavy CGI) operation. new_error_code = 3, _mem_free() passed MQX_NOT_RESOURCE_OWNER when called _task_set_error().

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Luis_Garabo — Thu, 13 Feb 2014 19:47:27 GMT

Hi Tim,

Perhaps you are getting out of PCBs try to add this before the RTCS task is created:

_RTCSPCB_init = 4;

_RTCSPCB_grow = 2;

_RTCSPCB_max = 20;

_RTCS_msgpool_init = 4;

_RTCS_msgpool_grow = 2;

_RTCS_msgpool_max = 20;

_RTCS_socket_part_init = 4;

_RTCS_socket_part_grow = 2;

_RTCS_socket_part_max = 20;

I hope this helps you.

Regards,

Garabo

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Fri, 14 Feb 2014 17:51:12 GMT

Hi Garabo,

Thank you for your suggestion, the settings you recommended were already specified in my application. For fun I tried doubling each value but experienced the same OS crash upon consistent CGI message activity. I did observe a slightly different task stack upon a task_set_error() function breakpoint (last night before doubling the _RTCS_ values) and am posting it here. It appears that the TCP/IP task is out of memory. Do you know how I increase the memory available to that task? Thanks for your help.

Best Regards,

Tim

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Fri, 14 Feb 2014 18:19:32 GMT

I have noticed that since I doubled the _RTCS_ values from the defaults, I haven't seen the task_set_error() function breaking on any more RTCS or TCP errors (that's a good thing : )... Continuous CGI activity still crashes the OS though : ( ... After a dozen or so crashes, the task_set_error() function broke with this stack capture:

The error being logged was MQX_INVALID_POINTER. Maybe the problem isn't with the _RTCS or the TCP/IP task after all? Thanks for your help.

Best,

Tim

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Martin_ — Mon, 17 Feb 2014 09:56:16 GMT

Hi Tim,

please give a check with the attached scheduler change in the PSP project:

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Tue, 18 Feb 2014 19:20:49 GMT

Hi Martin and Garabo,

I placed the new dispatch.s file in the "C:\Program Files (x86)\Freescale\Freescale MQX 3.8.1.1\mqx\source\psp\powerpc" folder (replacing the existing file) then rebuilt MQX and tried my application again but, unfortunately got the same results. I believe this is occurring when the CGI messages come in faster then they can be serviced and multiple sockets are opened to handle them. I'm pasting in a couple of screen caps that I got when my application hit a breakpoint I set in the _task_set_error() function (Indicated error number 4 which I believe is OUT_OF_MEMORY). Is there any other info I could try and capture that might provide more clues as to what's happening? Thanks much for your suggestion of trying the modification to the dispatch.s file, I appreciate it. Hope all is well.

Best Regards,

Tim Hutchinson

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 27 Feb 2014 00:06:25 GMT

Hi Martin and Garabo,

I'm starting to think that a high volume of CGI message traffic is not required to cause the MQX crash. I'm thinking that it just provides more opportunity for a crash. It looks like there may be an issue with the amount of memory allocated to the TCP/IP task (not the stack size). Do you know how I can increase that amount? How about any other information I might examine to determine what's happening here? Hope all is well.

Best Regards,

Tim

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Luis_Garabo — Thu, 27 Feb 2014 14:52:11 GMT

Hi Tim,

A good way to increase the TCPIP memory dedicated for it is by adding this:

_RTCSTASK_stacksize = 4500;

I hope this helps,

Regards,

Garabo

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 27 Feb 2014 17:23:16 GMT

Hi Garabo,

Looking at the above stack usage screen capture it looks like the stack size for the TCP/IP task isn't the problem with only 17% utilization. It looks like the RTCS task was trying to allocate a memory chunk (400 to 500 bytes if I remember correctly) and was unable to do so. Examining the MQX/Memory Pools and MQX/Memory blocks info provided by CodeWarrior indicated there was aprox 130MB of RAM available to the system, yet the call to _mem_alloc_zero() failed and _task_set_error() (where I had a breakpoint set) was called to document that error. It seems to me that with around 130MB of RAM available to MQX that _mem_alloc_zero() should have been able to allocate a few hundred bytes?

I'm trying to figure out where to look to determine how _mem_alloc_zero() could've been unable to provide the requested memory. Perhaps this problem wasn't the "cause" of the crash but just a symptom of some other problem? The MQX project seems to be completely stable until the web server is called into action, then it's only a matter of time before I run into this crash. Maybe it's an issue with porting the RTCS from 4.0.2 into MQX 3.8.1.1? Any thoughts on further troubleshooting? Thanks!

Best Regards,

Tim

Re: Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Thu, 27 Feb 2014 19:47:27 GMT

Hi Martin and Garabo,

I've had a thought about the crashes I'm seeing and wonder if something I'm doing in my code is the problem. When I need to send a reply to a web client, I build that reply in a temp buffer (pBuf in example below), issue the response using HTTPSRV_cgi_write(), and then free the buffer like this:

uint_16 CGI_ServicePwordMsg(HTTPSRV_CGI_REQ_STRUCT *param, char_ptr pRecvBuffer)
    {
    lReplyLen;
    char_ptr pBuf = NULL;
    HTTPSRV_CGI_RES_STRUCT response;

    pBuf = _mem_alloc(256);    //Allocate a 256 byte temp buffer
    if(pBuf == NULL)
        return(0);

/*
Function code goes here
  :
  lReplyLen = ???; //lReplyLen is set to actual number of bytes loaded in the buffer
*/

    //Fill out the response info and write the data
    //----------------------------------------------------------------------------
    ISSUE_RESPONSE:
    response.ses_handle = param->ses_handle;
    response.content_type = HTTPSRV_CONTENT_TYPE_PLAIN;
    response.status_code = 200;
    response.data = pBuf;
    response.data_length = lReplyLen;
    response.content_length = response.data_length;
    HTTPSRV_cgi_write(&response);

    //Release the allocated receive buffer
    //----------------------------------------------------------------------------
    if(_mem_free(pBuf) != MQX_OK)        //<-- OK TO RELEASE pBUF MEMORY HERE ???
    return(0);

    return(response.content_length);
    }

I'm wondering if my freeing the buffer immediately after the HTTPSRV_cgi_write() call is causing a problem? Perhaps the buffer memory can't be released back to MQX until after the reply transmission has completed? Perhaps a better practice is to allocate a persistent buffer to support replies to CGI service requests? I guess another good question would be, does HTTPSRV_cgi_write() block until it is finished transmitting the contents of the buffer? Thanks!

Best,

Tim

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Martin_ — Tue, 04 Mar 2014 07:27:44 GMT

Hi Tim, try using "_mem_alloc_system()" to allocate your memory (or _mem_alloc_system_zero() in case the memory needs to be cleared".

Martin

Re: Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Tue, 04 Mar 2014 17:40:28 GMT

Hi Martin,

I will give it a try and report back here. Did you have any opinion on my practice of freeing my temporary memory buffer immediately after calling HTTPSRV_cgi_write()? Hope all is well.

Best,

Tim

Re: Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

Tim562 — Tue, 04 Mar 2014 19:33:03 GMT

Hi Martin,

I changed all calls to _mem_alloc() in the CGI service functions to be calls to _mem_alloc_system() with pretty much the same results (out of memory crash). I had a breakpoint set in the _task_set_error() function and when it hit (on an attempt to allocate 496 bytes of memory) I made screen captures of the situation as reported by Code Warrior. The breakpoint was in place at application startup and no hits were encountered (at startup, init and running) until the web interface started servicing CGI client requests. I'm pasting in the various screen captures and attaching a larger capture of the MQX memory blocks (I thought it too large to paste into this msg). I'm also attaching the extram.lcf file for the project in case it sheds any light. I've noticed that the MQX Memory blocks report indicates 87MB of memory available from 0x0A00_0420 to 0x0F7F_FFA0 (aprox 3.0MB used) which the extram.lcf file indicates is in the "kernel" memory area. I appreciate your suggestions Martin and If you can think of anything I might be able to do to gather more info about what's happening I would be grateful. Thanks!

Best,

Tim

Captures were taken when _mem_alloc_zero() attempted to allocate 496 bytes and failed.

Current Stack when _task_set_error() breakpoint hit:

MQX Task Summary when _task_set_error() breakpoint hit (No TCP/IP error until after _task_set_error() returned):

MQX Stack Usage Report (Notice no overflows):

MQX Memory Pools report (tons of available memory right?):

Re: HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

yb — Tue, 18 Mar 2014 09:48:10 GMT

Hi Tim,

I've juste read your posts and I think I have the same problem : https://community.freescale.com/message/388034#388034

For me, it's not with CGI requests, but with any request (SHTML, HTML, GIF, JPEG or other file extensions).

I there are too many requests too close, the HTTPSRV will crashed...

I tried to add a patch in an external task to kill the open socket on timeout, but the result is not perfect.

Have you some fresh informations about your problem ?

Yvan