HTTP Server 4.0.2 System Crash On Recipt Of High Volume Of CGI POST Messages

[Tim562]

Hi All,

     I'm using the HTTP Server from RTCS 4.0.2 and have discovered a problem. When the client webpage sends a large volume of CGI POST messages the entire MQX system can be crashed. What I mean by "high volume" is I have a webpage with a gain slider on it and as it is moved it generates a CGI POST message that is about 100 bytes long. If I only allow the webpage to send one message per second there's no problem but, if I don't pace it, after a few (5, 10) seconds of continuous slider movement, MQX crashes. It's not a problem with my CGI function that process the incoming gain command (I've commented it out and just ignored these messages and the same thing happens).

     The main CGI function that is called when any incoming message is received reads from the socket into a temp buffer (allocated for this session), calls the appropriate function to handle the message (based on it's content), frees the memory allocated for the buffer and exits. The problem seems to occur when a new CGI POST message is received while a previous message is still being processed. Has anybody else run into a problem similar to this? When the system crashes (under debug control) I frequently observe that MQX is running the SOCK_STREAM_shutdown() function. Any thoughts on troubleshooting something likes this? Thanks for any replies!

Best Regards,

Tim

_mqx_int CGI_ParseMsg(HTTPSRV_CGI_REQ_STRUCT *param)
     {
     uint_32 lTotalMsgBytes, lReadLength, lBytesRead, lReadval;
     _mqx_int lRetval = 0;
     char_ptr sRecvBuffer = NULL;
     char_ptr sBufferPtr = NULL;
     //
     //----------------------------------------------------------------------------
     if(param->request_method != HTTPSRV_REQ_POST)  
          return(0);  
     //Attempt to allocate a sRecvBuffer to store the received data
     //----------------------------------------------------------------------------
     lTotalMsgBytes = param->content_length;
     sRecvBuffer = _mem_alloc(lTotalMsgBytes);
     if(sRecvBuffer == NULL)
          {
         PSERVE_LogMsg(local->cOurSlot, "CGI_ParseMsg() - Unable To Allocate sRecvBuf", gcLOGENTRY_ERROR);
        return(0);
        }

     //Read in the data until all bytes (per lTotalRecvdMsgBytes value) are received
     //----------------------------------------------------------------------------
     sBufferPtr = sRecvBuffer; //Use sBufferPtr as a movable temp pointer so sRecvBuffer can always point to buffer start
     lBytesRead = 0;
     while(param->content_length > 0)  
        {
        lReadLength = HTTPSRV_cgi_read(param->ses_handle, sBufferPtr, (lTotalMsgBytes-lBytesRead));
        param->content_length -= lReadLength;
        sBufferPtr += lReadLength;
        lBytesRead += lReadLength;
       }

     //Parse / decode the msg here
     //----------------------------------------------------------------------------
     :
     :
     :
     _mem_free(sRecvBuffer);
     return(0);
     }

[Tim562]

Hi Garabo,

     I haven't been able to catch a crash that leaves the debugger operational yet but, setting the breakpoint you suggested is catching semi-regular errors (mostly during start-up) as MQX attempts to allocate a memory partition in support of processing a received TCP/IP packet. The task_set_error() function parameter "new_error_code" value is 2050 and the function _partition_alloc_internal() that called task_set_error() passed it the constant "PARTITION_OUT_OF_BLOCKS". This error condition seems to occur semi-regularly during operation of the system (especially during boot up) but doesn't normally seem to crash things.

     Not sure that this is the task stack condition during a crash caused by heavy CGI activity but it may be a clue as to what's happening? I will keep attempting to catch a breakpoint hit that coincides with the CGI activity crash and post the stack if I catch it. Any thoughts on what I might do to prevent this error condition? The system (MQX 3.8.1.1 with RTCS 4.01) is running on a TWR-MPC-5125 module with 256MB of DDR memory and I have tons of working ram available to my application if I need it so I don' t think it's a shortage of ram, right? Any config changes I can make to try and make more memory available to MQK for this operation? Thanks for your help.

Best Regards,

Tim

Also, just caught this error during normal (non heavy CGI) operation. new_error_code = 3, _mem_free() passed MQX_NOT_RESOURCE_OWNER when called _task_set_error().

[Luis_Garabo]

Hi Tim,

Perhaps you are getting out of PCBs try to add this before the RTCS task is created:

_RTCSPCB_init = 4;

_RTCSPCB_grow = 2;

_RTCSPCB_max = 20;

_RTCS_msgpool_init = 4;

_RTCS_msgpool_grow = 2;

_RTCS_msgpool_max  = 20;

_RTCS_socket_part_init = 4;

_RTCS_socket_part_grow = 2;

_RTCS_socket_part_max  = 20;

I hope this helps you.

Regards,

Garabo

[Tim562]

I have noticed that since I doubled the _RTCS_ values from the defaults, I haven't seen the  task_set_error() function breaking on any more RTCS or TCP errors (that's a good thing : )... Continuous CGI activity still crashes the OS though : ( ... After a dozen or so crashes,  the task_set_error() function broke with this stack capture:

The error being logged was MQX_INVALID_POINTER. Maybe the problem isn't with the _RTCS or the TCP/IP task after all? Thanks for your help.

Best,

Tim

[Martin_ · Accepted solution]

Hi Tim, this one looks quite familiar to me, as we recently discovered a problem with _task_destroy_internal() on platforms with floating point context enabled - which is the case for MPC5125 I guess.

We fixed it for MQX 4.1.0, attached you may find the patch. Please give it a try.

I have httpsrv example, browser with 10 web pages open to HTTPSRV, all pages doing cgi requests each 100 ms, in cgi handler I allocate, use and free 256 bytes buffer. Seems working (with the above patch applied).

[Tim562]

Hi Martin,

     Your patch appears to have done the trick! I serviced more than 800,000 CGI messages from multiple web clients simultaneously throughout the day with no crashes. That is a huge relief, thank you. You and your staff do fantastic work and I'm happy to tell whoever asks. Have a great day.

Best regards,

Tim

[Tim562]

Hi Martin,

     This topic is getting a little dense. I just now found your post. I will give this a try and let you know what I find, thanks!

Best,

Tim

[Tim562]

Hi Garabo,

     Thank you for your suggestion, the settings you recommended were already specified in my application. For fun I tried doubling each value but experienced the same OS crash upon consistent CGI message activity. I did observe a slightly different task stack upon a task_set_error() function breakpoint (last night before doubling the _RTCS_ values) and am posting it here. It appears that the TCP/IP task is out of memory. Do you know how I increase the memory available to that task? Thanks for your help.

Best Regards,

Tim

[Tim562]

Will do Garabo, thanks for the tip.

Best,

Tim