MQX Software Solutions中的主题 MQX 3.8.1 httpd session/server deadlock ?

MQX 3.8.1 httpd session/server deadlock ?

pbanta — Tue, 16 Apr 2013 16:57:58 GMT

I'm trying to track down a report of "web server dies". I'm using MQX 3.8.1. I was able to create a condition today where there are two http session tasks that are blocked and where the http server task is also blocked.

Some background

I use Basic Authentication in this server. When a browser makes it's first connection to the server the user is prompted for username and password. I have tested the authentication with IE9, IE10, Safari, Firefox, Chrome & Opera. It works, but IE10 seems to only work once. I've been running one of my servers for a few days and using Chrome to access it. Works every time: I start Chrome, login, browse, close Chrome.

Problem

Today I was logged in with Chrome. I started IE10. I was prompted for username/password and was able to browse the server with no problem. I closed IE10. I started IE10 again and was prompted for username/password. I entered the credentials and the server did not send the page.

I'm connected to the board with a PEMicro JTAG unit so I can poke around. I found the following conditions (I've removed some of the irrelevant stack trace, but left the networking tasks on my development system). There are two http session tasks that are blocked in the same location on a message send. The http server task is blocked because maximum simultaneous sessions is 2.

    ARM Processors, guardian-lce.afx (Suspended)
        Thread [ID: 0x10002] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            5 DummyFn1() dispatch.s:264 0x1000044e
            4 _msgq_receive_internal() ms_recvi.c:220 0x10021c72
            3 _msgq_receive() ms_recv.c:95 0x10021dd2
            2 main_task() main.c:457 0x10000a08
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x10009] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            8 DummyFn1() dispatch.s:172 0x10000446
            7 _time_delay_internal() ti_deli.c:103 0x1001ed6e
            6 _time_delay_for() ti_delfo.c:80 0x1001ede2
            5 _msgq_receive_internal() ms_recvi.c:222 0x10021c7e
            4 _msgq_receive() ms_recv.c:92 0x10021dc2
            3 TCPIP_task() tcpip.c:178 0x1002baf4
            2 RTCS_task() rtcstask.c:70 0x1002a826
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x1000a] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            5 DummyFn1() dispatch.s:172 0x10000446
            4 _lwsem_wait() lws_wait.c:99 0x1001fe40
            3 httpd_server_task() httpd_task.c:178 0x100268f4
            2 RTCS_task() rtcstask.c:70 0x1002a826
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x1000b] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            8 DummyFn1() dispatch.s:264 0x1000044e
            7 _msgq_send_internal() ms_sendi.c:201 0x100217e6
            6 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2
            5 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c
            4 SOCK_STREAM_accept() sstream.c:405 0x10027d20
            3 FTPd_task() ftpd.c:177 0x10026482
            2 RTCS_task() rtcstask.c:70 0x1002a826
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x1000c] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            16 DummyFn1() dispatch.s:264 0x1000044e
            15 _msgq_send_internal() ms_sendi.c:201 0x100217e6
            14 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2
            13 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c
            12 SOCK_STREAM_recv() sstream.c:571 0x10027f16
            11 _io_socket_read() sockio.c:177 0x10024872
            10 _io_read() io_read.c:91 0x1001c524
            9 _io_telnet_read() telnetio.c:234 0x10023e48
            8 _io_fgetc() io_fgetc.c:85 0x1001d102
            7 _io_fgetline() io_fgetl.c:87 0x1001d068
            6 _io_fgets() io_fgets.c:88 0x1001d02e
            5 Shell() shell.c:142 0x100346d0
            4 TELNETSRV_child() telnsrv.c:389 0x10023cc6
            3 TELNETSRV_task() telnsrv.c:306 0x10023c26
            2 RTCS_task() rtcstask.c:70 0x1002a826
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x1000d] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            11 DummyFn1() dispatch.s:264 0x1000044e
            10 _msgq_send_internal() ms_sendi.c:201 0x100217e6
            9 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2
            8 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c
            7 SOCK_STREAM_recv() sstream.c:571 0x10027f16
            6 httpd_readln() httpd_supp.c:243 0x10026c7c
            5 httpd_readreq() httpd.c:312 0x100276ae
            4 httpd_ses_process() httpd.c:609 0x10027972
            3 httpd_session_dynamic_task() httpd_task.c:134 0x1002687e
            2 RTCS_task() rtcstask.c:70 0x1002a826
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x1000e] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
            11 DummyFn1() dispatch.s:264 0x1000044e
            10 _msgq_send_internal() ms_sendi.c:201 0x100217e6
            9 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2
            8 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c
            7 SOCK_STREAM_recv() sstream.c:571 0x10027f16
            6 httpd_readln() httpd_supp.c:243 0x10026c7c
            5 httpd_readreq() httpd.c:312 0x100276ae
            4 httpd_ses_process() httpd.c:609 0x10027972
            3 httpd_session_dynamic_task() httpd_task.c:134 0x1002687e
            2 RTCS_task() rtcstask.c:70 0x1002a826
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490
        Thread [ID: 0x10001] (Suspended: Signal 'Halt' received. Description: User halted thread.)
            2 _mqx_idle_task() idletask.c:65 0x100204fe
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490

I've got the memory to increase the maximum session count to something greater than 2, but I don't think that addresses the root cause. Something appears not to have cleaned up correctly. I'm wondering if I have a problem with handling sessions properly in all the CGI functions.

Any ideas?

Thanks.

Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Fri, 19 Apr 2013 06:16:45 GMT

I suggest to move on to the new httpd server:

https://community.freescale.com/message/320709#320709

Re: MQX 3.8.1 httpd session/server deadlock ?

pbanta — Fri, 03 May 2013 15:26:39 GMT

Hi Martin,

Can I use the new httpd server with MQX 3.8.1?

Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Mon, 06 May 2013 09:15:18 GMT

Yes, it can be used. I tested on MQX 3.8.1 and TWR-MPC5125. There are little differences compared to the MQX 4.0 post:

in rtcs25x.h

change from

#include <partition.h>

change to

#include <part.h>

in rtcs.h add:

#define AF_INET 1

#define AF_INET6 28 /* IPv6 */

#define PF_INET6 AF_INET6 /* Same as the address family. */

in user_config.h

#define sockaddr sockaddr_in

with attached httpsrv.zip (application); add shell.a library into the application build project.

then, builds and was running on target, but I was getting unexpected exception due to low stack size. so I increased stack size for _RTCSTASK_stacksize = 6500; and now the new web server runs on MQX 3.8.1 (tested on TWR-MPC5125).

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Mon, 12 Aug 2013 17:07:30 GMT

Hi Pbanta,

Thanks to posts by Martin and others, I was able to get the new httpd server working with MQX 8.3.1.1 on an existing project but have discovered a problem that I thought you and others might want to know about.

My project makes use of the http POST command to send a binary file from the web client back to my application (it's used to update the application firmware image). This has worked reliably for several legacy projects (including this one) using several older revisions of MQX. Once I installed this latest version of the httpd server this process stopped working. My CGI receive function is called properly as the data arrives and the session->request.content_len value shows the proper data length but my calls to httpd_read() stop short of transferring all incoming data to my temp receive buffer (about 740 bytes short actually). httpd_read() just keeps returning without receiving any data. I restored MQX back to stock revision3.8.1.1 and this process now works properly again. To ensure all data is received before starting to process it, I essentially continue to call http_read() in a while loop until I've read all the data bytes that session->request.content_len indicated were available into a temp receive buffer then I work out of that temp receive buffer. The files I'm transferring are around 900KB in size (some are far larger).

I don't know if you or anyone has had a chance to evaluate this feature of the new httpd server yet but I would be interested to know if anyone has. Also, if I should be doing this differently with the new server, feel free to make a suggestion. By the way, I have found this new server to provide much better performance then the old one so I would sure love to use it, but this needs to work. Thanks to all !

Best Regards,

Tim Hutchinson

Re: Re: MQX 3.8.1 httpd session/server deadlock ?

pbanta — Mon, 12 Aug 2013 19:34:05 GMT

Hi Tim,

I have not switched my firmware update code to the new server yet so I have not seen this problem. My code sounds very similar to yours. If session->request.content_len is > 0 then I make calls to httpd_read() in a loop until bytes_rcvd == total_expected_bytes.

Is the initial value of content_len correct when using the new server?

By the way, I reported a bug to Freescale related to content_len (details below) but I'm not sure it is the cause of the problem you see. If I recall correctly, I found this bug because a network management tool was requesting a very large URL that was not a valid CGI path in my MQX HTTP server.

What I reported:

In RTCS Source/httpd/httpd.c in the httpd_processreq() function there is a bug.

I have included the portion of the code with the bug below. The bug is in the calculation of the length parameter to the httpd_read() call. It is possible to have the following conditions:

1. res == -1

2. session->request.content_len > sizeof(buf)

If these conditions are met you will have buffer overflow. The error is in the comparison

((res > sizeof(buf)) ? sizeof(buf) : session->request.content_len)

Instead, this comparison should be

((session->request.content_len > sizeof(buf)) ? sizeof(buf) : session->request.content_len)

Code Snippet from httpd.c

---------------------------------------

// if request contain data

res = -1;

if ((cp1 = strrchr(cp, '.')) != 0)

{

if (0 == strcasecmp(cp1, ".cgi")) // file extension is cgi, maybe cgi script

{

cgi = 1;

// search cgi script

*cp1 = 0;

res = httpd_cgi(server, session, cp + 1); // cp + 1 = cgi script name begin

*cp1 = '.';

}

if (cgi || session->request.content_len)

{

if (res < 0)

{

if (res != -2)

cgi = 0;

// cgi script not found - read data from socket - discard it

while (session->request.content_len)

{

res = httpd_read(session, buf, (int)((res > sizeof(buf))? sizeof(buf) : session->request.content_len));

session->request.content_len -= res;

// TODO add timeout - can be deadlock

// temporary solution

if (res == 0)

break;

}

Best Regards,

Paul

Re: Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Mon, 12 Aug 2013 20:34:14 GMT

Hi Paul,

Yes, it sounds like I'm doing the same thing as you are, calling httpd_read() in a loop until bytes read is equal to session->request.content_len. I had thought the new httpd server was providing the correct value in session->request.content_len but I suppose it could be off by a few hundred bytes and that's why httpd_read() returns fewer bytes then I'm expecting. The value's I was seeing in session->request.content_len were about what I thought they should be (file size plus some for header info, 900+K ish) but maybe they were just a little high. Thanks for the idea, I'll let you know if I find the session->request.content_len value to be wrong (ie different from the MQX 3.8.1.1 value).

Best,

Tim

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Mon, 12 Aug 2013 22:04:02 GMT

Hi Paul,

To follow up, I've repeated my tests with the new httpd server to see if he value in session->request.content_len was correct or not and I find it to be very close to the value indicated in the old MQX 3.8.1.1. server. The binary file being transferred is 901,848 bytes and the original httpd server session->request.content_len value indicates 902,057 bytes. When you strip the 209 header bytes that precede the payload the size value is just right. The new server indicates 902,055 bytes (2 bytes less) but httpd_read() stops returning any data while session->request.content_len indicates that 741 bytes are still remaining.

I'm really hoping Martin or one of the other Freescale guys may have an idea on this issue because I'm finding the new server to be far better in just about every other area. Any ideas Martin?

Best,

Tim

Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Tue, 13 Aug 2013 05:37:58 GMT

Hi Tim, I will try file upload on MQX 4.0.2 version (public release very soon!) and will update the post with my result. Thanks for your feedback !

Re: Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Tue, 13 Aug 2013 08:03:16 GMT

Tim, just checked with MQX 4.0.2 on TWR-K70F120M with a 2.2 MBytes file and works fine. Below is the cgi I used (note it just writes incoming data into buffer[] without actually doing any processing on them):

static _mqx_int cgi_fileupload(HTTPSRV_CGI_REQ_STRUCT* param)
{    
    HTTPSRV_CGI_RES_STRUCT response;
    char buffer[1024];
    uint_32 len = 0;
    uint_32 cnt = 0;
    
    if (param->request_method != HTTPSRV_REQ_POST)
    {
        return(0);
    }
    
    len = param->content_length;
    if(param->content_length)
    { 
      while(param->content_length>0)
      {
        len = HTTPSRV_cgi_read(param->ses_handle, buffer, (len > sizeof(buffer)) ? sizeof(buffer) : len);    
        param->content_length -= len;
        cnt++;
      }
    }

    response.ses_handle = param->ses_handle;
    response.content_type = HTTPSRV_CONTENT_TYPE_PLAIN;
    response.status_code = 200;
    response.data_length = param->content_length;
    response.content_length = response.data_length;
    /* Send response */
    HTTPSRV_cgi_write(&response);
    return (response.content_length);
}

This returns with 0, cnt being number of 1 KB chunks read. Chrome also shows file upload progress from 0 to 100 % as the process runs.

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Tue, 13 Aug 2013 17:19:12 GMT

Hi Martin,

Thanks for taking the time to look at this. I'm doing almost exactly the same thing as your included code is doing. I'm not able to use the code you included since MQX 3.8.1.1 (with the new httpd server installed) doesn't know about the structs your using or the HTTPSRV_cgi_read() function you call. Any chance the HTTPSRV_cgi_read() function is making the difference? Should I be able to use those structs and that function?

I've pasted in a clip of the cgi code that I use to read the data into a temp buffer where I can work with it (note I'm calling httpd_read() ). It looks really similar to what you posted. Do you see any problems? This code works properly with the httpd server included with MQX 3.8.1.1 but is unable to retrieve the last 740 or so bytes using the newer httpd server.

By the way, How do you get that nicely formatted code into your posts?

Best Regards,

Tim Hutchinson

static uint_32 CGI_IMAGE_RecvImage2(HTTPD_SESSION_STRUCT *session)
     {
     uint_32 lReadLength, lTotalMsgBytes, lBytesRead;
     char_ptr sRecvBuffer = NULL;
     char_ptr sBufferPtr = NULL;

     lTotalMsgBytes = session->request.content_len;
     if(lTotalMsgBytes == 0)
     return(0);

//Allocate temp buffer

//----------------------------------------------------------------------------

     sRecvBuffer = _mem_alloc(lTotalMsgBytes);
     if(sRecvBuffer == NULL)
          return(0);

//Read in the data until all bytes (per lTotalMsgBytesvalue) are received
//----------------------------------------------------------------------------
     sBufferPtr = sRecvBuffer;          //Use sBufferPtr as a movable temp pointer so sRecvBuffer can always point to buffer start
     lReadLength = httpd_read(session, sBufferPtr, lTotalMsgBytes);     //Get any waiting bytes
     lBytesRead = lReadLength;                                                               //Record how many bytes we extracted from the incoming stream
     session->request.content_len -= lReadLength;                                 //
     while(session->request.content_len > 0)                                          //Keep reading the incoming stream until all data bytes spec'd
          {                                                                                                    //in session->request.content_len have been received
          sBufferPtr += lReadLength;                                                          //Advance buffer pointer to location where next bytes are to be stored

lReadLength = httpd_read(session, sBufferPtr, (lTotalMsgBytes - lBytesRead)); //Retrieve waiting bytes from incoming stream
session->request.content_len -= lReadLength;

lBytesRead += lReadLength;
}

// More code here to process the received data stored in the sRecvBuffer. . . . .

return(0);
}

Re: Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Wed, 14 Aug 2013 07:27:08 GMT

The httpsrv version you use is a preliminary one, and it seems to have the issue. It was intended to get some initial feedback from MQX users. I don't see anything obviously wrong on your code. The preliminary version also tries to keep the API compatible with the httpd server.

During development, we decided to change the API, too. The latest httpsrv has the API shown in my code snippet. We will provide you with document that describes how to migrate httpd applications to httpsrv. The document and the httpsrv release version code will be with MQX 4.0.2.

For syntax highlighting, I click reply, then "Use advanced editor" in upper right corner. In the advanced editor, there is ">>" sign on the toolbar, which gives you the option to do C++ syntax highlighting on a selected text block.

-Martin

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Wed, 14 Aug 2013 16:54:53 GMT

Thanks for that Martin, I look forward to migrating to the new httpsrv as soon as your able to get me that documentation on how to do it. Also wanted to make sure that you are aware that I'm using MQX on a Freescale MPC5125 processor which means that I'm not able to upgrade MQX beyond revision 3.8.1.1 right? I find the embedded httpd server very useful so I'm sure hoping that I will be able to use the new httpd server with MQX 3.8.1.1.

Best,

Tim

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Tue, 27 Aug 2013 18:52:09 GMT

Hi Martin,

I see that MQX 4.0.2 is released and am wondering if you are able to make the new httpd server(maybe you call it httpsrv now?) available to us using MQX 3.8.1.1 yet? If yes, how do I get the files (to update my MQX) and the document describing how to update my MQX 3.8.1.1 installation with those new files and use the new httpserr ? Thanks!

Best regards,

Tim

Re: Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Thu, 29 Aug 2013 09:05:51 GMT

Hi Tim,

I started with adding only few files and #defines from MQX 4.0.2, that relate to httpsrv (with the intention to make a step by step guide for you and other interested users). I was sucesfull, the httpsrv_demo code loaded properly and seemed to work fine. However, I found that web pages with more source code, like "tcpstat.html" for example, were not downloaded completely to the client. When I opened source code of the .html web page from the browser, it showed only about half of the web page source code.

This sounds very similar to your original problem - transferring of some large data - with the difference that this one was download and your issue upload.

So, I decided to update complete RTCS code. I take MQX 4.0.2 RTCS source code as is and integrate it into MQX 3.8.1, it is not that difficult as one might think. In general, my steps were - replace the RTCS source code completely, resolve compile issues, then try to build and link application httpsrv_demo, resolve all linker issues (undefined functions). Finally I got httpsrv_demo application working with MQX 3.8.1 and MQX 4.0.2 RTCS.

There were much more changes in the RTCS besides the HTTP server code. My recommendation for you is the same - update MQX 3.8.1 RTCS with MQX 4.0.2 RTCS.

The list of modifies/new files, in summary:

rtcs/source/*

rtcs/examples/httpsrv/*

mqx/source/fio/io_dopr.c

mqx/source/fio/io_fseek.c

mqx/source/include/fio.h

mqx/source/include/fio_prv.h

mqx/source/io/enet/enet.h

mqx/source/io/enet/enutil.c

mqx/source/io/tfs/tfs.c

mqx/source/io/tfs/tfs.h

rtcs/build/bat/rtcs.bat

shell/source/rtcs/sh_ping.c

Note there are some new files so they need to be added to the RTCS library build project in CodeWarrior.

I put all relevant files in the separate .zip file, while keeping the MQX 3.8.1 directory structure. I suggest you to make a backup of MQX 3.8.1 folder and then unzip the files from the attached archive to the MQX 3.8.1 root directory. Then try to build libraries, then httpsrv demo application project. It works very fine on my TWR-MPC5125.

Please note the attached files are for TWR-MPC5125 only.

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Thu, 29 Aug 2013 16:29:57 GMT

Thanks for your work Martin, I appreciate it. I will try out your suggestions and let you know how it works out. Have a great day.

Best,

Tim

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Thu, 29 Aug 2013 16:36:10 GMT

Hey Martin,

I think there's a problem with the zip attachment. I've tried downloading it several times and WinZip reports that the file "Doesn't Appear To Be A Valid Archive". Could you post it again? Thanks.

~Tim

Re: Re: MQX 3.8.1 httpd session/server deadlock ?

Martin_ — Fri, 30 Aug 2013 05:42:54 GMT

Here it is.

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Fri, 30 Aug 2013 16:55:27 GMT

That one came through ok. Thanks!

~Tim

Re: MQX 3.8.1 httpd session/server deadlock ?

Tim562 — Mon, 09 Sep 2013 20:59:34 GMT

Hey martin,

I was able to import the RTCS stuff you provided from MQX 4.0.2 into MQX 3.8.1 and build the libraries and run the httpd example project. All looks good so far. I have a question about the rtc.html page in that httpd example project. Anytime I try to select that page (from index.html) I get a demand for a username and password. I'm unable to figure out where this request is coming from or what the proper credentials would be? If I rename the rtc.html file something else (clock.html for example) no password demand is made. Any other html file that I name rtc.html and attempt to load (from index.html) presents me with that uname/pword demand. I don't see any code in cgi.c or in the html pages that would cause a pword request, can you tell me how this is done? So far the new httpd server looks great. The next step will be porting my existing project over to it. Thanks again for your assistance, I appreciate it.

~Tim