MQX 3.8.1 httpd session/server deadlock ?

[pbanta]

I'm trying to track down a report of "web server dies".  I'm using MQX 3.8.1.  I was able to create a condition today where there are two http session tasks that are blocked and where the http server task is also blocked.

Some background

I use Basic Authentication in this server.  When a browser makes it's first connection to the server the user is prompted for username and password.  I have tested the authentication with IE9, IE10, Safari, Firefox, Chrome & Opera.  It works, but IE10 seems to only work once.  I've been running one of my servers for a few days and using Chrome to access it.  Works every time: I start Chrome, login, browse, close Chrome.

Problem

Today I was logged in with Chrome.  I started IE10.  I was prompted for username/password and was able to browse the server with no problem.  I closed IE10.  I started IE10 again and was prompted for username/password.  I entered the credentials and the server did not send the page.

I'm connected to the board with a PEMicro JTAG unit so I can poke around.  I found the following conditions (I've removed some of the irrelevant stack trace, but left the networking tasks on my development system).  There are two http session tasks that are blocked in the same location on a message send.  The http server task is blocked because maximum simultaneous sessions is 2.

    ARM Processors, guardian-lce.afx (Suspended)   
        Thread [ID: 0x10002] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            5 DummyFn1() dispatch.s:264 0x1000044e   
            4 _msgq_receive_internal() ms_recvi.c:220 0x10021c72   
            3 _msgq_receive() ms_recv.c:95 0x10021dd2   
            2 main_task() main.c:457 0x10000a08   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x10009] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            8 DummyFn1() dispatch.s:172 0x10000446   
            7 _time_delay_internal() ti_deli.c:103 0x1001ed6e   
            6 _time_delay_for() ti_delfo.c:80 0x1001ede2   
            5 _msgq_receive_internal() ms_recvi.c:222 0x10021c7e   
            4 _msgq_receive() ms_recv.c:92 0x10021dc2   
            3 TCPIP_task() tcpip.c:178 0x1002baf4   
            2 RTCS_task() rtcstask.c:70 0x1002a826   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x1000a] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            5 DummyFn1() dispatch.s:172 0x10000446   
            4 _lwsem_wait() lws_wait.c:99 0x1001fe40   
            3 httpd_server_task() httpd_task.c:178 0x100268f4   
            2 RTCS_task() rtcstask.c:70 0x1002a826   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x1000b] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            8 DummyFn1() dispatch.s:264 0x1000044e   
            7 _msgq_send_internal() ms_sendi.c:201 0x100217e6   
            6 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2   
            5 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c   
            4 SOCK_STREAM_accept() sstream.c:405 0x10027d20   
            3 FTPd_task() ftpd.c:177 0x10026482   
            2 RTCS_task() rtcstask.c:70 0x1002a826   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x1000c] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            16 DummyFn1() dispatch.s:264 0x1000044e   
            15 _msgq_send_internal() ms_sendi.c:201 0x100217e6   
            14 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2   
            13 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c   
            12 SOCK_STREAM_recv() sstream.c:571 0x10027f16   
            11 _io_socket_read() sockio.c:177 0x10024872   
            10 _io_read() io_read.c:91 0x1001c524   
            9 _io_telnet_read() telnetio.c:234 0x10023e48   
            8 _io_fgetc() io_fgetc.c:85 0x1001d102   
            7 _io_fgetline() io_fgetl.c:87 0x1001d068   
            6 _io_fgets() io_fgets.c:88 0x1001d02e   
            5 Shell() shell.c:142 0x100346d0   
            4 TELNETSRV_child() telnsrv.c:389 0x10023cc6   
            3 TELNETSRV_task() telnsrv.c:306 0x10023c26   
            2 RTCS_task() rtcstask.c:70 0x1002a826   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x1000d] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            11 DummyFn1() dispatch.s:264 0x1000044e   
            10 _msgq_send_internal() ms_sendi.c:201 0x100217e6   
            9 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2   
            8 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c   
            7 SOCK_STREAM_recv() sstream.c:571 0x10027f16   
            6 httpd_readln() httpd_supp.c:243 0x10026c7c   
            5 httpd_readreq() httpd.c:312 0x100276ae   
            4 httpd_ses_process() httpd.c:609 0x10027972   
            3 httpd_session_dynamic_task() httpd_task.c:134 0x1002687e   
            2 RTCS_task() rtcstask.c:70 0x1002a826   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x1000e] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)   
            11 DummyFn1() dispatch.s:264 0x1000044e   
            10 _msgq_send_internal() ms_sendi.c:201 0x100217e6   
            9 _msgq_send_blocked_internal() ms_sendb.c:70 0x100219a2   
            8 RTCS_cmd_issue() rtcscmd.c:191 0x10028c2c   
            7 SOCK_STREAM_recv() sstream.c:571 0x10027f16   
            6 httpd_readln() httpd_supp.c:243 0x10026c7c   
            5 httpd_readreq() httpd.c:312 0x100276ae   
            4 httpd_ses_process() httpd.c:609 0x10027972   
            3 httpd_session_dynamic_task() httpd_task.c:134 0x1002687e   
            2 RTCS_task() rtcstask.c:70 0x1002a826   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   
        Thread [ID: 0x10001] (Suspended: Signal 'Halt' received. Description: User halted thread.)   
            2 _mqx_idle_task() idletask.c:65 0x100204fe   
            1 _task_exit_function_internal() ta_exit.c:60 0x1001f490   

I've got the memory to increase the maximum session count to something greater than 2,  but I don't think that addresses the root cause.  Something appears not to have cleaned up correctly.  I'm wondering if I have a problem with handling sessions properly in all the CGI functions.

Any ideas?

Thanks.

[Tim562]

Hi Paul,

     Yes, it sounds like I'm doing the same thing as you are, calling httpd_read() in a loop until bytes read is equal to session->request.content_len. I had thought the new httpd server was providing the correct value in session->request.content_len but I suppose it could be off by a few hundred bytes and that's why httpd_read() returns fewer bytes then I'm expecting. The value's I was seeing in session->request.content_len were about what I thought they should be (file size plus some for header info, 900+K ish) but maybe they were just a little high. Thanks for the idea, I'll let you know if I find the session->request.content_len value to be wrong (ie different from the MQX 3.8.1.1 value).

Best,

Tim

[Martin_]

Yes, it can be used. I tested on MQX 3.8.1 and TWR-MPC5125. There are little differences compared to the MQX 4.0 post:

in rtcs25x.h

change from

#include <partition.h>

change to

#include <part.h>

in rtcs.h add:

#define AF_INET 1

#define AF_INET6         28         /* IPv6 */
#define PF_INET6         AF_INET6   /* Same as the address family. */

in user_config.h

#define sockaddr   sockaddr_in

with attached httpsrv.zip (application); add shell.a library into the application build project.

then, builds and was running on target, but I was getting unexpected exception due to low stack size. so I increased stack size for _RTCSTASK_stacksize = 6500; and now the new web server runs on MQX 3.8.1 (tested on TWR-MPC5125).