httpsrv gzip compression

[seandema]

I'm running MQX 4.1.0.1 on a Kinetis K60 and attempting to use compressed files stored in TFS running on the built in RTCS web server. However I'm having some trouble getting it to work with compressed files. My page loads fine if I decompress the files leading me to believe the webpage code is ok. But if I have files compressed (in .gz or .zip) the web page does not display correctly.

I have 4 files, 1 .html, 2 .js and 1 .css file. I have gzip compressed the .css and .js files. So they now have a .js.gz or .css.gz style file name. I realize .gz is not supported by default but using the supported .zip does not work either. The page loads just as poorly with both.

Following the instructions listed here Re: Re: How to set Content Disposition? I have added support for .gz files. I'm guess I did this ok since it at least works the same as the included .zip support.

typedef enum httpsrv_content_type
{
    HTTPSRV_CONTENT_TYPE_OCTETSTREAM = 1,
    HTTPSRV_CONTENT_TYPE_PLAIN,
    HTTPSRV_CONTENT_TYPE_HTML,
    HTTPSRV_CONTENT_TYPE_CSS,
    HTTPSRV_CONTENT_TYPE_GIF,
    HTTPSRV_CONTENT_TYPE_JPG,
    HTTPSRV_CONTENT_TYPE_PNG,
    HTTPSRV_CONTENT_TYPE_JS,
    HTTPSRV_CONTENT_TYPE_ZIP,
    HTTPSRV_CONTENT_TYPE_PDF,
    HTTPSRV_CONTENT_TYPE_GZIP,
} HTTPSRV_CONTENT_TYPE;

static const httpsrv_table_row content_type[] = {
        { HTTPSRV_CONTENT_TYPE_PLAIN,       "text/plain" },
        { HTTPSRV_CONTENT_TYPE_HTML,        "text/html" },
        { HTTPSRV_CONTENT_TYPE_CSS,         "text/css" },
        { HTTPSRV_CONTENT_TYPE_GIF,         "image/gif" },
        { HTTPSRV_CONTENT_TYPE_JPG,         "image/jpeg" },
        { HTTPSRV_CONTENT_TYPE_PNG,         "image/png" },
        { HTTPSRV_CONTENT_TYPE_JS,          "application/javascript" },
        { HTTPSRV_CONTENT_TYPE_ZIP,         "application/zip" },
        { HTTPSRV_CONTENT_TYPE_GZIP,        "application/x-gzip" },
        { HTTPSRV_CONTENT_TYPE_PDF,         "application/pdf" },
        { HTTPSRV_CONTENT_TYPE_OCTETSTREAM, "application/octet-stream" },
        { 0,    0 }
};

static httpsrv_content_table_row content_tbl[] = {
    /* Size,          extension, MIME type,                        Cache? */
    {sizeof("js")-1 ,   "js",    HTTPSRV_CONTENT_TYPE_JS,          TRUE},
    {sizeof("gz")-1,    "gz",    HTTPSRV_CONTENT_TYPE_GZIP,        FALSE},
    {sizeof("css")-1,   "css",   HTTPSRV_CONTENT_TYPE_CSS,         TRUE},
    {sizeof("gif")-1,   "gif",   HTTPSRV_CONTENT_TYPE_GIF,         TRUE}, 
    {sizeof("htm")-1,   "htm",   HTTPSRV_CONTENT_TYPE_HTML,        TRUE},
    {sizeof("jpg")-1,   "jpg",   HTTPSRV_CONTENT_TYPE_JPG,         TRUE},
    {sizeof("pdf")-1,   "pdf",   HTTPSRV_CONTENT_TYPE_PDF,         FALSE}, 
    {sizeof("png")-1,   "png",   HTTPSRV_CONTENT_TYPE_PNG,         TRUE},
    {sizeof("txt")-1,   "txt",   HTTPSRV_CONTENT_TYPE_PLAIN,       FALSE},
    {sizeof("zip")-1,   "zip",   HTTPSRV_CONTENT_TYPE_ZIP,         FALSE},
    {sizeof("html")-1,  "html",  HTTPSRV_CONTENT_TYPE_HTML,        TRUE},
    {sizeof("shtm")-1,  "shtm",  HTTPSRV_CONTENT_TYPE_HTML,        FALSE},
    {sizeof("shtml")-1, "shtml", HTTPSRV_CONTENT_TYPE_HTML,        FALSE},
    /* Following row MUST have length set to zero so we have proper array termination */
    {0,                      "", HTTPSRV_CONTENT_TYPE_OCTETSTREAM, FALSE}
};

I'm not an expert on the inner working of RTCS or the web server but is there anything else I need to do to enable support for compression?

Thanks,

Sean

[seandema]

Found out a little more info. Digging around in the web server code it appeared that it was searching for files with a .js or .css extension and not the .gz that is should be. All of my .js and .css files are gzipped. Just to see what would happen I modified httpsrv_process_req_method() in httpsrv_supp.c to append .gz to the 3 files that need it.

if(strcmp(uri_begin, "/bootstrap.min.css") == 0 ||
   strcmp(uri_begin, "/bootstrap.min.js") == 0 ||
   strcmp(uri_begin, "/jquery.min.js") == 0)
{
    written = snprintf(session->request.path, server->params.max_uri, "%s.gz", uri_begin);
}
else
{
    written = snprintf(session->request.path, server->params.max_uri, "%s", uri_begin);
}

I then did a Wireshark capture and I'm now seeing the expected Content-type of "application/x-gzip" but my web page still displays incorrectly.

I'm still not sure what is wrong but thought this might be useful info to someone who knows more than I do about the web server.

Thanks,

Sean

[RadekS · Accepted solution]

Unfortunately compressed files are not supported in current MQX web server, but your modification looks OK and it should work however we didn’t test such modification till now.

Ideas:

I guess that HTTP header should contain in this case

Content-Encoding: gzip

for let client to know which decoding mechanisms has to be selected in order to obtain the media-type referenced by the Content-Type header field.

For details:

http://tools.ietf.org/html/rfc2616#section-14.11

Please try additionally modify code for generating header file and add there Content-Encoding entity-header field.

Your picture shows Wireshark capture with filter (not all data, it seems that bootstrap.min.css.gz has 30kB). Are you sure that files were fully transferred?

Note: You selected FALSE as option for Cache. I suppose that css and js files will not be changed dynamically. So, I guess that cache at client side has some sense for these files. Anyway, this should not have any influence on main result – if it will work or not.

I hope it helps you.

Have a great day,
RadekS

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[seandema]

"Content-Encoding: gzip" was the fix. I also had to have "Content-Type: application/javascript" or what ever is appropriate or IE would not load the page. Though Chrome and Firfox were fine without Content-Type.

I ended up modifying the webserver to always put in "Content-Encoding: gzip" for .js and .css files since all of mine are gzipped.

Thanks for the help!

Sean

[igorspectr]

Hi Sean

Can you please post the code you have modified to make gzip work?

I would really appreciate it if you can.

thanks in advance

[seandema]

Sorry for the slow reply, somehow I just now saw this.  Anyway here are the changes I made. To get things working I have my .js and .css file gzipped but I changed the file extension to just read .css and .js instead of .css.gz or .js.gz even though they really are still gz compressed. This is done on my PC before running mktfs.bat.

In httpsrv.h

typedef enum httpsrv_content_type
{
    HTTPSRV_CONTENT_TYPE_OCTETSTREAM = 1,
    HTTPSRV_CONTENT_TYPE_PLAIN,
    HTTPSRV_CONTENT_TYPE_HTML,
    HTTPSRV_CONTENT_TYPE_CSS,
    HTTPSRV_CONTENT_TYPE_GIF,
    HTTPSRV_CONTENT_TYPE_JPG,
    HTTPSRV_CONTENT_TYPE_PNG,
    HTTPSRV_CONTENT_TYPE_JS,
    HTTPSRV_CONTENT_TYPE_ZIP,
    HTTPSRV_CONTENT_TYPE_PDF,
    HTTPSRV_CONTENT_TYPE_GZIP, // "Encoding-Type: " for gzipped files
    HTTPSRV_CONTENT_TYPE_JSON  // "Content-Type: " for JSON CGI data
} HTTPSRV_CONTENT_TYPE;

In httpsrv_supp.c

static const httpsrv_table_row content_type[] = {
        { HTTPSRV_CONTENT_TYPE_PLAIN,       "text/plain" },
        { HTTPSRV_CONTENT_TYPE_HTML,        "text/html" },
        { HTTPSRV_CONTENT_TYPE_CSS,         "text/css" },
        { HTTPSRV_CONTENT_TYPE_GIF,         "image/gif" },
        { HTTPSRV_CONTENT_TYPE_JPG,         "image/jpeg" },
        { HTTPSRV_CONTENT_TYPE_PNG,         "image/png" },
        { HTTPSRV_CONTENT_TYPE_JS,          "application/javascript" },
        { HTTPSRV_CONTENT_TYPE_ZIP,         "application/zip" },
        { HTTPSRV_CONTENT_TYPE_GZIP,        "gzip" },  // this is the "Encoding-Type: " for gzipped filed
        { HTTPSRV_CONTENT_TYPE_JSON,        "application/json" }, // "Content-Type: " for JSON CGI data
        { HTTPSRV_CONTENT_TYPE_PDF,         "application/pdf" },
        { HTTPSRV_CONTENT_TYPE_OCTETSTREAM, "application/octet-stream" },
        { 0,    0 }
};

static httpsrv_content_table_row content_tbl[] = {
    /* Size,          extension, MIME type,                        Cache? */
    {sizeof("js")-1 ,   "js",    HTTPSRV_CONTENT_TYPE_JS,          TRUE},
    {sizeof("gz")-1,    "gz",    HTTPSRV_CONTENT_TYPE_GZIP,        TRUE},  // Adding gz, keeping order of length(shortest first)
    {sizeof("css")-1,   "css",   HTTPSRV_CONTENT_TYPE_CSS,         TRUE},
    {sizeof("gif")-1,   "gif",   HTTPSRV_CONTENT_TYPE_GIF,         TRUE}, 
    {sizeof("htm")-1,   "htm",   HTTPSRV_CONTENT_TYPE_HTML,        TRUE},
    {sizeof("jpg")-1,   "jpg",   HTTPSRV_CONTENT_TYPE_JPG,         TRUE},
    {sizeof("pdf")-1,   "pdf",   HTTPSRV_CONTENT_TYPE_PDF,         FALSE}, 
    {sizeof("png")-1,   "png",   HTTPSRV_CONTENT_TYPE_PNG,         TRUE},
    {sizeof("txt")-1,   "txt",   HTTPSRV_CONTENT_TYPE_PLAIN,       FALSE},
    {sizeof("zip")-1,   "zip",   HTTPSRV_CONTENT_TYPE_ZIP,         FALSE},
    {sizeof("html")-1,  "html",  HTTPSRV_CONTENT_TYPE_HTML,        TRUE},
    {sizeof("shtm")-1,  "shtm",  HTTPSRV_CONTENT_TYPE_HTML,        FALSE},
    {sizeof("json")-1,  "json",  HTTPSRV_CONTENT_TYPE_JSON,        FALSE},
    {sizeof("shtml")-1, "shtml", HTTPSRV_CONTENT_TYPE_HTML,        FALSE},
    /* Following row MUST have length set to zero so we have proper array termination */
    {0,                      "", HTTPSRV_CONTENT_TYPE_OCTETSTREAM, FALSE}
};

/* If there will be entity body send content type */
    if (has_entity)
    {
        httpsrv_print(session, "Content-Type: %s\r\n", httpsrv_get_table_str((httpsrv_table_row*)content_type, session->response.content_type));

        if(session->response.content_type == HTTPSRV_CONTENT_TYPE_GZIP ||
           session->response.content_type == HTTPSRV_CONTENT_TYPE_JS ||
           session->response.content_type == HTTPSRV_CONTENT_TYPE_CSS) // using "Content-Encoding: gzip" for .gz files, assume all .js and .css files are actually gzipped 
        {
            httpsrv_print(session, "Content-Encoding: %s\r\n", httpsrv_get_table_str((httpsrv_table_row*)content_type, HTTPSRV_CONTENT_TYPE_GZIP /*session->response.content_type*/));
        }
    }

Note this last section of code is where I make the server set "Content-Encoding: gzip" for all .js .css & .gz files. This was the only way I was able to get all browsers to reliably decompress the gz content.

It's a little bit of a hack changing the file extensions and then having the server set the content encoding to "gzip" but it was the only way I could get it to work and doing that was not a problem for my project so I just left it that way.

Hope that helps.

-Sean

[adyr]

Thanks for posting your solution Sean. I have done it slightly differently but your guide helped me a lot. As I wanted a mixture of compressed and uncompressed I created two new file extension .js_gz and .css_gz then after gzipping I just replace the last '.' with an '_' so my code is as follows:

typedef enum httpsrv_content_type

{

    HTTPSRV_CONTENT_TYPE_OCTETSTREAM = 1,

    HTTPSRV_CONTENT_TYPE_PLAIN,

    HTTPSRV_CONTENT_TYPE_HTML,

    HTTPSRV_CONTENT_TYPE_CSS,

    HTTPSRV_CONTENT_TYPE_GIF,

    HTTPSRV_CONTENT_TYPE_JPG,

    HTTPSRV_CONTENT_TYPE_PNG,

    HTTPSRV_CONTENT_TYPE_SVG,

    HTTPSRV_CONTENT_TYPE_JS,

    HTTPSRV_CONTENT_TYPE_ZIP,

    HTTPSRV_CONTENT_TYPE_XML,

    HTTPSRV_CONTENT_TYPE_PDF,

    HTTPSRV_CONTENT_TYPE_CSS_GZ,    //Adrian Rockall

    HTTPSRV_CONTENT_TYPE_JS_GZ,        //Adrian Rockall

} HTTPSRV_CONTENT_TYPE;

static const HTTPSRV_TABLE_ROW content_type[] = {

        { HTTPSRV_CONTENT_TYPE_PLAIN,      "text/plain" },

        { HTTPSRV_CONTENT_TYPE_HTML,        "text/html" },

        { HTTPSRV_CONTENT_TYPE_CSS,        "text/css" },

        { HTTPSRV_CONTENT_TYPE_GIF,        "image/gif" },

        { HTTPSRV_CONTENT_TYPE_JPG,        "image/jpeg" },

        { HTTPSRV_CONTENT_TYPE_PNG,        "image/png" },

        { HTTPSRV_CONTENT_TYPE_SVG,        "image/svg+xml"},

        { HTTPSRV_CONTENT_TYPE_JS,          "application/javascript" },

        { HTTPSRV_CONTENT_TYPE_XML,        "application/xml" },

        { HTTPSRV_CONTENT_TYPE_ZIP,        "application/zip" },

        { HTTPSRV_CONTENT_TYPE_PDF,        "application/pdf" },

        { HTTPSRV_CONTENT_TYPE_OCTETSTREAM, "application/octet-stream" },

        { HTTPSRV_CONTENT_TYPE_CSS_GZ,      "text/css" },                    //New css_gz file extension

        { HTTPSRV_CONTENT_TYPE_JS_GZ,      "application/javascript" },    //New js_gz file extension

        { 0,                                0}

};

static const HTTPSRV_CONTENT_TABLE_ROW content_tbl[] = {

    /* Size,          extension, MIME type,                        Cache? */

    {sizeof("js")-1 ,  "js",    HTTPSRV_CONTENT_TYPE_JS,          TRUE},

    {sizeof("css")-1,  "css",  HTTPSRV_CONTENT_TYPE_CSS,        TRUE},

    {sizeof("gif")-1,  "gif",  HTTPSRV_CONTENT_TYPE_GIF,        TRUE},

    {sizeof("htm")-1,  "htm",  HTTPSRV_CONTENT_TYPE_HTML,        TRUE},

    {sizeof("jpg")-1,  "jpg",  HTTPSRV_CONTENT_TYPE_JPG,        TRUE},

    {sizeof("pdf")-1,  "pdf",  HTTPSRV_CONTENT_TYPE_PDF,        FALSE},

    {sizeof("png")-1,  "png",  HTTPSRV_CONTENT_TYPE_PNG,        TRUE},

    {sizeof("svg")-1,  "svg",  HTTPSRV_CONTENT_TYPE_SVG,        TRUE},

    {sizeof("txt")-1,  "txt",  HTTPSRV_CONTENT_TYPE_PLAIN,      FALSE},

    {sizeof("xml")-1,  "xml",  HTTPSRV_CONTENT_TYPE_XML,        FALSE},

    {sizeof("zip")-1,  "zip",  HTTPSRV_CONTENT_TYPE_ZIP,        FALSE},

    {sizeof("html")-1,  "html",  HTTPSRV_CONTENT_TYPE_HTML,        TRUE},

    {sizeof("shtm")-1,  "shtm",  HTTPSRV_CONTENT_TYPE_HTML,        FALSE},

    {sizeof("shtml")-1, "shtml", HTTPSRV_CONTENT_TYPE_HTML,        FALSE},

    {sizeof("js_gz")-1 ,"js_gz", HTTPSRV_CONTENT_TYPE_JS_GZ,      TRUE},          //New js_gz file extension

    {sizeof("css_gz")-1,"css_gz",HTTPSRV_CONTENT_TYPE_CSS_GZ,      TRUE},    //New css_gz file extension

    /* Following row MUST have length set to zero so we have proper array termination */

    {0,                      "", HTTPSRV_CONTENT_TYPE_OCTETSTREAM, FALSE}

};

if (has_entity)

{

     httpsrv_print(session, "Content-Type: %s\r\n", httpsrv_get_table_str((HTTPSRV_TABLE_ROW*)content_type, session->response.content_type));

     //If we are serving a GZIP file then add the required header so the browser knows to decompress it

      if(session->response.content_type == HTTPSRV_CONTENT_TYPE_JS_GZ ||

        session->response.content_type == HTTPSRV_CONTENT_TYPE_CSS_GZ)

     {

           httpsrv_print(session, "Content-Encoding: gzip\r\n");

     }

}

I have one large JavaScript library that gzips from 400k down to 80k so serving the gzip version is much faster. My slowest page, that has two large .js files and two large .css files, was taking about 15 seconds to serve and now only takes about 3 seconds with the gzip in place so definitely worth the effort.

Adrian.

[igorspectr]

Thanks a lot for sharing the code Sean, i really appreciate it.