HTTP Expires header not respected by browser? - http

I have a situation where my (embedded) web server is sending Expires header, but the browser does not seem to respect the header setting, i.e., if I refresh the page, the browser requests the resources that are supposed to be cached. Following are the headers that are getting exchanged:
https://192.168.1.180/scgi-bin/ajax/ajax.cgi
GET /scgi-bin/ajax/ajax.cgi HTTP/1.1
Host: 192.168.1.180
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cache-Control: max-age=0
HTTP/1.x 200 OK
Date: Wed, 24 Jun 2009 20:26:47 GMT
Server: Embedded HTTP Server.
Connection: close
Content-Type: text/html
----------------------------------------------------------
https://192.168.1.180/scgi-bin/ajax/static.cgi?fn=images/logo.jpg&ts=20090624201057
GET /scgi-bin/ajax/static.cgi?fn=images/logo.jpg&ts=20090624201057 HTTP/1.1
Host: 192.168.1.180
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://192.168.1.180/scgi-bin/ajax/ajax.cgi
Cache-Control: max-age=0
HTTP/1.x 200 OK
Date: Wed, 24 Jun 2009 20:26:47 GMT
Server: Embedded HTTP Server.
Connection: close
Expires: Wed, 1 Jun 2011 20:00:00 GMT
Content-Type: image/jpg
----------------------------------------------------------
The ajax.cgi returns an html page with a logo graphic (via the static.cgi script), which I'd like cached, but the browser is asking for the logo on every refresh.

The browser ignores the Expires header if you refresh the page. It always checks whether the cache entry is still valid by contacting the web server. Ideally, it will use the If-Modified-Since request header so that the server can return '304 Not modified' if the cache entry is still valid.
You're not setting the Last-Modified header, so the browser has to perform an unconditional GET of the content to ensure that it is up to date.
Some rules of thumb for setting Expires and Last-Modified are described in this blog post:
http://blog.httpwatch.com/2007/12/10/two-simple-rules-for-http-caching/

What are you doing in your browser? I looks like you click the reload button or even something like shift+Reload. Normally, the browser wouldn't send a Cache-Control: max-age=0 header. That means the browser has thrown away the cached image and wants to get it again.
If you just navigate to another page and then back again, the browser should respect your Expires header.
Additionally, you could add a Cache-control: public header to your response. That allows proxies and the browser explicitly to cache the image.

Any errors in your https certificate will cause the browser to not respect your headers.
Try it without https and see if it works over plain http.
See this answer https://stackoverflow.com/a/17716911

The CGI script looks like it has a timestamp parameter...this isn't changing, is it? The browser should be treating each unique URL as a different object in the cache, so if that is updating with every request, it won't match with the cached image.
Additionally, the Expires field is not exactly in RFC 1123 format, because you need two digits for the date. This may or may not be an issue, but it's something to check. The browser is including Cache-Control: max-age=0, which indicates that it believes its cache to be potentially out of date.
Once the server gets this validation request, it can return 304 (Not Modified), or 200 (OK), as it is doing currently.

Related

If-None-Match headers ignore Content-Type and Vary

I have a web application that serves both HTML and multiple RDF formats (in the example below, it's RDF/XML).
A page loads as HTML (naturally), and then requests its own URL as RDF/XML.
The problem: it looks like Firefox 74.0 (64-bit) (on Windows) is mixing up ETag values from those two requests, ignoring different Content-Types as well as Vary: Accept being present.
When I reload the page, I can see it uses the ETag: "95e11fbc9e816b56" from the second (RDF/XML) response in the request for HTML, and vice versa:
Request URL: https://localhost:4443/6a6283d2-2a40-4882-b89d-8073a7c30e17/
Host: localhost:4443
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://localhost:4443/6a6283d2-2a40-4882-b89d-8073a7c30e17/
Connection: keep-alive
Cookie: _ga=GA1.1.828629977.1584086266; LinkedDataHub.first-time-message=true
Upgrade-Insecure-Requests: 1
If-None-Match: "95e11fbc9e816b56"
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000;includeSubDomains
ETag: "95e11fbc139f56de"
Cache-Control: max-age=3600, public
Last-Modified: Wed, 12 Feb 2020 23:05:15 GMT
Vary: Accept-Charset,Accept,Accept-Encoding
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Date: Sun, 22 Mar 2020 10:13:43 GMT
Request URL: https://localhost:4443/6a6283d2-2a40-4882-b89d-8073a7c30e17/
Host: localhost:4443
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: application/rdf+xml
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://localhost:4443/d376ee88-ff7d-48ee-81c4-1220c9f482f0/
Connection: keep-alive
Cookie: _ga=GA1.1.828629977.1584086266; LinkedDataHub.first-time-message=true
If-None-Match: "95e11fbc139f56de"
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000;includeSubDomains
ETag: "95e11fbc9e816b56"
Last-Modified: Wed, 12 Feb 2020 23:05:15 GMT
Vary: Accept-Charset,Accept
Content-Type: application/rdf+xml;charset=UTF-8
Transfer-Encoding: chunked
Date: Sun, 22 Mar 2020 10:13:55 GMT
On Chrome, I cannot get it to send If-None-Match headers at all, but this is probably due to the self-signed certificate.
Note that the ETag values are similar, but different: "95e11fbc139f56de" vs. "95e11fbc9e816b56".
This doesn't make any sense to me. Any explanations? Thanks.
The relevant specification is Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests.
The problem, essentially, is that you're relying on behavior that isn't mandated by the HTTP standard, and doesn't happen to be implemented by browsers.
For your scheme to work, browsers would have to store multiple representations of a single resource in their cache. Unfortunately, as discussed in articles like these, they don't do that.
Browsers typically do not implement the capability to store multiple variations per URL. The rationale for this is that the things we typically use Vary for (mainly Accept-Encoding and Accept-Language) do not change frequently within the context of a single user.
So the issue isn't the ETags, it's that the browser is just overwriting the single representation in its cache each time it gets a different representation.
If the browser did store multiple representations, the scheme should work fine. In that case, note that it would be the server, not the client, that selects between multiple ETags. The client would send an If-None-Match header with all the ETags it knows about, and it would be up to the server to decide which one, if any, matched the requested representation.
According to the article above, edge servers (as opposed to browsers) do keep multiple representations in the cache for each resource, so it's still possible that your scheme could generate performance gains.

Cross-domain chunked uploads using CORS

I have user-submitted files that I'm trying to upload in 10 MB chunks. I'm currently using raw XMLHttpRequest (and XDomainRequest) to push each individual slice (File.prototoype.slice) on the front end. The back end is Nginx using the upload module.
Just for reference, here's the synopsis of how I'm using slice:
element.files[0].slice(...)
I understand the cross-browser prefixed methods webkitSlice and mozSlice and all that.
The problem I have is with actually making the cross-domain request. I'm uploading from server.local to upload.server.local. In Firefox, the options request goes through fine and then the actual post fails. In Chrome and Opera, the options request fails with
OPTIONS https://URL Resource failed to load
Here are the headers from Firefox:
Request Headers
OPTIONS /path/to/asset HTTP/1.1
Host: upload.server.local:8443
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Origin: https://server.local:8443
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-disposition,content-type,x-content-range,x-session-id
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Response Headers
HTTP/1.1 204 No Content
Server: nginx/1.2.6
Date: Wed, 13 Feb 2013 03:27:44 GMT
Connection: keep-alive
access-control-allow-origin: https://server.local:8443
Access-Control-Allow-Methods: POST, OPTIONS
Access-Control-Allow-Headers: x-content-range, origin, content-disposition, x-session-id, content-type, cache-control, pragma, referrer, host
access-control-allow-credentials: true
Access-Control-Max-Age: 10000
The actual post request never leaves the browser. Nginx access logs never see the post. The browser halts it for some reason. How do I unravel why this post is being blocked?
Chromium 24
Firefox 18
Opera 12.14
I've verified all browsers support CORS properly here.
By pointing my uploads to https://cors-test.appspot.com/test, I have confirmed that the problem is definitely with the server-side headers.
The POST won't leave the browser if the preflight check does not return sufficient permissions and thus the POST request is not fully authorized. The request/response included in the question does look sufficient to me.
Are you sure you are setting withCredentials = true in your XMLHttpRequest?
Are you sure that you have valid (not self-signed) SSL certificates on your servers? The HTTPS might fail the CORS check even if you have added an exception for browsing the site with an invalid certificate.
Have you tried emptying your cache? You have Access-Control-Max-Age: 10000 set in your response headers. That's close to 3 hours. I know you've been working on this longer than that but while testing especially, set that header to zero instead so you don't go crazy with browser caching of old access permissions.
In general I'd start with going as permissive as possible with the CORS headers and slowly ratcheting up the the security to see where it fails. However, this is not completely straightforward. For example, according to the MDN documentation on CORS,
When responding to a credentialed request, server must specify a domain, and cannot use wild carding. The above example would fail if the header was wildcarded as: Access-Control-Allow-Origin: *
When I send the request part of your question to https://cors-test.appspot.com/test, I get back the following:
HTTP/1.1 200 OK
Cache-Control: no-cache
Access-Control-Allow-Origin: https://server.local:8443
Access-Control-Allow-Headers: content-disposition,content-type,x-content-range,x-session-id
Access-Control-Allow-Methods: POST
Access-Control-Max-Age: 0
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Type: application/json
Content-Encoding: gzip
Content-Length: 35
Vary: Accept-Encoding
Date: Thu, 23 May 2013 06:37:34 GMT
Server: Google Frontend
So you can start from there and add more and more security until it breaks to figure out what is the culprit.

Why could a browser not caching a resource with Cache-Control HTTP headers?

I've read all the caching documentation for HTTP, and this are the headers for my file:
Edit: I am now using amazon S3 to see if it helps and still the same problem! Here go the request and the response> What am I missing? Even thought the response asks for the browser to cache the file, it doesn't! I've checked the Google Chrome cache and the file is not there.
Request URL:https://s3-sa-east-1.amazonaws.com/combo-combat-release/WebPlayer.unity3d
Request Method:GET
Status Code:200 OK
**Request Headers**
GET /combo-combat-release/WebPlayer.unity3d HTTP/1.1
Host: s3-sa-east-1.amazonaws.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: es,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
**Response Headers**
HTTP/1.1 200 OK
x-amz-id-2: xu/RKgR4Xt6G3jQ1qkdO5rzd4TlngDYYERZ+bbGlalitJFMDjAgiLAoP4GXhmkeo
x-amz-request-id: 6A9389D16407D28D
Date: Wed, 16 Jan 2013 22:42:26 GMT
Cache-Control: max-age=2592000, public, must-revalidate
Last-Modified: Wed, 16 Jan 2013 22:29:55 GMT
ETag: "0f8d22257da9fcae61f21fd30b7a1fd4"
Accept-Ranges: bytes
Content-Type: application/octet-stream
Content-Length: 39222189
Server: AmazonS3
But the browser doesn't cache the file. Each time, the request is sent and all the data transferred from scratch (200 OK instead of 304). :(
File size is 37.4 MB
Any tip on what could be wrong?
Thanks,
Manuel
Try HTTP 1.1 instead of 1.0.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html states:
Note that HTTP/1.0 caches might not implement Cache-Control and
might only implement Pragma: no-cache

LiveHttpHeaders: which cache-control info is right

Using LiveHttpHeaders for Firefox 6 I was trying to see if my css, JS files being cached using Headers Module from Apache using htaccess. But I confuse, there are 2 values from the 'Cache-Control' data:
GET /proz/css/global.css HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0
Accept: text/css,*/*;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://localhost/proz/
Cookie: PHPSESSID=el34de37pe3bnp4rdtbst1kd43
If-Modified-Since: Fri, 16 Sep 2011 21:15:32 GMT
If-None-Match: "400000000b06a-2999-4ad157e5b4583"
Cache-Control: max-age=0
HTTP/1.1 304 Not Modified
Date: Sat, 17 Sep 2011 03:04:50 GMT
Server: Apache/2.2.17 (Win32) PHP/5.2.8
Connection: Keep-Alive
Keep-Alive: timeout=5, max=99
Etag: "400000000b06a-2999-4ad157e5b4583"
Cache-Control: max-age=604800, public
Vary: Accept-Encoding
Which one is the true data, the first Cache-Control data (max-age=0) or the latter one.
I also would like to know how do I make sure that my JS, CSS, HTML files being compress after I use deflate module in htaccess. And yes, both headers and deflate modules are turn on.
There are two parts in this listing:
The part before the blank line is the request, sent by your browser
The part after the blank line is the response, sent by the server
The Cache-Control: max-age=0 sent by the client (your browser) tells the server (or any proxy in the middle) to send the most fresh version of the file. The browser usually sends this when you hit the refresh button.
The Cache-Control: max-age=604800, public sent by the server tells the client (your browser or a proxy) that the file is valid for 604800 seconds and can be cached for that time. (The browser won't even attemps to ask the server if a newer version exists, unless you hit refresh, as you did in this case.)
The server replied 304 Not Modified, this means that your browser already has the most recent version and it doesn't need to download it again (it did not downloaded it again).
The Vary: Accept-Encoding header indicate that the server taken some decisions based on the client's Accent-Encoding header. This may indicate that, if the server didn't replied 304 Not Modified, it would have compressed the file.
To verify this last point, clear your cache, and request the file again, and look at the content of the Content-Encoding header (must be gzip or deflate if the data is compressed).

Firefox & IE: Corrupted data when retrieved from cache

I have a request to the server which looks like this:
http://localhost/Combine.aspx/flag/file1.js,file2.js,file3.js
I have also tried this:
http://localhost/Combine.aspx/flag/value/file1.js/file2.js/file3.js
It simply combines the three files which it was given: this works fine.
Upon the first request from Firefox, the page displays correctly. When the page is refreshed again, the content looks like this:
��������`I�%&/m�{J�J��t��`$�����iG#)�*��eVe]f#�흼��{���{��;�N'
In that case, no request is made to the server, the file is only retrieved from the firefox cache, as far as I can tell.
If I force-refresh the page with Control-F5, the page looks normal again.
Here is the request and response (in firebug) with Control-F5 (works):
Response
Cache-Control public
Content-Type application/javascript; charset=utf-8
Content-Encoding gzip
Expires Sun, 26 Feb 2012 02:57:26 GMT
Last-Modified Fri, 07 Jan 2011 21:08:54 GMT
Etag CLYvZwbFmainWEMi8h-6x-zNEIQ1
Vary Accept-Encoding
Server Microsoft-IIS/7.5
Set-Cookie auth=longstring; path=/
X-AspNet-Version 2.0.50727
X-Powered-By ASP.NET
Date Thu, 03 Mar 2011 02:57:26 GMT
Content-Length 918
Request
Host localhost
User-Agent Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.14) Gecko/20110218 Firefox/3.6.14
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Cookie auth=longstring
Pragma no-cache
Cache-Control no-cache
Here is the same with just F5 (doesn't work):
Response:
Cache-Control public
Content-Type application/javascript; charset=utf-8
Content-Encoding gzip
Expires Sun, 26 Feb 2012 02:57:26 GMT
Last-Modified Fri, 07 Jan 2011 21:08:54 GMT
Etag CLYvZwbFmainWEMi8h-6x-zNEIQ1
Vary Accept-Encoding
Server Microsoft-IIS/7.5
X-AspNet-Version 2.0.50727
X-Powered-By ASP.NET
Date Thu, 03 Mar 2011 03:01:31 GMT
Content-Length 1533
Request:
Host localhost
User-Agent Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.14) Gecko/20110218 Firefox/3.6.14
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Cookie auth=longstring
The only thing that I can tell is different is the Content-Length (which is likely causing the corruption), but since it never hits the server, I have no control over it.
What could be causing this?
Edit:
This happens on IE8 as well, but instead of viewing a corrupt file, it downloads the corrupt file. No request is ever made to the server (set a breakpoint in visual studio).
As for chrome, when viewing the url directly, a request to the server is made every time, and it works every time. I believe, though, that if it were included in a tag, it would behave similar to Firefox.
Oh, and also:
If I turn off the Expires header (which is one of the flags in the URL), everything works fine.
Edit 2:
I was able to work around this by not sending both an Expires and ETag header, only one or the other. I have no clue why an ETag header would cause such corruption, though.
It is because of your IIS configuration. Adjust IIS as ASP 4.0 Classic, not Default Application Pool

Resources