While optimizing the caching-behaviour of our website, I noticed that a whole lot of if-none-match-requests are sent to our site. As far as I understand caching, this should not be the case as long as the cache is still valid.
One particular request generates the following response-header:
HTTP/1.1 200 OK
Cache-Control: public, max-age=25920000
Transfer-Encoding: chunked
Content-Type: application/javascript; charset=utf-8
Content-Encoding: gzip
Expires: Thu, 04 Feb 2016 17:20:09 GMT
Last-Modified: Mon, 01 Jan 2001 23:00:00 GMT
ETag: W/"0"
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
Date: Fri, 10 Apr 2015 16:20:09 GMT
As you can see, the cache should be valid for 300 days. The way I understand it, the browser should use its cache directly during that period. Only after this period is over, it should issue a request with the header if-none-match.
But browsers seem to ignore that and send this if-none-match -request each and every time the page is loaded just to receive a 304-response ("Not Modified").
What do I need to change to keep browsers from sending these useless requests?
Yes, while the cache is fresh browsers should use a local copy without revalidation. However, this is not guaranteed. For example, when users use the Refresh button browsers make requests to the origin server anyway.
There is a Cache-Control: immutable, max-age=… extension that tells browsers you really really mean they should use the cached resource without contacting the server.
Related
I'm getting reports of sporadic instances where users in various browsers get a 302 page, but instead of automatically redirecting, the browser just displays the Object moved to here HTML sent by the browser. What could possibly cause this? If it were one browser on one machine, I'd blame a bad installation or something, but I've had a handful of reports, just in the past couple of days, from a number of machines and browsers, so I'm nervous that something is actually wrong with the HTTP Response, even though it couldn't be that wrong or the app wouldn't work almost everywhere else, as it does.
Anyway, here's the Response packet:
HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=utf-8
Location: /nextpage
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
Set-Cookie: cookiename={...data...}; path=/
Date: Fri, 10 Oct 2014 14:31:44 GMT
Content-Length: 124
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Am I missing anything here?
A grid of EC2 web servers is running behind an ELB load balancer. The ELB is behind Amazon's CloudFront content delivery network. Content Delivery Networks are very new to me. My understanding is that CloudFront is supposed to speed up performance by caching static content at its "edges". But this isn't what's happening.
Consider my EC2 instances whose content should always have a lifetime of five minutes. For static content this usually means declaring the following in my web.config file:
<staticContent>
<clientCache cacheControlCustom="public" cacheControlMode="UseMaxAge" cacheControlMaxAge="00.00:05:00"/>
</staticContent>
...and for the dynamic stuff, it usually means executing the following commands against an HttpResponse object:
resp.Cache.SetCacheability(HttpCacheability.Public);
resp.Cache.SetMaxAge(TimeSpan.FromMinutes(5));
With that as background...
When my browser hits the ELB directly, everything works as expected. Firebug consistently shows that 304 (Not Modified) is returned for content that exists in the browser's cache, has passed its five minute expiration, but has not been changed on the server. Here are the response headers for a download of defs.js, for example:
HTTP/1.1 304 Not Modified
Accept-Ranges: bytes
Cache-Control: public,max-age=300
Date: Tue, 22 Apr 2014 13:54:16 GMT
Etag: "0152435d158cf1:0"
Last-Modified: Tue, 15 Apr 2014 17:36:18 GMT
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Connection: keep-alive
IIS correctly sees that the file hasn't been changed since April 15th and returns 304.
But looks what happens when the file is grabbed through CloudFront.
HTTP/1.1 200 OK
Content-Type: application/x-javascript
Content-Length: 205
Connection: keep-alive
Accept-Ranges: bytes
Cache-Control: public,max-age=300
Date: Tue, 22 Apr 2014 14:07:33 GMT
Etag: "0152435d158cf1:0"
Last-Modified: Tue, 15 Apr 2014 17:36:18 GMT
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Age: 16
X-Cache: Hit from cloudfront
Via: 1.1 0f140ef1be762325ad24a7167aa57e65.cloudfront.net (CloudFront)
X-Amz-Cf-Id: Evfdhs-pxFojnzkQWuG-Ubp6B2TC5xbunhavG8ivXURdp2fw_noXjw==
In this case CloudFront forces the browser to download the entire file again even though, as you can see:
(a) it knows the file hasn't been modified since April 15th (see Last-Modified header), and
(b) CloudFront does have a cached copy of the file on hand (see X-Cache header)
Perhaps you're wondering if my browser is sending a valid If-Modified-Since header. Indeed it is. Here are the request headers:
GET /code/shared/defs.js HTTP/1.1
Host: d2fn6fv5a0cu3b.cloudfront.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://d2fn6fv5a0cu3b.cloudfront.net/
Connection: keep-alive
If-Modified-Since: Tue, 15 Apr 2014 17:36:18 GMT
If-None-Match: "0152435d158cf1:0"
Cache-Control: max-age=0
It's an odd situation. If I just sit in front of my browser and keep doing page Reloads (Cmd-R), maybe about half the time CloudFront will correctly return a 304 and the other half of the time it'll incorrectly return 200 along with all of the content. Waiting for the five minute expiration before interacting with the page yields primarily 200's and only a few 304's. This odd behavior applies to all of the files (.css, .js, .png, etc.) referenced on the HTML page as well as for the containing HTML page itself. I know my app is coded properly because as mentioned above, hitting the ELB directly without going through CloudFront results in the expected 304 result. Any ideas?
The answer was found in an obscure sentence written in a seemingly unrelated piece of Amazon documentation:
When you configure CloudFront to forward cookies to your origin [...] If-Modified-Since and If-None-Match conditional requests are not supported.
Strange, but indeed the reality of the situation is far worse; It's not that forwarding cookies to your origin servers disables conditional requests, but rather that is disables them sometimes -- to the point where the HTTP result code (304 vs 200) is virtually random.
It's important to note that you'll be bitten by this bizarre behavior even if you're not using cookies at all. It's still absolutely essential that the Forward Cookies drop-down be set to "None" as shown in the image below:
Switching the setting to "None" fixes the errant behavior described in my original post.
This solution presents you with another problem though. You're telling CloudFront to totally strip out all cookies prior to forwarding the request to your origin. But your origin server might need those cookies. Further, if you're using the ELB (load balancer) as your origin, a critical cookie that the ELB depends upon to maintain sticky sessions will be totally dropped. Not good.
The solution to the cookie-stripping problem will depend on how your site is organized. In my case, transmission of cookies (session-related or otherwise) is only necessary when posting AJAX data to myDomain.com/ajax/. Because all cookie-dependent URLs fall under the category of ajax/* , a new behavioral rule for that path had to be created and in that rule, and that rule only, the Forward Cookies drop-down is set to "All" instead of "None."
So there it is. Hope this helps someone.
I see big player (i.e. akamai) started to drop the Expires header all together and only use Cache-Control, e.g.
curl -I https://fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-snc7/395029_379645875452936_1719075242_n.jpg
HTTP/1.1 200 OK
Last-Modified: Fri, 01 Jan 2010 00:00:00 GMT
Date: Sun, 25 Nov 2012 16:46:43 GMT
Connection: keep-alive
Cache-Control: max-age=1209600
So still any reason to keep using Expires?
Cache-Control was introduced in HTTP 1.1 to replace Expires. If both headers are present, Cache-Control is preferred over Expires:
If a response includes both an Expires header and a max-age
directive, the max-age directive overrides the Expires header, even
if the Expires header is more restrictive. This rule allows an origin
server to provide, for a given response, a longer expiration time to
an HTTP/1.1 (or later) cache than to an HTTP/1.0 cache. This might be
useful if certain HTTP/1.0 caches improperly calculate ages or
expiration times, perhaps due to desynchronized clocks.
But there are still clients out there that can only HTTP 1.0. So for HTTP 1.0 requests/responses, you should still use Expires.
Here is the logo currently used on www.google.com:
http://www.google.com/images/logos/ps_logo2.png
Here's its HTTP response:
HTTP/1.1 200 OK
Content-Type: image/png
Last-Modified: Thu, 05 Aug 2010 22:54:44 GMT
Date: Fri, 25 Mar 2011 16:41:05 GMT
Expires: Fri, 25 Mar 2011 16:41:05 GMT
Cache-Control: private, max-age=31536000
X-Content-Type-Options: nosniff
Server: sffe
Content-Length: 26209
Age: 0
Via: 1.1 localhost.localdomain
The Cache-Control header says it's good for 1 year. But Expires is the same as Date, i.e. it's stale immediately.
Why the difference?
Cache-Control overrides Expires on any HTTP/1.1 cache or client.
So I assume Google wants to cache the image for HTTP/1.1 but not cache it at all for HTTP/1.0.
I don't know why Google cares. I would think they'd want to cache the logo even for older clients.
The reason is that google wants the user to cache the image but not intermediate shared caches (hence the private directive).
Many intermediate cache systems can be outdated and ignore new HTTP features (as the cache-control header), so this approach makes them not to cache the resource (via the expires header). For the rest of agents understanding both, the cache-control overrides expires header.
This is a common practice referenced in rfc2616 sec14.9.3
An origin server might wish to use a relatively new HTTP cache control feature, such as the "private" directive, on a network including older caches that do not understand that feature. The origin server will need to combine the new feature with an Expires field whose value is less than or equal to the Date value. This will prevent older caches from improperly caching the response.
I have a file proxy IHttpHandler to ensure authentication and to log requests. It works fine on the development server and IIS 6. In IIS 7, I have two problems:
Microsoft Office (Word, Excel...) sends WebDAV requests with OPTION and PROPFIND verbs. ASP.NET throws an exception since it doesn't support them. Is there any way to disable these verbs at the IIS level so that it never reaches ASP.NET? I'm guessing it would be returning a 405 Method Not Allowed error (http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_Error).
IIS 7 turns on chunked encoding. In that case the Content-Length header is not valid and apparently IIS 7 removes it: http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.4. However, it also removes the Content-Type header, causing the files to show up as text in the browser. So how can I stop IIS 7 from removing Content-Type, OR how do I turn off chunked encoding for this one page? Below are the response headers for you to compare.
Development server response:
HTTP/1.1 200 OK
Server: ASP.NET Development Server/9.0.0.0
Date: Thu, 23 Dec 2010 17:57:09 GMT
X-AspNet-Version: 2.0.50727
Content-Length: 68096
Content-Disposition: inline; filename=test.doc
Cache-Control: private
Last-Modified: Thu, 23 Dec 2010 09:14:18 GMT
Content-Type: application/msword
Connection: Close
IIS 7 response:
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Last-Modified: Thu, 23 Dec 2010 09:30:31 GMT
Server: Microsoft-IIS/7.5
Content-Disposition: inline; filename=test.doc
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Thu, 23 Dec 2010 17:57:59 GMT
My question on chunked encoding was inaccurate. I had made one small change on my development machine: I added Content-Length. On the development machine it didn't make a difference--it always worked. In IIS 7, adding Content-Length actually disabled chunked encoding and everything worked as expected.
For the WebDAV requests, IIS 7 doesn't send them through to ASP.NET so we're fine. The development server does, however. I saw a suggestion to add the DefaultHttpHandler to handle them, but on the development server that means the raw aspx page is served.