After reading a lot about caching validators (more intensively after reading this answer on SO), I had a doubt that didn't find the answer anywhere.
My use-case is to serve a static asset (a javascript file, ie: https://example.com/myasset.js) to be used in other websites, so messing with their Gpagespeed/gmetrix score matters the most.
I also need their users to receive updated versions of my static asset every time I deploy new changes.
For this, I have the following response headers:
Cache-Control: max-age=10800
etag: W/"4efa5de1947fe4ce90cf10992fa"
In short, we can see the following flow in terms of how browser behaves using etag
For the first request, the browser has no value for the If-None-Match Request Header, so the Server will send back the status code 200 (Ok), the content itself, and a Response header with ETag value.
For the subsequent requests, the browser will add the previously received ETag value in a form of the If-None-Match Request Header. This way, the server can compare this value with the current value from ETag and, if both match, the server can return 304 (Not Modified) telling the browser to use the latest version of the file, or just 200 followed by the new content and the related ETag value instead.
However, I couldn't find any information in regards to using the Cache-Control: max-age header and how will this affect the above behavior, like:
Will the browser request for new updates before max-age has met? Meaning that I can define a higher max-age value (pagespeed/gmetrix will be happy about it) and force this refresh using only etag fingerprint.
If not, then what are the advantages of using etag and adding extra bits to the network?
No, the browser will not send any requests until max-age has passed.
The advantage of using ETag is that, if the file hasn't changed, you don't need to resend the entire file to the client. The response will be a small 304.
Note that you can achieve the best of both worlds by using the stale-while-revalidate directive, which allows stale responses to be served while the cache silently revalidates the resource in the background.
I see a lot of sites that return an html page with no caching parameters in the header. No pragma, no cache-control, no e-tag, no expiration date... nothing... From the http 1.2 spec, it seems like this means it can be cached but I am not sure. Any one know the exact rule that governs caching if there are no cache directives in the response?
I think I found the answer. The http 1.1 spec allows for a scenario where there are no cache directives. In this case the receiving cache can use a Heuristic Expiration algorithm. For me that is the answer. It is cache-able. However, I have found that firefox and Chrome will not cache it. I did see a post that found the chrome source that sets "DEFAULT_CACHE_TIME = 300" which I believe is about 5 minutes or maybe it's 5 ms.. I don't know... Anyway, I just needed to know if the spec allowed a cache to cache an object that had had no cache directives.
Below quote is from:
http://home.anadolu.edu.tr/~egermen/EEM534/Refreshment%20policies%20for%20Web%20content%20caches%20.pdf
Otherwise, no explicit
freshness lifetime is provided by the origin
server and a heuristic is used: the freshness lifetime
is assigned to be a fraction (HTTP/1.1 mentions
10% as an example) of the time difference between
the timestamp at the DATE header and the time
specified by the LAST-MODIFIED header, subject
to a maximum allowed value (usually 24 h, since
HTTP/1.1 requires that the cache must attach a
warning if heuristic expiration is used and the
object’s age exceeds 24 h).
I'm trying to work out a new caching policy for the static resources on a website. A common problem is whenever javascript, CSS etc. is updated, many users hold onto stale versions because currently there are no caching specific HTTP headers included in the file responses.
This becomes a serious problem when, for example, the javascript updates are linked to server-side updates, and the stale javascript chokes on the new server responses.
Eliminating browser caching completely with a cache-control: max-age=0, no-cache seems like overkill, since I'd still like to take some pressure off the server by letting browsers cache temporarily. So, setting the cache policy to a maximum of one hour seems alright, like cache-control: max-age=3600, no-cache.
My understanding is that this will always fetch a new copy of the resource if the cached copy is older than one hour. I'd specifically like to know if it's possible to set a HTTP header or combination of headers that will instruct browsers to only fetch a new copy if the resource was last checked more than one hour ago AND if the resource has changed.
I'm just trying to avoid browsers blindly fetching new copies just because the cached resource is older than one hour, so I'd also like to add the condition that the resource has been changed.
Just to illustrate further what I'm asking:
New user arrives at site and gets fresh copy of script.js
User stays on site for 45 mins, browser uses cached copy of script.js all the time
User comes back to site 2 hours later, and browser asks the server if script.js has changed
If it has, then it gets a fresh copy and the process repeats
If it has not changed, then it uses the cached copy for the next hour, after which it will check again
Have I misunderstood things? Is what I'm asking how it actually works, or do I have to do something different?
Have I misunderstood things? Is what I'm asking how it actually works,
or do I have to do something different?
You have some serious misconceptions about what the various cache control directives do and why cache behaves as it does.
Eliminating browser caching completely with a cache-control:
max-age=0, no-cache seems like overkill, since I'd still like to take
some pressure off the server by letting browsers cache temporarily ...
The no-cache option is wrong too. Including it means the browser will always
check with the server for modifications to the file every time.
That isn't what the no-cache means or what it is intended for - it means that a client MUST NOT used a cached copy to satisfy a subsequent request without successful revalidation - it does not and has never meant "do not cache" - that is what the no-store directive is for
Also the max-age directive is just the primary means for caches to calculate the freshness lifetime and expiration time of cached entries. The Expires header (minus the value of the Date header can also be used) - as can a heuristic based on the current UTC time and any Last-Modified header value.
Really if your goal is to retain the cached copy of a resource for as long as it is meaningful - whilst minimising requests and responses you have a number of options.
The Etag (Entity Tag) header - this is supplied by the server in response to a request in either a "strong" or "weak" form. It is usually a hash based on the resource in question. When a client re-requests a resource it can pass the stored value of the Etag with the If-None-Match request header. If the resource has not changed then the server will respond with 304 Not Modified.
You can think Etags as fingerprints for resources. They can be used to massively reduce the amount of information sent over the wire - as only fresh data is served - but they do not have any bearing on the number of times or frequency of requests.
The last-modified header - this is supplied by the server in response to a request in HTTPdate format - it tells the client the last time the resource was modified.
When a client re-requests a resource it can pass the stored value of the last-modified header with the If-Modified-Since request header. If the resource has not changed since the time it was last modified then the server will respond with 304 Not Modified.
You can think of last modified as a weaker form of entity checking than Etags. It addresses the same problem (bandwidth/redundancy) it in a less robust way and again it has no bearing at all on the actual number of requests made.
Revving - a technique that use a combination of the Expires header and the name (URN) of a resource. (see stevesouders blog post)
Here one basically sets a far forward Expires header - say 5 years from now - to ensure the static resource is cached for a long time.
You then have have two options for updating - either by appending a versioning query string to the requests URL - e.g. "/mystyles.css?v=1.1" - and updating the version number as and when the resource changes. Or better - versioning the file name itself e.g. "/mystyles.v1.1.css" so that each version is cached for as long as possible.
This way not only do you reduce the amount of bandwidth - you will as eliminate all checks to see if the resource has changed until you rename it.
I suppose the main point here is none of the catch control directives you mention max-age, public, etc have any bearing at all on if a 304 response is generated or not. For that use either Etag / If-None-Match or last-modified / If-Modified-Since or a combination of them (with If-Modified-Since as a fallback mechanism to If-None-Match).
It seems that I have misunderstood how it works, because some testing in Chrome has revealed exactly the behavior that I was looking for in the 5 steps I mentioned.
It doesn't blindly grab a fresh copy from the server when the max-age has expired. It does a GET, and if the response is 304 (Not Modified), it continues using the cached copy until the next hour has expired, at which point it checks for changes again etc.
The no-cache option is wrong too. Including it means the browser will always check with the server for modifications to the file every time. So what I was really looking for is:
Cache-Control: public, max-age=3600
I have a mess with this header, I have read that Cache-Control:must-revalidate oblige to validate all requests with the source before serving a cached item, but just the stale ones? or all no matter if stale or fresh? I have read both things in different places.
What is the difference with Cache-Control:no-cache ? Because these headers look equivalent to me.
UPDATE 1: I have read this from a book:
The Cache-Control: must-revalidate response header tells the cache
to bypass the freshness calculation mechanisms and revalidate on every
access:
#Peter O. has pointed out what the RFC says. So that old book is wrong.
UPDATE 2: In this tutorial : http://www.mnot.net/cache_docs/
no-cache — forces caches to submit the request to the origin server
for validation before releasing a cached copy, every time. This is
useful to assure that authentication is respected (in combination with
public), or to maintain rigid freshness, without sacrificing all of
the benefits of caching.
must-revalidate — tells caches that they must
obey any freshness information you give them about a representation.
HTTP allows caches to serve stale representations under special
conditions; by specifying this header, you’re telling the cache that
you want it to strictly follow your rules.
Section 14.9.4 of HTTP/1.1:
When the must-revalidate directive is present in a response
received by
a cache, that cache MUST NOT use the entry after it becomes
stale
to respond to a subsequent request without first revalidating it
with the
origin server
Section 14.8 of HTTP/1.1:
If the response includes the "must-revalidate" cache-control
directive, the cache MAY use that response in replying to a
subsequent request. But if the response is stale, all caches
MUST first revalidate it with the origin server...
So it appears that only stale responses must be revalidated if
must-revalidate is received.
For no-cache, see section 14.9.1:
If the no-cache directive does not specify a field-name [which is
the case
here], then a cache MUST NOT use the response to satisfy a
subsequent
request without successful revalidation with the origin server...
Thus, no-cache applies both to fresh and stale responses.
EDIT:
This phrase may be relevant here (section 13.3):
When a cache has a stale entry that it would like to use as a response
to a client's request, it first has to check with the origin server
(or possibly an intermediate cache with a fresh response) to see if
its cached entry is still usable.
So, must-revalidate is probably relevant when the cache has intermediate
caches, since otherwise the cache can check the intermediate cache for a
fresh response rather than check the origin server directly.
I'm told to prevent user-info leaking, only "no-cache" in response is not enough. "no-store" is also necessary.
Cache-Control: no-cache, no-store
After reading this spec http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html, I'm still not quite sure why.
My current understanding is that it is just for intermediate cache server. Even if "no-cache" is in response, intermediate cache server can still save the content to non-volatile storage. The intermediate cache server will decide whether using the saved content for following request. However, if "no-store" is in the response, the intermediate cache sever is not supposed to store the content. So, it is safer.
Is there any other reason we need both "no-cache" and "no-store"?
I must clarify that no-cache does not mean do not cache. In fact, it means "revalidate with server" before using any cached response you may have, on every request.
must-revalidate, on the other hand, only needs to revalidate when the resource is considered stale.
If the server says that the resource is still valid then the cache can respond with its representation, thus alleviating the need for the server to resend the entire resource.
no-store is effectively the full do not cache directive and is intended to prevent storage of the representation in any form of cache whatsoever.
I say whatsoever, but note this in the RFC 2616 HTTP spec:
History buffers MAY store such responses as part of their normal operation
But this is omitted from the newer RFC 7234 HTTP spec in potentially an attempt to make no-store stronger, see:
https://www.rfc-editor.org/rfc/rfc7234#section-5.2.1.5
Under certain circumstances, IE6 will still cache files even when Cache-Control: no-cache is in the response headers.
The W3C states of no-cache:
If the no-cache directive does not
specify a field-name, then a cache
MUST NOT use the response to satisfy a
subsequent request without successful
revalidation with the origin server.
In my application, if you visited a page with the no-cache header, then logged out and then hit back in your browser, IE6 would still grab the page from the cache (without a new/validating request to the server). Adding in the no-store header stopped it doing so. But if you take the W3C at their word, there's actually no way to control this behavior:
History buffers MAY store such responses as part of their normal operation.
General differences between browser history and the normal HTTP caching are described in a specific sub-section of the spec.
From the HTTP 1.1 specification:
no-store:
The purpose of the no-store directive is to prevent the inadvertent release or retention of sensitive information (for example, on backup tapes). The no-store directive applies to the entire message, and MAY be sent either in a response or in a request. If sent in a request, a cache MUST NOT store any part of either this request or any response to it. If sent in a response, a cache MUST NOT store any part of either this response or the request that elicited it. This directive applies to both non- shared and shared caches. "MUST NOT store" in this context means that the cache MUST NOT intentionally store the information in non-volatile storage, and MUST make a best-effort attempt to remove the information from volatile storage as promptly as possible after forwarding it.
Even when this directive is associated with a response, users might explicitly store such a response outside of the caching system (e.g., with a "Save As" dialog). History buffers MAY store such responses as part of their normal operation.
The purpose of this directive is to meet the stated requirements of certain users and service authors who are concerned about accidental releases of information via unanticipated accesses to cache data structures. While the use of this directive might improve privacy in some cases, we caution that it is NOT in any way a reliable or sufficient mechanism for ensuring privacy. In particular, malicious or compromised caches might not recognize or obey this directive, and communications networks might be vulnerable to eavesdropping.
no-store should not be necessary in normal situations, and can harm both speed and usability. It is intended for use where the HTTP response contains information so sensitive it should never be written to a disk cache at all, regardless of the negative effects that creates for the user.
How it works:
Normally, even if a user agent such as a browser determines that a response shouldn't be cached, it may still store it to the disk cache for reasons internal to the user agent. This version may be utilised for features like "view source", "back", "page info", and so on, where the user hasn't necessarily requested the page again, but the browser doesn't consider it a new page view and it would make sense to serve the same version the user is currently viewing.
Using no-store will prevent that response being stored, but this may impact the browser's ability to give "view source", "back", "page info" and so on without making a new, separate request for the server, which is undesirable. In other words, the user may try viewing the source and if the browser didn't keep it in memory, they'll either be told this isn't possible, or it will cause a new request to the server. Therefore, no-store should only be used when the impeded user experience of these features not working properly or quickly is outweighed by the importance of ensuring content is not stored in the cache.
My current understanding is that it is just for intermediate cache server. Even if "no-cache" is in response, intermediate cache server can still save the content to non-volatile storage.
This is incorrect. Intermediate cache servers compatible with HTTP 1.1 will obey the no-cache and must-revalidate instructions, ensuring that content is not cached. Using these instructions will ensure that the response is not cached by any intermediate cache, and that all subsequent requests are sent back to the origin server.
If the intermediate cache server does not support HTTP 1.1, then you will need to use Pragma: no-cache and hope for the best. Note that if it doesn't support HTTP 1.1 then no-store is irrelevant anyway.
If you want to prevent all caching (e.g. force a reload when using the back button) you need:
no-cache for IE
no-store for Firefox
There's my information about this here:
http://blog.httpwatch.com/2008/10/15/two-important-differences-between-firefox-and-ie-caching/
For chrome, no-cache is used to reload the page on a re-visit, but it still caches it if you go back in history (back button). To reload the page for history-back as well, use no-store. IE needs must-revalidate to work in all occasions.
So just to be sure to avoid all bugs and misinterpretations I always use
Cache-Control: no-store, no-cache, must-revalidate
if I want to make sure it reloads.
If a caching system correctly implements no-store, then you wouldn't need no-cache. But not all do. Additionally, some browsers implement no-cache like it was no-store. Thus, while not strictly required, it's probably safest to include both.
Note that Internet Explorer from version 5 up to 8 will throw an error when trying to download a file served via https and the server sending Cache-Control: no-cache or Pragma: no-cache headers.
See http://support.microsoft.com/kb/812935/en-us
The use of Cache-Control: no-store and Pragma: private seems to be the closest thing which still works.
Originally we used no-cache many years ago and did run into some problems with stale content with certain browsers... Don't remember the specifics unfortunately.
We had since settled on JUST the use of no-store. Have never looked back or had a single issue with stale content by any browser or intermediaries since.
This space is certainly dominated by reality of implementations vs what happens to have been written in various RFCs. Many proxies in particular tend to think they do a better job of "improving performance" by replacing the policy they are supposed to be following with their own.
Just to make things even worse, in some situations, no-cache can't be used, but no-store can:
http://faindu.wordpress.com/2008/04/18/ie7-ssl-xml-flex-error-2032-stream-error/
To answer the question, there are two players here, the client (request) and the server (response).
Client:
The client can only request with ONE cache method. There are different methods and if not specified, will use default.
default: Inspect browser cache:
If cached and "fresh": Return from cache.
If cached, stale, but still "valid": Return from cache, and schedule a fetch to update cache (for next use).
If cached and stale: Fetch with conditions, cache, and return.
If not cached: Fetch, cache, and return.
no-store: Fetch and return.
reload: Fetch, cache, and return. (default-4)
no-cache: Inspect browser cache:
If cached: Fetch with conditions, cache, and return. (default-3)
If not cached: Fetch, cache, and return. (default-4)
force-cache: Inspect browser cache:
If cached: Return it regardless if stale.
If not cache: Fetch, cache, and return. (default-4)
only-if-cached: Inspect browser cache:
If cached: Return it regardless if stale.
If not cached: Throw network error.
Notes:
Still "valid" means the current age is within the stale-while-revalidate lifetime. It needs "revalidation", but is still acceptable to return.
"Fetch" here, for simplicity, is short for "non-conditional network
fetch".
"Fetch with conditions" means fetch using headers like
If-Modified-Since, or ETag so the server can respond with 304: (Not Modified).
https://fetch.spec.whatwg.org/#concept-request-cache-mode
Server::
Now that we understand what the client can do, the server responses make more sense.
Looking at the Cache-Control header, if the server returns:
no-store: Tells client to not use cache at all
no-cache: Tells client it should do conditional requests and ignore freshness
max-age: Tells client how long a cache is "fresh"
stale-while-revalidate: Tells client how long cache is "valid"
immutable: Cache forever
Now we can put it all together. That means the only possibilities are:
Non-conditional network fetch
Conditional network fetch
Return stale cache
Return stale but valid cache
Return fresh cache
Return any cache
Any combination of client, or server can dictate what method, or set of methods, to use. If the server returns no-store, it's not going to hit the cache, no matter what the client request type. If the client request was no-store, it doesn't matter what the server returns, it won't cache. If the client doesn't specify a request type, the server will dictate it with Cache-Control.
It makes no sense for a server to return both no-cache and no-store since no-store overrides everything. Yes, you've probably seen both together, and it's useless outside of broken browser implementations. Still, no-store has been part of spec since 1999: https://datatracker.ietf.org/doc/html/rfc2616#section-14.9.2
In real life usage, if your server supports 304: Not Modified, and you want to use client cache as a way to improve speed, but still want to force a network fetch, use no-cache. If don't support 304, and want to force a network fetch, use no-store. If you're okay with cache sometimes, use freshness and revalidation headers.
In reality, if you're mixing up no-cache and no-store on the client, very little would change. Then, just a couple of headers get sent and there will different internal responses handled by the browser. An issue can occur if you use no-cache and then forget to use it later. no-cache tells it to store the response in the cache, and a later request without it might trigger internal cache.
There are times when you may want to mix methods even on the same resource based on context. For example, you may want to use reload on a service worker and background sync, but use default for the web page itself. This is where you can manipulate the user agent (browser) cache to your liking. Just remember that the server generally has the final say as to how the cache should work.
To clarify some possible future confusion. The client can use the Cache-Control header on the request, to tell the server to not use its own cache system when responding. This is unrelated to the browser/server dynamic, and more about the server/database dynamic.
Also no-store technically means must not store to any non-volatile storage (disk) and release it from volatile storage (memory) ASAP. In practice, it means don't use a cache at all. The command actually goes both ways. A client request with no-store shouldn't write to disk or database and is meant to transient.
TL;DR: no-store overrides no-cache. Setting both is useless, unless we are talking out-of-spec or HTTP/1.0 browsers that don't support no-store (Maybe IE11?). Use no-cache for 304 support.
A pretty old topic but I'll share some recent ideas:
no-store: Must not attempt to store anything, and must also take action to delete any copy it might have.
no-cache: Never use a local copy without first validating with the origin server. It prevents all possibility of a cache hit, even with fresh resources.
So, answering the question, using only one of them is enough.
Also, some (not very) recent works prove that browsers are more Cache-Control compatible nowadays.
OWASP discusses this:
What's the difference between the cache-control directives: no-cache, and no-store?
The no-cache directive in a response indicates that the response must not be used to serve a subsequent request i.e. the cache must not display a response that has this directive set in the header but must let the server serve the request. The no-cache directive can include some field names; in which case the response can be shown from the cache except for the field names specified which should be served from the server. The no-store directive applies to the entire message and indicates that the cache must not store any part of the response or any request that asked for it.
Am I totally safe with these directives?
No. But generally, use both Cache-Control: no-cache, no-store and Pragma: no-cache, in addition to Expires: 0 (or a sufficiently backdated GMT date such as the UNIX epoch). Non-html content types like pdf, word documents, excel spreadsheets, etc often get cached even when the above cache control directives are set (although this varies by version and additional use of must-revalidate, pre-check=0, post-check=0, max-age=0, and s-maxage=0 in practice can sometimes result at least in file deletion upon browser closure in some cases due to browser quirks and HTTP implementations). Also, 'Autocomplete' feature allows a browser to cache whatever the user types in an input field of a form. To check this, the form tag or the individual input tags should include 'Autocomplete="Off" ' attribute. However, it should be noted that this attribute is non-standard (although it is supported by the major browsers) so it will break XHTML validation.
Source here.