What HTTP response headers are required to be sent from server to the client?
I working to optimize the HTTP response headers to minimize the HTTP response overhead. I know "overhead" is somewhat exaggerated, but I like a clean output.
I see a lot of websites sending redundant cache headers, etc..
e.g.
It is redundant to specify both Expires and Cache-Control: max-age, or to specify both Last-Modified and ETag.
Source
HTTP/1.1: Header Field Definitions
It depends on what you define as being required: there are no header fields that must be sent with every response no matter what the circumstances are, but there are header fields that you really should send. The only header field that comes close is Date, but even it has circumstances under which it is not required.
In the parlance of RFC 2119, the term MUST means that something is a requirement of the specification and not meeting the requirement would be invalid. There are no header fields defined by RFCs 7230, 7231, 7232, 7233, 7234, or 7235 that MUST be sent by an origin server in all cases.
The following headers, for example, can be omitted (though you probably should send them):
7.1.1.2. Date
An origin server MUST NOT send a Date header field if it does not
have a clock capable of providing a reasonable approximation of the
current instance in Coordinated Universal Time. An origin server MAY
send a Date header field if the response is in the 1xx
(Informational) or 5xx (Server Error) class of status codes. An
origin server MUST send a Date header field in all other cases.
Note the last sentence of the quote. The Date header field MUST be sent if the origin server is capable of providing a "reasonable approximation" of the date in UTC, but there is nothing stopping a server from misrepresenting itself.
7.4.2. Server
An origin server MAY generate a Server field in its responses.
3.3.2. Content-Length
Aside from [a finite number of predefined cases], in the absence of
Transfer-Encoding, an origin server SHOULD send a Content-Length
header field when the payload body size is known prior to sending the
complete header section.
On the subject of Content-Length and Transfer-Encoding, note that neither can be sent, in which case the length of the response is "determined by the number of octets received prior to the server closing the connection."
3.1.1.5. Content-Type
If a Content-Type header field is not present, the recipient
MAY either assume a media type of application/octet-stream
(RFC2046, Section 4.5.1) or examine the data to determine its type.
There are circumstances under which particular headers can be required, for example:
An origin server that does not support persistent connections MUST send the Connection: close in every response that does not have a 1xx status code.
An origin server MUST generate an Allow header in a 405 (Method Not Allowed) response.
An origin server generating a 401 (Unauthorized) response MUST send a WWW-Authenticate header field containing at least one challenge.
It depends on the specifics of the response, but generally, a response from an origin server should have:
Date
Content-Type
Server
and either Content-Length, Transfer-Encoding or Connection: close.
If you want to do caching, add Cache-Control (e.g., with max-age); Expires isn't generally necessary any more. If you want clients to be able to validate, add Last-Modified or ETag.
Related
I would like to know how many headers are allowed in HTTP protocol. Is there any limit on this?
Is this limit different for Request and Response?
Gone through HTTP RFC, but no luck. Could you please provide me any official document or link about this.
HTTP/1.1 does not define such a limit. See RFC 7230, Section 3.2.5 (https://greenbytes.de/tech/webdav/rfc7230.html#field.limits):
3.2.5. Field Limits
HTTP does not place a predefined limit on the length of each header
field or on the length of the header section as a whole, as described
in Section 2.5. Various ad hoc limitations on individual header field
length are found in practice, often depending on the specific field
semantics.
A server that receives a request header field, or set of fields,
larger than it wishes to process MUST respond with an appropriate 4xx
(Client Error) status code. Ignoring such header fields would increase
the server's vulnerability to request smuggling attacks (Section 9.5).
A client MAY discard or truncate received header fields that are
larger than the client wishes to process if the field semantics are
such that the dropped value(s) can be safely ignored without changing
the message framing or response semantics.
HTTP does not define any limit. But beware that web servers may define the max size of headers they receive.
"Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]." Source: https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html
The http spec says about the HEAD request:
The HEAD method is identical to GET except that the server MUST NOT
return a message-body in the response. The metainformation contained
in the HTTP headers in response to a HEAD request SHOULD be identical
to the information sent in response to a GET request.
And also:
The Content-Length entity-header field indicates the size of the
entity-body, in decimal number of OCTETs, sent to the recipient or, in
the case of the HEAD method, the size of the entity-body that would
have been sent had the request been a GET.
So if the requested resource is dynamically generated, for the HEAD request, will the resource ALSO be generated? Should there a Content-Length header or Transfer-Encoding: chunked header?
(I feel this is about how to implement the HTTP protocol.)
ADD 1
I don't see the spec mandates whether to send Content-Length header or how to send it for a dynamically generated resource. Such dynamic resource will be sent with chunked transfer encoding and there'll be no Content-Length header if accessed with a GET method. So, if HEAD response should mimic GET response, the Contnet-Length should not be sent either.
a) That's not the HTTP spec. It is the W3C copy of the now obsolete RFC 2616, dated 1999.
b) For the current spec, see https://greenbytes.de/tech/webdav/rfc7231.html#HEAD
To answer your question: if you don't know the payload size without actually generating it, not sending Content-Length in the HEAD response is ok.
I've been reading RFC 2616 - sec14.26 and RFC 2616 - sec13.3.3 for quite a long time, but I'm still not sure if I understand correctly.
The section 14.26 states that
If any of the entity tags match the entity tag of the entity that would have been returned in the response to a similar GET request (without the If-None-Match header) on that resource, or if "*" is given and any current entity exists for that resource, then the server MUST NOT perform the requested method, unless required to do so because the resource's modification date fails to match that supplied in an If-Modified-Since header field in the request. Instead, if the request method was GET or HEAD, the server SHOULD respond with a 304 (Not Modified) response, including the cache- related header fields (particularly ETag) of one of the entities that matched.
,
If none of the entity tags match, then the server MAY perform the requested method as if the If-None-Match header field did not exist, but MUST also ignore any If-Modified-Since header field(s) in the request. That is, if no entity tags match, then the server MUST NOT return a 304 (Not Modified) response.
and the section 13.3.3 states that
An HTTP/1.1 origin server, upon receiving a conditional request that includes both a Last-Modified date (e.g., in an If-Modified-Since or If-Unmodified-Since header field) and one or more entity tags (e.g., in an If-Match, If-None-Match, or If-Range header field) as cache validators, MUST NOT return a response status of 304 (Not Modified) unless doing so is consistent with all of the conditional header fields in the request.
I'm quite confused whether a unconditionally compliant HTTP server should response to a client request (GET / HEAD) where
the condition If-Modified-Since matches, but If-None-Match doesn't.
or, the condition If-None-Match matches, but If-Modified-Since doesn't.
According to the section 14.26, 304 Not Modified should be returned in the first case and 200 OK in the second case since If-Modified-Since is ignored.
But the section 13.3.3 says that "it MUST NOT return a response status of 304 (Not Modified) unless doing so is consistent with all of the conditional header fields in the request.", meaning that 200 OK should be returned in the both cases.
from RFC 7232
A recipient must ignore If-Modified-Since if the request contains an If-None-Match header field; the condition in If-None-Match is considered to be a more accurate replacement for the condition in If-Modified-Since, and the two are only combined for the sake of interoperating with older intermediaries that might not implement If-None-Match.
Is the Content-Length header required for a HTTP/1.0 response? The HTTP spec mentions that it is required for the request, but doesn't mention anything about the response:
http://www.w3.org/Protocols/HTTP/1.0/draft-ietf-http-spec.html#Content-Length
A valid Content-Length field value is required on all HTTP/1.0
request messages containing an entity body.
If it is not required for the response, how does the client read the response when it's larger than 1MB?
Section 10.4 of the spec (which you linked to) doesn't say anything about requirements on responses itself, but instead links to section 7.2.2, which specifies that the server can indicate the length of a response containing an entity body by
sending a Content-Length header, or
closing the connection when the entire response has been sent.
Section 7.2 says that responses to HEAD requests, and 1xx, 204 or 304 responses, should not include an entity body, and therefore need not include a Content-Length header; and
All other responses must include an entity body or a Content-Length header field defined with a value of zero (0).
So to answer the question: When no Content-Length is received, the client keeps reading until the server closes the connection.
What status code should be returned if a client sends an HTTP request and specifies a Content-Encoding header which cannot be decoded by the server?
Example
A client POSTs JSON data to a REST resource and encodes the entity body using the gzip coding. However, the server can only decode DEFLATE codings because it failed the gzip class in server school.
What HTTP response code should be returned? I would say 415 Unsupported Media Type but it's not the entity's Content-Type that is the problem -- it's the encoding of the otherwise supported entity body.
Which is more appropriate: 415? 400? Perhaps a custom response code?
Addendum: I have, of course, thoroughly checked rfc2616. If the answer is there I may need some new corrective eyewear, but I don't believe that it is.
Update:
This has nothing to do with sending a response that might be unacceptable to a client. The problem is that the client is sending the server what may or may not be a valid media type in an encoding the server cannot understand (as per the Content-Encoding header the client packaged with the request message).
It's an edge-case and wouldn't be encountered when dealing with browser user-agents, but it could crop up in REST APIs accepting entity bodies to create/modify resources.
As i'm reading it, 415 Unsupported Media Type sounds like the most appropriate.
From RFC 2616:
10.4.16 415 Unsupported Media Type
The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method.
Yeah, the text part says "media type" rather than "encoding", but the actual description doesn't include any mention of that distinction.
The new hotness, RFC 7231, is even explicit about it:
6.5.13. 415 Unsupported Media Type
The 415 (Unsupported Media Type) status code indicates that the
origin server is refusing to service the request because the payload
is in a format not supported by this method on the target resource.
The format problem might be due to the request's indicated
Content-Type or Content-Encoding, or as a result of inspecting the
data directly.
They should make that the final question on Who Wants To Be a Millionaire!
Well the browser made a request that the server cannot service because the information the client provided is in a format that cannot be handled by the server. However, this isn't the server's fault for not supporting the data the client provided, it's the client's fault for not listening to the server's Acccept-* headers and providing data in an inappropriate encoding. That would make it a Client Error (400 series error code).
My first instinct is 400 Bad Request is the appropriate response in this case.
405 Method Not Allowed isn't right because it refers to the HTTP verb being one that isn't allowed.
406 Not Acceptable looks like it might have promise, but it refers to the server being unable to provide data to the client that satisfies the Accept-* request headers that it sent. This doesn't seem like it would fit your case.
412 Precondition Failed is rather vaguely defined. It might be appropriate, but I wouldn't bet on it.
415 Unsupported Media Type isn't right because it's not the data type that's being rejected, it's the encoding format.
After that we get into the realm of non-standard response codes.
422 Unprocessable Entity describes a response that should be returned if the request was well-formed but if it was semantically incorrect in some way. This seems like a good fit, but it's a WebDAV extension to HTTP and not standard.
Given the above, I'd personally opt for 400 Bad Request. If any other HTTP experts have a better candidate though, I'd listen to them instead. ;)
UPDATE: I'd previously been referencing the HTTP statuses from their page on Wikipedia. Whilst the information there seems to be accurate, it's also less than thorough. Looking at the specs from W3C gives a lot more information on HTTP 406, and it's leading me to think that 406 might be the right code after all.
10.4.7 406 Not Acceptable
The resource identified by the request is only capable of generating
response entities which have content characteristics not acceptable
according to the accept headers sent in the request.
Unless it was a HEAD request, the response SHOULD include an entity
containing a list of available entity characteristics and location(s)
from which the user or user agent can choose the one most appropriate.
The entity format is specified by the media type given in the
Content-Type header field. Depending upon the format and the
capabilities of the user agent, selection of the most appropriate
choice MAY be performed automatically. However, this specification
does not define any standard for such automatic selection.
Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
If the response could be unacceptable, a user agent SHOULD temporarily
stop receipt of more data and query the user for a decision on further
actions.
While it does mention the Content-Type header explicitly, the wording mentions "entity characteristics", which you could read as covering stuff like GZIP versus DEFLATE compression.
One thing worth noting is that the spec says that it may be appropriate to just send the data as is, along with the headers to tell the client what format it's in and what encoding it uses, and just leave it for the client to sort out. So if the client sends a header indicating it accepts GZIP compression, but the server can only generate a response with DEFLATE, then sending that along with headers saying it's DEFLATE should be okay (depending on the context).
Client: Give me a GZIPPED page.
Server: Sorry, no can do. I can DEFLATE pack it for you. Here's the DEFLATE packed page. Is that okay for you?
Client: Welllll... I didn't really want DEFLATE, but I can decode it okay so I'll take it.
(or)
Client: I think I'll have to clear that with my user. Hold on.