I'm trying to write a program to match HTTP requests with their corresponding responses. Seems that everything is working well for most of the scenarios (when the transfer is perfectly ordered and even when its not, by using TCP sequence numbers).
The only problem I found is for when I have pipelined requests. After that, I get several responses but I don't know which packets are the answer to a specific request and which are not. I read in another post that the responses will come sequentially and combining this property with information on the Content-Length field seems to be a solution. The problem is that Content-length is not a mandatory field, so I'm not sure if I can always rely on that.
Does anyone know how the web-browsers that support this feature (btw, not most of them do) actually do it?
The information about the bodies length has to be present in the headers. It's just not always in 'content-length'. In order to work it all out you will have to study the relevant RFC 2616. Most notably section 4.4 deals with the different headers
Some more relevant rules from the RFC 2616:
When pipelining:
A server MUST send its responses to those requests in the same order that the requests were received.
From 9.2
If no response body is included, the response MUST include a Content-Length field with a field-value of "0".
From 10.2.7 206 Partial Content
The response MUST include .... Either a Content-Range header field ... or a multipart/byteranges
Content-Type including Content-Range fields for each part.
From 14.13 Content-Length
Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.
Current responses are a bit old. Need a refresh.
The new HTTP 1.1 RFC is RFC 7230. And contains more precise information on parsing the messages size.
Message Body Length
Associating a response to a request
Security Considerations
Detecting the size of a message is quite complex. You can have a Content-length, or Transfer-Encoding: chunked, or both, or none. And some sepcial codes like 100 Continue which may alter all this.
The first link contains 7 entries that should be checked in the right order to guess the right size.
And as stated in the last link, failing to detect the right message length may lead to HTTP Smuggling (splitting, cache poisoning) issues.
Pipelining support is the source of most smuggling issues. You should really take care of the whole RFC7230 document if you want to implement it.
Related
I would like to know how many headers are allowed in HTTP protocol. Is there any limit on this?
Is this limit different for Request and Response?
Gone through HTTP RFC, but no luck. Could you please provide me any official document or link about this.
HTTP/1.1 does not define such a limit. See RFC 7230, Section 3.2.5 (https://greenbytes.de/tech/webdav/rfc7230.html#field.limits):
3.2.5. Field Limits
HTTP does not place a predefined limit on the length of each header
field or on the length of the header section as a whole, as described
in Section 2.5. Various ad hoc limitations on individual header field
length are found in practice, often depending on the specific field
semantics.
A server that receives a request header field, or set of fields,
larger than it wishes to process MUST respond with an appropriate 4xx
(Client Error) status code. Ignoring such header fields would increase
the server's vulnerability to request smuggling attacks (Section 9.5).
A client MAY discard or truncate received header fields that are
larger than the client wishes to process if the field semantics are
such that the dropped value(s) can be safely ignored without changing
the message framing or response semantics.
HTTP does not define any limit. But beware that web servers may define the max size of headers they receive.
"Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]." Source: https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html
Let's say a web application gets the following request:
POST /some/endpoint HTTP/1.1
Host: <something>
Accept: application/json
Accept-Language: pt
Content-Type: application/json
If-Match: "blabla"
Some body
If the server doesn't support HTTP 1.1 and the endpoint /some/endpoint does not exist, the former problem should likely be checked first, and a 505 rather than 404 should be returned.
If it just so happens that none of the endpoints of the server accept POST and the endpoint /some/endpoint doesn't exist, the latter should get priority, and 404 should be returned rather than 405.
If the Accept can't be provided and the body can't be appropriately decoded/validated, probably 406 should take precedence over 400.
These are cases where intuition might suffice. But there are a myriad other ones where it is not clear which of two non-2XX status codes should be preferred/checked first. For example, should Content-Type (resulting in 415) or Accept-Language (406) be returned if both would fail? 415 or 412? And on it goes...
Much of the time errors are pairwise independent: if the aspect that is relevant to one error being thrown (such as a particular header value) is fixed, the success/error status of another will not be affected. In those cases, the wrong error "priority" is perhaps only a nuisance. But sometimes it may be the case that these errors are not independent: I might have a resources as HTML in Portuguese, but in JSON only in English (humour me), so that if a client expects me to prioritise Accept-Language over Accept, and I do the opposite, the result will be quite bad.
The question should be evident now: are there any standards about which errors should be prioritised?
I haven't come across any relevant RFCs, or even much serious and general discussion. I know of the webmachine diagram, which sort of helps, but primarily just seems to describe a particular (well thought out) implementation rather than any standard.
Obviously, you can’t expect this question to be answered “no,” even though that’s probably the correct answer.
So let me address a particular point of yours instead:
I might have a resources as HTML in Portuguese, but in JSON only in English (humour me), so that if a client expects me to prioritise Accept-Language over Accept, and I do the opposite, the result will be quite bad.
In your example, you tell the server that Portuguese JSON is good, but all other combinations are equally bad. If that’s not the case, you can elaborate your preferences like this:
Accept: text/json
Accept-Language: pt, en;q=0.1
The server can then multiply your weights, getting 1×0.1=0.1 for English JSON and 0×1=0 for Portuguese HTML, and choosing the former.
(Sidenote 1: there is no text/json media type in the registry. You probably want application/json.)
(Sidenote 2: 415 Unsupported Media Type is not a correct response code for the scenarios you mention. It concerns the request body. If you cannot honor the Accept header, you can respond with 406 Not Acceptable, just as with Accept-Language.)
TL;DR: The specifications give the server ultimate authority in how it honors the request, even allowing the server to ignore the acceptable formats the client requests. However, the specifications instruct the server to make a best effort and to respond in a way that best helps the client recover from errors.
The specifications provide guidance, even if they don't (or can't) prioritize all possible error modes.
RFC 2616 § 10.4.7 says:
HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
RFC 7231 § 3 says:
An origin server might be provided with, or be capable of generating,
multiple representations that are each intended to reflect the
current state of a target resource. In such cases, some algorithm is
used by the origin server to select one of those representations as
most applicable to a given request, usually based on content
negotiation.
RFC 7231 § 3.4 says:
Note that, in all cases, HTTP is not aware of the resource semantics.
The consistency with which an origin server responds to requests ... is determined entirely by whatever entity or algorithm selects
or generates those responses. HTTP pays no attention to the man
behind the curtain.
RFC 7231 § 3.3 says:
Response messages with an error status code
usually contain a payload that represents the error condition, such
that it describes the error state and what next steps are suggested
for resolving it.
RFC 2616 § 14.46 says:
The Warning general-header field is used to carry additional information about the status or transformation of a message which might not be reflected in the message. This information is typically used to warn about a possible lack of semantic transparency from caching operations or transformations applied to the entity body of the message.
(Emphases all mine.)
Section 3 of RFC 7231 gives the origin server ultimate authority to decide the appropriate response, even if that response is repugnant. Simultaneously, section 3 encourages the origin server to satisfy the request, or provide notice that it satisfied some of the request (Vary), or provide selectable options ("Passive negotiation").
Even though the server has ultimate authority, the specification makes clear to me that the responses should help the user resolve the problem. In my mind, the best error code is the one that helps the user best fix the problem!
Considering your pair-wise examples:
"If the server doesn't support HTTP 1.1 and the endpoint /some/endpoint does not exist, the former problem should likely be checked first, and a 505 rather than 404 should be returned."
No. Per the spec, an HTTP 1.1 client can GET from 1.0 server by protocol downgrade, so this kind of version negotiation is handled by the specification. Send a 404 (or a 301 if that's known) so the user can correct it.
"If it just so happens that none of the endpoints of the server accept POST and the endpoint /some/endpoint doesn't exist, the latter should get priority, and 404 should be returned rather than 405."
Yes, 404. If you're not getting to a resource, the method hardly matters.
"If the Accept can't be provided and the body can't be appropriately decoded/validated, probably 406 should take precedence over 400."
Never send 400 when you know 406 applies. You're giving the client less information, which is less helpful. However, the origin server is free to ignore the Accept header per RFC 7231 § 5.3.2:
If the [Accept] header field is
present in a request and none of the available representations for
the response have a media type that is listed as acceptable, the
origin server can either honor the header field by sending a 406 (Not
Acceptable) response or disregard the header field by treating the
response as if it is not subject to content negotiation.
"I might have a resources as HTML in Portuguese, but in JSON only in English (humour me), so that if a client expects me to prioritise Accept-Language over Accept, and I do the opposite, the result will be quite bad."
I disagree that the result will be bad. See RFC 7231 § 5.3.5:
the origin server can either disregard the [Accept-Language] header field by treating the response as if it is not subject to content negotiation or honor the header field by sending a 406 (Not Acceptable) response. However, the latter is not encouraged, as doing so can prevent users from accessing content that they might be able to use (with translation software, for example).
This pattern of specification language occurs more than once. "The server may disregard [whatever the client requested] by treating the response as if it's not subject to [this part of the specification], or the server may honor [the client request] and send [an applicable error code]. But, it's better to [send something intelligible] than only send [an inscrutable error code]."
At the end of the day, it's your API. HTTP provides only a window into your semantics. Document what you accept, how you respond, and with what. Send intelligible responses (HATEOAS is good) and, when applicable, the most specific error codes available.
What status code should be returned if a client sends an HTTP request and specifies a Content-Encoding header which cannot be decoded by the server?
Example
A client POSTs JSON data to a REST resource and encodes the entity body using the gzip coding. However, the server can only decode DEFLATE codings because it failed the gzip class in server school.
What HTTP response code should be returned? I would say 415 Unsupported Media Type but it's not the entity's Content-Type that is the problem -- it's the encoding of the otherwise supported entity body.
Which is more appropriate: 415? 400? Perhaps a custom response code?
Addendum: I have, of course, thoroughly checked rfc2616. If the answer is there I may need some new corrective eyewear, but I don't believe that it is.
Update:
This has nothing to do with sending a response that might be unacceptable to a client. The problem is that the client is sending the server what may or may not be a valid media type in an encoding the server cannot understand (as per the Content-Encoding header the client packaged with the request message).
It's an edge-case and wouldn't be encountered when dealing with browser user-agents, but it could crop up in REST APIs accepting entity bodies to create/modify resources.
As i'm reading it, 415 Unsupported Media Type sounds like the most appropriate.
From RFC 2616:
10.4.16 415 Unsupported Media Type
The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method.
Yeah, the text part says "media type" rather than "encoding", but the actual description doesn't include any mention of that distinction.
The new hotness, RFC 7231, is even explicit about it:
6.5.13. 415 Unsupported Media Type
The 415 (Unsupported Media Type) status code indicates that the
origin server is refusing to service the request because the payload
is in a format not supported by this method on the target resource.
The format problem might be due to the request's indicated
Content-Type or Content-Encoding, or as a result of inspecting the
data directly.
They should make that the final question on Who Wants To Be a Millionaire!
Well the browser made a request that the server cannot service because the information the client provided is in a format that cannot be handled by the server. However, this isn't the server's fault for not supporting the data the client provided, it's the client's fault for not listening to the server's Acccept-* headers and providing data in an inappropriate encoding. That would make it a Client Error (400 series error code).
My first instinct is 400 Bad Request is the appropriate response in this case.
405 Method Not Allowed isn't right because it refers to the HTTP verb being one that isn't allowed.
406 Not Acceptable looks like it might have promise, but it refers to the server being unable to provide data to the client that satisfies the Accept-* request headers that it sent. This doesn't seem like it would fit your case.
412 Precondition Failed is rather vaguely defined. It might be appropriate, but I wouldn't bet on it.
415 Unsupported Media Type isn't right because it's not the data type that's being rejected, it's the encoding format.
After that we get into the realm of non-standard response codes.
422 Unprocessable Entity describes a response that should be returned if the request was well-formed but if it was semantically incorrect in some way. This seems like a good fit, but it's a WebDAV extension to HTTP and not standard.
Given the above, I'd personally opt for 400 Bad Request. If any other HTTP experts have a better candidate though, I'd listen to them instead. ;)
UPDATE: I'd previously been referencing the HTTP statuses from their page on Wikipedia. Whilst the information there seems to be accurate, it's also less than thorough. Looking at the specs from W3C gives a lot more information on HTTP 406, and it's leading me to think that 406 might be the right code after all.
10.4.7 406 Not Acceptable
The resource identified by the request is only capable of generating
response entities which have content characteristics not acceptable
according to the accept headers sent in the request.
Unless it was a HEAD request, the response SHOULD include an entity
containing a list of available entity characteristics and location(s)
from which the user or user agent can choose the one most appropriate.
The entity format is specified by the media type given in the
Content-Type header field. Depending upon the format and the
capabilities of the user agent, selection of the most appropriate
choice MAY be performed automatically. However, this specification
does not define any standard for such automatic selection.
Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
If the response could be unacceptable, a user agent SHOULD temporarily
stop receipt of more data and query the user for a decision on further
actions.
While it does mention the Content-Type header explicitly, the wording mentions "entity characteristics", which you could read as covering stuff like GZIP versus DEFLATE compression.
One thing worth noting is that the spec says that it may be appropriate to just send the data as is, along with the headers to tell the client what format it's in and what encoding it uses, and just leave it for the client to sort out. So if the client sends a header indicating it accepts GZIP compression, but the server can only generate a response with DEFLATE, then sending that along with headers saying it's DEFLATE should be okay (depending on the context).
Client: Give me a GZIPPED page.
Server: Sorry, no can do. I can DEFLATE pack it for you. Here's the DEFLATE packed page. Is that okay for you?
Client: Welllll... I didn't really want DEFLATE, but I can decode it okay so I'll take it.
(or)
Client: I think I'll have to clear that with my user. Hold on.
I am working on an existing Silverlight file uploader that breaks files into multiple chunks and transmits the file using multiple HTTP requests.
Currently, it sends the start and total byte information on the querystring, but as learning exercise, I'd like to use a more standards-based approach.
I've previously used the HTTP Content-Range header when implementing an endpoint that serves content. Is this header also appropriate to use when posting content from a client to the server?
Yes.
RFC 2616 (HTTP 1.1), Section 14 begins by stating:
For entity-header fields, both sender and recipient refer to either
the client or the server, depending on who sends and who receives the
entity.
Other than that, Section 14.16, which defines the Content-Range header, does not appear to contain any language limiting its use to either the request or response.
Probably not, at least as of 2014 (the original answer is from 2011).
The updated HTTP 1.1 specification, rfc7231 (4.3.3), says the following about valid POST responses:
An origin server indicates response semantics by choosing an
appropriate status code depending on the result of processing the
POST request; almost all of the status codes defined by this
specification might be received in a response to POST (the exceptions
being 206 (Partial Content), 304 (Not Modified), and 416 (Range Not
Satisfiable)).
Given that this language was explicitly added to the updated spec, I doubt the authors intended that the Content-Range header be used with the POST method.
My colleagues and I are implementing a number of RESTful HTTP services, and we're trying to make sure we are a) following the spec, and b) doing the "right" thing where the spec is short of detail.
Here is a particular situation that we have come to and are looking for opinions from the community on:
Suppose you have a resource /People/Bob, and your client is going to update it with a PUT. The server can produce representations for /People/Bob in application/json and text/html. The server can interpret representations for /People/Bob in application/json.
Given this request:
PUT /People/Bob
Content-Type: application/json
Accept: application/xml
{ name: "Still Bob" }
The server can't produce an XML representation, but it can process the incoming JSON. So we know the correct answer is for the server to return status 406.
The question is: should the server have performed the update to /People/Bob?
+1 for Philosophy of REST.
Without detailed knowledge of the HTTP spec, I would simply choose one of the options and document the quandary and the choice.
My preference would be that the server cannot respond as requested, then it should not process any of the request at all.
But that may not work in some scenarios, so you might have to do the opposite.
The question is: should the server have performed the update to /People/Bob?
From the HTTP spec, a 406 means:
The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.
Unless it was a HEAD request, the response SHOULD include an entity containing a list of available entity characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. Depending upon the format and the capabilities of the user agent, selection of the most appropriate choice MAY be performed automatically. However, this specification does not define any standard for such automatic selection.
Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
If the response could be unacceptable, a user agent SHOULD temporarily stop receipt of more data and query the user for a decision on further actions.
That note in the middle about HTTP/1.1 may be your answer. I read it as saying "you may return a 200 in response to the PUT request to /People/Bob when the user agent specifies application/xml in the Accept header, selecting any suitable content-type, and that this outcome may be preferable to returning 406."
Under this scenario, the PUT would succeed on the server, return a 200, but the client would get an application/json representation. The client needs to be able to handle that possibility by making sure that it understands the media type given in the Content-type header, and behaving in a well-defined manner if it doesn't.
But this is always true anyway.
One more thing: you may want to consider not using plain-vanilla media types like application/xml and application/json, but instead define your own custom media types, maybe based on XHTML or JSON. All of the client-server coupling in a RESTful application happens through media types. Without media types rich enough to capture your domain concepts, you're incompletely specifying your REST API.
I would argue 'yes' in theory, but 'no' for real-world application.
I see the logic in not processing if there's an error. Since you return a 406, not a 500, I would know that it's not an error in the data I provided, but rather in the way the result is being presented to me.
That said, some applications won't check for error codes; they will just see that it came back with an error rather than the XML it asked for, and assume the transaction failed.
I assume your not handling application/xml is not an actual problem, but for the purposes of the question - if this is actually being deployed as a real-world service, you'd almost certainly want to be able to have an XML representation, as that's (I suspect) the most common RESTful interaction, and many callers would probably be hard-coded to use XML.
To sum up: if you actually aren't providing application/xml, then I would say, don't perform the update. If you're handling all the standards, but you're planning for the contingency where a user will ask for application/fooSomethingNonStandard, then go ahead and perform the update, but be sure you respond with a 406.
One way out of your conundrum is to have a successful PUT return a 204 (No Content). That way the client's Accept header is irrelevant to the issue of whether the update is performed.
A "RESTful" (or at least "HTTP-embracing") client will know not to update its current "page", and that it will have to do a GET in order to refresh its view of the just-PUT resource. The Accept header on that GET is now, of course, a separate concern from the update atomicity.
I would either succeed and return a 200 using the method Rich suggests above or a 406 and fail. The protocol does not allow for a more nuanced approach mixing 2xx (Success) with 4xx (Error) codes so 4xx can be read to imply NOT Success.