I'm writing a HTTP client, and wondering if HTTP responses are required to have any headers?
So is the following a valid HTTP response? (where \r\n == CRLF). How should a client handle it?
HTTP/1.1 200 OK\r\n
\r\n
The related questions Http response with no http header and What HTTP response headers are required are similar, but I think they are subtly-but-crucially different. They are both from the point of view of a server generating headers, rather than a parser parsing a response.
For example, a detailed answer to the current question could include how the client should interpret the above minimal response: whether it should accept the 200 response or abort, and what it should do with the underlying connection in terms of keeping it open or closing it.
The syntax of the message is defined in RFC 7231 (see https://greenbytes.de/tech/webdav/rfc7230.html#http.message), and that says that header fields are syntactically optional. So a message parser should accept that.
Related
does having a space in the http post headers result in BAD request??
I see this in one of the requests:
Content-Type "text/xml; c harset=utf-8"
and I get a HTTP/1.1 400 Bad Request
But if the same request is posted with
Content-Type "text/xml; charset=utf-8"
i.e no space in charset it works.
In my implementation I am not doing any validation.
So I am assuming my Jetty server throws a bad request since there is a space in charset??
Am I right or is my interpretation wrong.
Thanks!!
Yes, having a space where you are putting it should result in a bad request.
HTTP 1.1 is a protocol defined by a standard. By referencing the standard documentation, it is possible to determine what is and what isn't a valid request.
You can find the standard for HTTP/1.1 at RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1, and you might specifically want to look at sections 14.17 Content-Type and 3.7 Media Types.
Essentially, by inserting the space into "charset", you are creating an invalid HTTP request because the protocol doesn't understand the "c" and "harset" portions. Those aren't defined as valid text in that context.
Moreover, while the protocol knows what appears to be valid and what doesn't, it isn't intelligent enough to infer how to fix even a simple typo like this. As such, for the server to respond "400 Bad Request" is appropriate and conforming to the protocol standard. For what it's worth, you'll also find the HTTP status codes in the RFC. Status code 400 Bad Request means:
The request could not be understood by the server due to malformed
syntax. The client SHOULD NOT repeat the request without
modifications.
i.e. Don't do that. :)
I want to avoid ever getting chunked encoded HTTP server response from (conforming) HTTP server. I am reading RFC 2616 section "14.39 TE" and it seems to me that I could avoid it by specifying TE: chunked;q=0. If I cannot avoid the chunked encoding, I want do avoid the trailers. Will specifying TE: trailers;q=0 work?
From rfc2616 - Hypertext Transfer Protocol -- HTTP/1.1 in section 3.6.1 Chunked Transfer Coding:
All HTTP/1.1 applications MUST be able to receive and decode the
"chunked" transfer-coding, and MUST ignore chunk-extension extensions
they do not understand.
This is still the case in the updated RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing in section 4.1. Chunked Transfer Coding although in a slightly different wording:
A recipient MUST be able to parse and decode the chunked transfer
coding.
So if you want to be conform to HTTP/1.1, you will have to accept chunked encoding.
##Update##
As for the trailers: I think if you don't send a TE header field in your request, a conforming server shouldn't send you any trailers. If it still sends trailers you are probably save to ignore them (again section 3.6.1):
A server using chunked transfer-coding in a response MUST NOT use the
trailer for any header fields unless at least one of the following is
true:
a) the request included a TE header field that indicates "trailers" is
acceptable in the transfer-coding of the response, as described in
section 14.39; or,
b) the server is the origin server for the response, the trailer
fields consist entirely of optional metadata, and the recipient
could use the message (in a manner acceptable to the origin server)
without receiving this metadata. In other words, the origin server
is willing to accept the possibility that the trailer fields might
be silently discarded along the path to the client.
Is it possible to set the content-length header and also use chunked transfer encoding? and does doing so solve the problem of not knowing the length of the response at the client side when using chunked?
the scenario I'm thinking about is when you have a large file to transfer and there's no problem in determining its size, but it's too large to be buffered completely.
(If you're not using chunked, then the whole response must get buffered first? Right??)
thanks.
No:
"Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding. If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored." (RFC 2616, Section 4.4)
And no, you can use Content-Length and stream; the protocol doesn't constrain how your implementation works.
Well, you can always send a header stating the size of the file.
Something like response.addHeader("File-Size","size of the file");
And ignore the Content-Length header.
The client implementation has to be tweaked to read this value, but hey you can achieve both the things you want :)
You have to use either Content-Length or chunking, but not both.
If you know the length in advance, you can use Content-Length instead of chunking even if you generate the content on the fly and never have it all at once in your buffer.
However, you should not do that if the data is really large because a proxy might not be able to handle it. For large data, chunking is safer.
This headers can be cause of Postman Parse Error:
"Content-Length" and "Transfer-Encoding" can't be present in the response headers together.
Using parametrized ResponseEntity<?> except raw ResponseEntity in controller can fixed the issue.
The question asks:
Is it possible to set the content-length header and also use chunked transfer encoding?
The RFC HTTP/1.1 spec, quoted in Julian's answer, says:
Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding.
There is an important difference between what's possible, and what's allowed by a protocol. It is certainly possible, for example, for you to write your own HTTP/1.1 client which sends malformed messages with both headers. You would be violating the HTTP/1.1 spec in doing so, and so you'd imagine some alarm bells would go off and a bunch of Internet police would burst into your house and say, "Stop, arrest that client!" But that doesn't happen, of course. Your request will get sent to wherever it's going.
OK, so you can send a malformed message. So what? Surely on the receiving end, the server will detect the HTTP/1.1 protocol client-side violation, vanquish your malformed request, and serve you back a stern 400 response telling you that you are due in court the following Monday for violating the protocol. But no, actually, that probably won't happen. Of course, it's beyond the scope of HTTP/1.1 to prescribe what happens to misbehaving clients; i.e. while the HTTP/1.1 protocol is analogous to the "law", there is nothing in HTTP/1.1 analogous to the judicial system.
The best that the HTTP/1.1 protocol can do is dictate how a server must act/respond in the case of receiving such a malformed request. However, it's quite lenient in this case. In particular, the server does not have to reject such malformed requests. In fact, in such a scenario, the rule is:
If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored.
Unfortunately, though, some HTTP servers will violate that part of the HTTP/1.1 protocol and will actually give precedence to the Content-Length header, if both headers are present. This can cause a serious problem, if the message visits two servers in sequence in the same system and they disagree about where one HTTP message ends and the next one starts. It leaves the system vulnerable to HTTP Desync attacks a.k.a. Request Smuggling.
I can't tell from RFC 2616 whether an HTTP client should accept a 204 No Content response that includes Content-Length: 0 or Transfer-Encoding: chunked headers. These headers seem to break some HTTP clients and proxies, which apparently attempt to read the empty response body, but the spec reads:
Any response message which "MUST NOT" include a message-body (such as
the 1xx, 204, and 304 responses and
any response to a HEAD request) is
always terminated by the first empty
line after the header fields,
regardless of the entity-header fields
present in the message.
To me "regardless of the entity-header fields" implies that clients should tolerate this state of affairs. The Erlang HTTP library chose this interpretation. However, lighthttpd and IBM chose the opposite interpretation -- that the server should not include these headers for responses that are prohibited from having bodies.
So should the web application remove those headers from the response, or should the network infrastructure and clients tolerate those headers on 204 No Content, 304 Not Modified, etc. ?
According to the RFC7230 (please see https://www.rfc-editor.org/rfc/rfc7230#section-3.3.1) at the end of Page 29 it states:
A server MUST NOT send a Transfer-Encoding header field in any
response with a status code of 1xx (Informational) or 204 (No
Content).
Therefore the web application should not send those headers with the response.
I can't tell you what is correct, in that I don't know the HTTP protocol in that detail.
However, you got to ask yourself:
Can I change my application to remove those headers?
or can I change the consumers of my application to ignore those headers?
In my opinion, the application should not send those headers in that case.
I have an HTTP server that returns large bodies in response to POST requests (it is a SOAP server). These bodies are "streamed" via chunking. If I encounter an error midway through streaming the response how can I report that error to the client and still keep the connection open? The implementation uses a proprietary HTTP/SOAP stack so I am interested in answers at the HTTP protocol level.
Once the server has sent the status line (the very first line of the response) to the client, you can't change the status code of the response anymore. Many servers delay sending the response by buffering it internally until the buffer is full. While the buffer is filling up, you can still change your mind about the response.
If your client has access to the response headers, you could use the fact that chunked encoding allows the server to add a trailer with headers after the chunked-encoded body. So, your server, having encountered the error, could gracefully stop sending the body, and then send a trailer that sets some header to some value. Your client would then interpret the presence of this header as a sign that an error happened.
Also keep in mind that chunked responses can contain "footers" which are just like HTTP headers. After failing, you can send a footer such as:
X-RealStatus: 500 Some bad stuff happened
Or if you succeed:
X-RealStatus: 200 OK
you can change the status code as long as response.iscommitted() returns false.
(fot HttpServletResponse in java, im sure there exists an equivalent in other languages)