Today, HTTP headers all need to be sent before a single bit of HTTP body is sent to the browser.
This is especially problematic with new technologies such as React 18 Streaming where certain headers, such as caching headers and 103 Early Hints, can be determined with certainty only at the end of the HTTP stream. Ideally these late headers would be sent to the browser just before ending the stream.
Are there efforts from spec working groups or browser vendors to enable headers to be sent during/after the HTTP body?
After doing research, it seems that there is no spec work about this, but I wonder if there is a browser vendor working on this? (Some browser folks are active here on StackOverflow.)
Context: I'm the author of vite-plugin-ssr and react-streaming.
There is a specification for Trailer fields for use with Chunked Encoding (Http 1.1, https://httpwg.org/specs/rfc7230.html#header.trailer).
The HTTP2 spec (which does not support Chunked Encoding) directly allows for a headers frame following the Data frames that contain the http body https://datatracker.ietf.org/doc/html/rfc7540#section-8.1.
Library support may vary as most http libraries attempt to abstract away the differences in the underlying protocols. In Javascript you will be interested in enabling trailing headers in the cross-browser standard fetch API. The MDN docs suggest that support is coming with reference to a trailers field on the Response object: https://developer.mozilla.org/en-US/docs/Web/API/Response.
does having a space in the http post headers result in BAD request??
I see this in one of the requests:
Content-Type "text/xml; c harset=utf-8"
and I get a HTTP/1.1 400 Bad Request
But if the same request is posted with
Content-Type "text/xml; charset=utf-8"
i.e no space in charset it works.
In my implementation I am not doing any validation.
So I am assuming my Jetty server throws a bad request since there is a space in charset??
Am I right or is my interpretation wrong.
Thanks!!
Yes, having a space where you are putting it should result in a bad request.
HTTP 1.1 is a protocol defined by a standard. By referencing the standard documentation, it is possible to determine what is and what isn't a valid request.
You can find the standard for HTTP/1.1 at RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1, and you might specifically want to look at sections 14.17 Content-Type and 3.7 Media Types.
Essentially, by inserting the space into "charset", you are creating an invalid HTTP request because the protocol doesn't understand the "c" and "harset" portions. Those aren't defined as valid text in that context.
Moreover, while the protocol knows what appears to be valid and what doesn't, it isn't intelligent enough to infer how to fix even a simple typo like this. As such, for the server to respond "400 Bad Request" is appropriate and conforming to the protocol standard. For what it's worth, you'll also find the HTTP status codes in the RFC. Status code 400 Bad Request means:
The request could not be understood by the server due to malformed
syntax. The client SHOULD NOT repeat the request without
modifications.
i.e. Don't do that. :)
Does anyone know the usage of "Proxy-agent" response header ?
Proxy-Agent: is a non-standard header - it is not defined by any HTTP RFC. As a result, it should technically be X-Proxy-Agent:. If you want a definition of anything about the way that HTTP works, RFC 2616 is your bible.
Because it is non-standard, there is no definite answer to this question. At a guess, from the name of the header, it would contain the name of the HTTP proxy software that is handling the request on behalf of the User-Agent:. But, because it is non-standard, what it actually means could be something completely different.
What I can tell you is that, because it is non-standard, it is not likely to be pertinent to the way you handle the request - although it likely informs you that there is at least one proxy somewhere along the route between the client and the server.
Is it possible to set the content-length header and also use chunked transfer encoding? and does doing so solve the problem of not knowing the length of the response at the client side when using chunked?
the scenario I'm thinking about is when you have a large file to transfer and there's no problem in determining its size, but it's too large to be buffered completely.
(If you're not using chunked, then the whole response must get buffered first? Right??)
thanks.
No:
"Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding. If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored." (RFC 2616, Section 4.4)
And no, you can use Content-Length and stream; the protocol doesn't constrain how your implementation works.
Well, you can always send a header stating the size of the file.
Something like response.addHeader("File-Size","size of the file");
And ignore the Content-Length header.
The client implementation has to be tweaked to read this value, but hey you can achieve both the things you want :)
You have to use either Content-Length or chunking, but not both.
If you know the length in advance, you can use Content-Length instead of chunking even if you generate the content on the fly and never have it all at once in your buffer.
However, you should not do that if the data is really large because a proxy might not be able to handle it. For large data, chunking is safer.
This headers can be cause of Postman Parse Error:
"Content-Length" and "Transfer-Encoding" can't be present in the response headers together.
Using parametrized ResponseEntity<?> except raw ResponseEntity in controller can fixed the issue.
The question asks:
Is it possible to set the content-length header and also use chunked transfer encoding?
The RFC HTTP/1.1 spec, quoted in Julian's answer, says:
Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding.
There is an important difference between what's possible, and what's allowed by a protocol. It is certainly possible, for example, for you to write your own HTTP/1.1 client which sends malformed messages with both headers. You would be violating the HTTP/1.1 spec in doing so, and so you'd imagine some alarm bells would go off and a bunch of Internet police would burst into your house and say, "Stop, arrest that client!" But that doesn't happen, of course. Your request will get sent to wherever it's going.
OK, so you can send a malformed message. So what? Surely on the receiving end, the server will detect the HTTP/1.1 protocol client-side violation, vanquish your malformed request, and serve you back a stern 400 response telling you that you are due in court the following Monday for violating the protocol. But no, actually, that probably won't happen. Of course, it's beyond the scope of HTTP/1.1 to prescribe what happens to misbehaving clients; i.e. while the HTTP/1.1 protocol is analogous to the "law", there is nothing in HTTP/1.1 analogous to the judicial system.
The best that the HTTP/1.1 protocol can do is dictate how a server must act/respond in the case of receiving such a malformed request. However, it's quite lenient in this case. In particular, the server does not have to reject such malformed requests. In fact, in such a scenario, the rule is:
If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored.
Unfortunately, though, some HTTP servers will violate that part of the HTTP/1.1 protocol and will actually give precedence to the Content-Length header, if both headers are present. This can cause a serious problem, if the message visits two servers in sequence in the same system and they disagree about where one HTTP message ends and the next one starts. It leaves the system vulnerable to HTTP Desync attacks a.k.a. Request Smuggling.
I'm asking in generalities - why would any server not set and return headers and/or status codes? I can't think of a good reason for this. Perhaps I'm overlooking something.
The Status-Code is a required part of a HTTP Response.
By definition, the only reason for a server not to provide a Status-Line is that it is not a HTTP server.
RFC 2616, section 6: Response.
Or said in a slightly less pedant way: if it does this, the server is hopelessly buggy and you should run away from it screaming.
Status codes were introduced in HTTP/1.0 - prior to this, things were much simpler - there were no headers in the request or the response.
A request was simply like this, with no indication of the protocol version
GET /
The response would be all body, with no headers.
So it looks like you are talking to some kind of antique webserver which only speaks HTTP/0.9!