Have read in the blog that in HTTP/2 the trailer data is present in HEADER frame, and I am confused what exactly is the difference when we set headers to gRPC response vs setting trailer metadata to it.
Any help would be appreciated.
In gRPC's protocol definition, trailer metadata is the HEADER frames sent marking the end of an RPC. There is always trailer metadata sent by server even if the call produces an immediate error (e.g., UNIMPLEMENTED), which is Trailers-Only .
Related
I'm working with an HTTP request tool (similar to cURL) and having an issue with the server response. Either that or my understanding of the RFC for HTTP 1.1 and chunked data.
What I'm seeing is chunked data should be in this format:
4\r\n
Wiki\r\n
5\r\n
pedia\r\n
e\r\n
in\r\n\r\nchunks.\r\n
0\r\n
\r\n
what I'm actually seeing is the following:
4\r\n
Wiki\r\n
5\r\n
pedia\r\n
e\r\n
in\r\n\r\nchunks.\r\n
0
In other words, the few servers I've tested with send no more data after the 0.. not CRLF, much less CRLFCRLF.
How are we supposed to know it's the end of the chunked data without the proper format of the chunked tags? Timeouts happen looking for the CRLFs after the 0, and that's no sufficient.
Yes, it violates standard. But we want to be compatible with all possible http servers and clients, so we have to understand a way how it can be violated.
Chunked is used often in a way of content streaming over http 1.1 protocol. Standard ask to end content with additional CRLF. So we can see the following pseudo code:
def stream(endpoint)
Socket.open(endpoint) do |socket|
sleep 10
more_data do |data|
print data.length.to_s(16)
print data
print "CRLF"
end
end
print "CRLF"
end
But the right code is the following:
def stream(endpoint)
Socket.open(endpoint) do |socket|
sleep 10
more_data do |data|
print data.length.to_s(16)
print data
print "CRLF"
end
end
ensure
print "CRLF"
end
It means that after input socket interruption of any other exception wrong version of method won't be able to print additional "CRLF" to output socket.
How are we supposed to know it's the end of the chunked data without
the proper format of the chunked tags? Timeouts happen looking for the
CRLFs after the 0, and that's no sufficient.
Many implementations ignores this violation because they don't need to know the size of content. They just tries to receive as much data as possible before socket will be closed.
Use Content-Length, definitely whenever I know it; for file download, checking the filesize is insignificant in terms of resources. For chunked transfer we do not scan the message body for a CRLF pair. It first reads the specified number of bytes, and then reads two more bytes to confirm that they are CR and LF. If they're not, the message body is ill-formed, and either the size was specified improperly or the data was otherwise corrupted.
For more information read RCF, which says
A server using chunked transfer-coding in a response MUST NOT use the
trailer for any header fields unless at least one of the following is
true:
a)the request included a TE header field that indicates "trailers" is
acceptable in the transfer-coding of the response, as described in
section 14.39; or,
b)the server is the origin server for the response, the trailer fields
consist entirely of optional metadata, and the recipient could use the
message (in a manner acceptable to the origin server) without
receiving this metadata. In other words, the origin server is willing
to accept the possibility that the trailer fields might be silently
discarded along the path to the client.
Way to Determine Message Body Length:
If header has Transfer-Encoding and the chunked transfer is final encoding, then message body length is determined by reading and decoding the chunked data until the transfer coding indicates the data is complete.
If header has Transfer-Encoding and the chunked transfer is not final encoding, then message body length is determined by reading the connection until it is closed by the server.
If header has Transfer-Encoding in request and the chunked transfer is not final encoding, then message body length cannot be determined reliably; the server MUST respond with the 400 (Bad Request) status code and then close the connection.
If a message is received with both a Transfer-Encoding and Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request response splitting and ought to be handled as an error. A sender MUST remove the received Content-Length field prior to forwarding such a message downstream.
I am testing the Last.fm api using row socket interface.
Now i noticed that some of api's http response have not contain a field Content-Length .
But I want to know is there a way to ask the server presenting it?
Because i can't take good care of this in my program elegantly.
Quoth the RFC:
7.2.2 Length
When an Entity-Body is included with a message, the length of that body may be determined in one of two ways. If a Content-Length header field is present, its value in bytes represents the length of the Entity-Body. Otherwise, the body length is determined by the closing of the connection by the server.
The right RFC to look at is RFC 7230 (Section 3.3.2).
And no, in HTTP/1.1 a client has to be able to process chunked encoding (which would be the only legitimate reason not to provide a Content-Length header field).
I know that the server can send the data to the client only with the transfer codings sfecified by the "TE" header of the request (or only chunked if no "TE" header is pressent and the client is HTTP/1.1). But, how the client knows what transfer codings are accepted by the server ? Because I understand that transfer codings can be used both ways.
For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always acceptable, even if not listed in the TE (transfer encoding) request header field, and when used with other transfer mechanisms, should always be applied last to the transferred data and never more than one time. This transfer coding method also allows additional entity header fields to be sent after the last chunk if the client specified the "trailers" parameter as an argument of the TE field. The origin server of the response can also decide to send additional entity trailers even if the client did not specify the "trailers" option in the TE request field, but only if the metadata is optional (i.e. the client can use the received entity without them). Whenever the trailers are used, the server should list their names in the Trailer header field.
Source: http://en.wikipedia.org/wiki/Chunked_transfer_encoding
The client does not know which one can be used, it works under the assumption that its either chuncked or get a proper header by the webserver.
I have a doubt regarding sending of mime attachments over HTTP:
in http specs the following is quoted :
“C.4 No Content-Transfer-Encoding: HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC 1521. Proxies and gateways from MIME-compliant protocols to HTTP must remove any non-identity CTE ("quoted-printable" or "base64") encoding prior to delivering the response message to an HTTP client. Proxies and gateways from HTTP to MIME-compliant protocols are responsible for ensuring that the message is in the correct format and encoding for safe transport on that protocol, where "safe transport" is defined by the limitations of the protocol being used. Such a proxy or gateway should label the data with an appropriate Content-Transfer-Encoding if doing so will improve the likelihood of safe transport over the destination protocol.”
Does this mean that specifically for sending MIME attachments only over http, we shouldn't specify content-transfer-encoding as quoted-printable or base64 ?
Also, what is the behavior of conetent-transfer-encoding when i send such attachments over other transports like JMS, or over Mail? For example in a SOAP over JMS message?
Also the found following relevant from RFC 4130 :
“5.2. Unused MIME Headers and Operations
5.2.1. Content-Transfer-Encoding Not Used in HTTP Transport
HTTP can handle binary data and so there is no need to use the content transfer encodings of MIME [1]. This difference is discussed in [3], Section 19.4.5. However, a content transfer encoding value of binary or 8-bit is permissible but not required. The absence of this header MUST NOT result in transaction failure. Content transfer encoding of MIME body parts within the AS2 message body is also allowed.”
So i am basically thoroughly confused over the behavior of mime attachments specific to the HTTP protocol, and would like to get its behavior clarified.
HTTP is not MIME, it just borrows from the MIME message format. Payloads in HTTP are binary, and there simply is no Content-Transfer-Encoding header field. You can specify it, but it has zero effect and keeps distracting people looking at wire traces.
Can some experts explain the differences between the two? Is it true that chunked is a streaming protocol and multipart is not? What is the benefit of using multipart?
More intuitively,
Chunking is a way to send a single message from server to client, where the server doesn't have to wait for the entire response to be generated but can send pieces (chunks) as and when it is available. Now this happens at data transfer level and is oblivious to the client. Appropriately it is a 'Transfer-Encoding' type.
While Multi-part happens at the application level and is interpreted at the application logic level. Here the server is telling client that the content , even if it is one response body it has different logical parts and can be parsed accordingly. Again appropriately, this is a setting at 'Content-Type' as the clients ought to know it.
Given that transfer can be chunked independent of the content types, a multi-part http message can be transferred using chunked encoding by the server if need be.
Neither is a protocol. HTTP is the protocol. In fact, the P in HTTP stands for Protocol.
You can read more on chunked and multipart under Hypertext Transfer Protocol 1.1
Chunked is a transfer coding found in section 3.6 Transfer Codings.
Multipart is a media type found in section 3.7.2 Multipart Types a subsection of 3.7 Media Types.
Chunked also affects other aspects of the protocol such as the content-length as specified under 4.4 as chunked must be used when message length cannot be predetermined (mainly when delivering dynamic content).
From 14.41 (Transfer-Encoding header field)
The Transfer-Encoding general-header field indicates what (if any)
type of transformation has been applied to the message body in order
to safely transfer it between the sender and the recipient. This
differs from the content-coding in that the transfer-coding is a
property of the message, not of the entity.
Put more simply, chunking is how you transfer a block of data, while multipart is the shape of the data.