Until today I thought that HTTP is a pure text-based protocol.
Now I go to Telnet in Windows and type:
telnet reactos.org 80
GET /favicon.ico HTTP/1.0
press Enter twice and see following:
This is definitely not Base64.
Does HTTP really support binary transfer? I couldn't find any document about that.
The HTTP protocol is text-based, but the message body is defined by whatever Content-Type is specified in the headers. In your case the Content-Type is image/x-icon, which is a binary format.
Also note that encoding and the protocol being text-based are unrelated. Base-64 is only required when the encoding is ASCII. The default Content-Encoding is identity (no compression, no modification). With another Content-Encoding, you would get binary data in the body even for text Content-Type:s as they would be compressed.
Related
I read in the IETF AS2 spec here: https://www.ietf.org/rfc/rfc4130.txt
That "EDI envelope headers are encrypted."
But I'm trying to determine if the MIME headers are encrypted. If I turn on WireShark, will I be able to read the MIME headers?
Reason is that in BizTalk 2016 we seem to be missing one of the headers, so I need to see in raw format if the trading partner sent it or not. The field I'm looking for is X-Cyclone-True-Receiver sent by an Axway software program.
(Related question: https://social.msdn.microsoft.com/Forums/en-US/543147b1-d38d-4a74-89b5-085d52e353bf/as2-promoted-field-xcyclonetruereceiver?forum=biztalkediandas2)
Reading this Structure of an AS2 Message documentation. I would say, yes, the MIME headers are encrypted if you set the encryption option.
e.g.
Encrypted, No compression, No signature = HTTP, AS2, MIME Header CMS-PKCS7 MIME Encryption EDI/XML Payload (encrypted)
The HTTP and AS2 headers are not encrypted unless you go via HTTPS protocol
I have a question on usage of Content-Encoding and Transfer-Encoding:
Please let me know if my below understanding is right:
Client in its request can specify which encoding types it is willing to accept using accept-encoding header. So, if Server wishes to encode the message before transmission, eg. gzip, it can zip the entity (content) and add content-encoding: gzip and send across the HTTP response. On reception, client can receive and decompress and parse the entity.
In case of Transfer Encoding, Client may specify what kind of encoding it is willing to accept and perform its action on fly. i.e. if Client sends a TE: gzip; q=1, it means that if Server wishes, it can send a 200 OK with Transfer-Encoding: gzip and as it tries sending the stream, it can compress and send across, and client upon receiving the content, can decompress on fly and perform its parsing.
Is my understanding right here? Please comment.
Also, what is the basic advantage of compressing the entity on fly vs compressing the entity first and then transmitting it across? Is transfer-encoding valid only for chunked responses as we do not know the size of the entity before transmission?
The difference really is not about on-the-fly or not -- Content-Encoding can be both pre-computed and on the fly.
The differences are:
Transfer Encoding is hop-by-hop, not end-to-end
Transfer Encodings other than "chunked" (sadly) aren't implemented in practice
Transfer Encoding is on the message layer, Content Encoding on the payload layer
Using Content Encoding affects entity tags etc.
See http://greenbytes.de/tech/webdav/rfc7230.html#transfer.codings and http://greenbytes.de/tech/webdav/rfc7231.html#data.encoding.
I have a doubt regarding sending of mime attachments over HTTP:
in http specs the following is quoted :
“C.4 No Content-Transfer-Encoding: HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC 1521. Proxies and gateways from MIME-compliant protocols to HTTP must remove any non-identity CTE ("quoted-printable" or "base64") encoding prior to delivering the response message to an HTTP client. Proxies and gateways from HTTP to MIME-compliant protocols are responsible for ensuring that the message is in the correct format and encoding for safe transport on that protocol, where "safe transport" is defined by the limitations of the protocol being used. Such a proxy or gateway should label the data with an appropriate Content-Transfer-Encoding if doing so will improve the likelihood of safe transport over the destination protocol.”
Does this mean that specifically for sending MIME attachments only over http, we shouldn't specify content-transfer-encoding as quoted-printable or base64 ?
Also, what is the behavior of conetent-transfer-encoding when i send such attachments over other transports like JMS, or over Mail? For example in a SOAP over JMS message?
Also the found following relevant from RFC 4130 :
“5.2. Unused MIME Headers and Operations
5.2.1. Content-Transfer-Encoding Not Used in HTTP Transport
HTTP can handle binary data and so there is no need to use the content transfer encodings of MIME [1]. This difference is discussed in [3], Section 19.4.5. However, a content transfer encoding value of binary or 8-bit is permissible but not required. The absence of this header MUST NOT result in transaction failure. Content transfer encoding of MIME body parts within the AS2 message body is also allowed.”
So i am basically thoroughly confused over the behavior of mime attachments specific to the HTTP protocol, and would like to get its behavior clarified.
HTTP is not MIME, it just borrows from the MIME message format. Payloads in HTTP are binary, and there simply is no Content-Transfer-Encoding header field. You can specify it, but it has zero effect and keeps distracting people looking at wire traces.
I want to avoid ever getting chunked encoded HTTP server response from (conforming) HTTP server. I am reading RFC 2616 section "14.39 TE" and it seems to me that I could avoid it by specifying TE: chunked;q=0. If I cannot avoid the chunked encoding, I want do avoid the trailers. Will specifying TE: trailers;q=0 work?
From rfc2616 - Hypertext Transfer Protocol -- HTTP/1.1 in section 3.6.1 Chunked Transfer Coding:
All HTTP/1.1 applications MUST be able to receive and decode the
"chunked" transfer-coding, and MUST ignore chunk-extension extensions
they do not understand.
This is still the case in the updated RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing in section 4.1. Chunked Transfer Coding although in a slightly different wording:
A recipient MUST be able to parse and decode the chunked transfer
coding.
So if you want to be conform to HTTP/1.1, you will have to accept chunked encoding.
##Update##
As for the trailers: I think if you don't send a TE header field in your request, a conforming server shouldn't send you any trailers. If it still sends trailers you are probably save to ignore them (again section 3.6.1):
A server using chunked transfer-coding in a response MUST NOT use the
trailer for any header fields unless at least one of the following is
true:
a) the request included a TE header field that indicates "trailers" is
acceptable in the transfer-coding of the response, as described in
section 14.39; or,
b) the server is the origin server for the response, the trailer
fields consist entirely of optional metadata, and the recipient
could use the message (in a manner acceptable to the origin server)
without receiving this metadata. In other words, the origin server
is willing to accept the possibility that the trailer fields might
be silently discarded along the path to the client.
Is it possible to set the content-length header and also use chunked transfer encoding? and does doing so solve the problem of not knowing the length of the response at the client side when using chunked?
the scenario I'm thinking about is when you have a large file to transfer and there's no problem in determining its size, but it's too large to be buffered completely.
(If you're not using chunked, then the whole response must get buffered first? Right??)
thanks.
No:
"Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding. If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored." (RFC 2616, Section 4.4)
And no, you can use Content-Length and stream; the protocol doesn't constrain how your implementation works.
Well, you can always send a header stating the size of the file.
Something like response.addHeader("File-Size","size of the file");
And ignore the Content-Length header.
The client implementation has to be tweaked to read this value, but hey you can achieve both the things you want :)
You have to use either Content-Length or chunking, but not both.
If you know the length in advance, you can use Content-Length instead of chunking even if you generate the content on the fly and never have it all at once in your buffer.
However, you should not do that if the data is really large because a proxy might not be able to handle it. For large data, chunking is safer.
This headers can be cause of Postman Parse Error:
"Content-Length" and "Transfer-Encoding" can't be present in the response headers together.
Using parametrized ResponseEntity<?> except raw ResponseEntity in controller can fixed the issue.
The question asks:
Is it possible to set the content-length header and also use chunked transfer encoding?
The RFC HTTP/1.1 spec, quoted in Julian's answer, says:
Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding.
There is an important difference between what's possible, and what's allowed by a protocol. It is certainly possible, for example, for you to write your own HTTP/1.1 client which sends malformed messages with both headers. You would be violating the HTTP/1.1 spec in doing so, and so you'd imagine some alarm bells would go off and a bunch of Internet police would burst into your house and say, "Stop, arrest that client!" But that doesn't happen, of course. Your request will get sent to wherever it's going.
OK, so you can send a malformed message. So what? Surely on the receiving end, the server will detect the HTTP/1.1 protocol client-side violation, vanquish your malformed request, and serve you back a stern 400 response telling you that you are due in court the following Monday for violating the protocol. But no, actually, that probably won't happen. Of course, it's beyond the scope of HTTP/1.1 to prescribe what happens to misbehaving clients; i.e. while the HTTP/1.1 protocol is analogous to the "law", there is nothing in HTTP/1.1 analogous to the judicial system.
The best that the HTTP/1.1 protocol can do is dictate how a server must act/respond in the case of receiving such a malformed request. However, it's quite lenient in this case. In particular, the server does not have to reject such malformed requests. In fact, in such a scenario, the rule is:
If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored.
Unfortunately, though, some HTTP servers will violate that part of the HTTP/1.1 protocol and will actually give precedence to the Content-Length header, if both headers are present. This can cause a serious problem, if the message visits two servers in sequence in the same system and they disagree about where one HTTP message ends and the next one starts. It leaves the system vulnerable to HTTP Desync attacks a.k.a. Request Smuggling.