I have to pass a meta-information in my HTTP response so I figured out that I could use the response header, for instance "X-MyData: 123456". Is that safe? I mean, there is a possibility that a client proxy remove this header?
Thanks!
For reference, X- headers are also referred to as x-token in the BNF of RFC 2045, as user-defined ("X-") in section 5 of RFC 2047 and as Experimental headers in section 4.2.2.1 of the News Article Format draft.
Deprecating Use of the "X-" Prefix in Application Protocols (BCP, June 2012):
deprecates the "X-" convention for most application protocols and
makes specific recommendations about how to proceed in a world without
the distinction between standard and non-standard parameters
A client proxy could do anything it wanted, but in general would not strip any headers.
Headers starting with an X- are typically reserved for nonstandard usage (i.e. no future standard will introduce a header starting X-) but a proxy may understand them and choose to modify them as it wants.
It is possible for proxy servers or any intermediate links in the chain to modify your headers, but this usually isn't a problem.
More often than not, specifying custom headers is fine as long as they're unique enough not to conflict with other people's headers and you don't expect anyone else to use yours.
Related
I'm currently trying to optimize http-based data transfer between several applications.
Our current approach, downloading first and then creating the post-request, obviously add extra IO/memory load and latencies, which I'd like to circumvent.
The core question of all:
Is it required to send a "Content-Length" header in HTTP POST requests?
IIRC, HTTP 2616 declares that it's optional, but I'm not sure how applications actually behave at this point.
Depends what you mean by optional. If you mean that you can just omit the header anytime you like then no, it is not optional. The HTTP spec has very specific rules when to use that header. There are different ways of sending the data if you don't know the length. Chunked encoding for example.
4.4 Message Length
The HTTP protocol has supported multipart responses for a long time. I've used them before for APIs with appropriately equipped consumers, but it doesn't appear browser support for them is very good, nor has it improved in the last half-decade. I've had difficulty finding much information on why this might be. I'd love to be able to cut down on HTTP requests by sending all of the assets I know a webapp will need on the initial request, especially for apps that employ client-side frameworks like Backbone.js.
Are there any white-papers, trade articles, failed experiments, or other evidence on why neither browser-makers or web-performance evangelists are paying this long-time HTTP construct any attention?
To be utterly clear, I'm not looking for an opinion, but veritable evidence indicating why this might be. For example, if Mozilla published something about this a few years ago, or there is a closed ticket in the Firefox bug tracker where a lead developer comments about why they won't implement this.
Actually older versions of IE would process multipart application/octet-stream responses and save all the files during a download operation, but this was recently removed (as of IE7, I think) and was specific to downloading only.
I doubt you're going to find the "evidence" you're looking for, because I don't think what you have proposed is in keeping with the "spirit" of the HTTP specification. I will try to explain what I mean by that. The basic paradigm of HTTP is a client-driven request and the server's response to that request. But you seem to be proposing that the server would return arbitrary files with the assumption that the client would know what to do with them.
However, if you were to propose that the client first explicitly requests multiple files, then I would say you could be on to something. The HTTP 1.1 specification does allow the Accept client-request header to indicate multipart support, so this would appear to be how the HTTP designers envisioned this working. Unfortunately the spec is silent about how the client should identify the files it expects to receive, and this is understandable if you look at HTTP in a vacuum, as it is defined, rather than through the lens of browsers and websites. That is an implementation detail that is left up to the client and server to settle. It is a concern which applies to a different layer -- what the content is and how it is consumed, rather than how to request it and transport it.
It is easy to imagine various solutions, of course, but without a standard to refer to, it wouldn't seem to warrant the effort on the part of the browser developers. I could imagine someone like Microsoft (with control over a widely-adopted server and browser) implementing this, but they'd be inventing a spec and people would complain. Apparently we've decided it's better to wait 10 years for the W3C to agree on something...
Directly from W3 org itself (at http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.2):
3.7.2 Multipart Types
MIME provides for a number of "multipart" types -- encapsulations of
one or more entities within a single message-body. All multipart types
share a common syntax, as defined in section 5.1.1 of RFC 2046
[40], and MUST include a boundary parameter as part of the media type
value. The message body is itself a protocol element and MUST
therefore use only CRLF to represent line breaks between body-parts.
Unlike in RFC 2046, the epilogue of any multipart message MUST be
empty; HTTP applications MUST NOT transmit the epilogue (even if the
original multipart contains an epilogue). These restrictions exist in
order to preserve the self-delimiting nature of a multipart message-
body, wherein the "end" of the message-body is indicated by the ending
multipart boundary.
In general, HTTP treats a multipart message-body no differently than
any other media type: strictly as payload. The one exception is the
"multipart/byteranges" type (appendix 19.2) when it appears in a 206
(Partial Content) response, which will be interpreted by some HTTP
caching mechanisms as described in sections 13.5.4 and 14.16. In all
other cases, an HTTP user agent SHOULD follow the same or similar
behavior as a MIME user agent would upon receipt of a multipart type.
The MIME header fields within each body-part of a multipart message-
body do not have any significance to HTTP beyond that defined by their
MIME semantics.
In general, an HTTP user agent SHOULD follow the same or similar
behavior as a MIME user agent would upon receipt of a multipart type.
If an application receives an unrecognized multipart subtype, the
application MUST treat it as being equivalent to "multipart/mixed".
Note: The "multipart/form-data" type has been specifically defined
for carrying form data suitable for processing via the POST
request method, as described in RFC 1867 [15].
I have tried downloading files from different servers, NOT all of them respond with the Content-MD5 field in their headers.
I wanted to know if that it is the standard to HTTP response without the hash of the resource file or not?
thanks
The Content-MD5 header field MAY be generated by an origin server or client to function as an integrity check of the entity-body. Only origin servers or clients MAY generate the Content-MD5 header field; proxies and gateways MUST NOT generate it, as this would defeat its value as an end-to-end integrity check. Any recipient of the entity- body, including gateways and proxies, MAY check that the digest value in this header field matches that of the entity-body as receive
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
As of June 2014:
The Content-MD5 header field has been removed because it was
inconsistently implemented with respect to partial responses.
RFC 7231 - Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content - https://www.rfc-editor.org/rfc/rfc7231 (page 92)
HTTPbis is deprecating that header field (see http://trac.tools.ietf.org/wg/httpbis/trac/ticket/178 for details).
Pure MD5 does not support partial verification and is obsoleted. If you try to use use pure hash functions for anything advanced, eventually you'll meet the following situation:
I don't get it... As soon as a file is ready to finish, it starts all
over again. I also get the message "Verifying File Contents"... What
am I to do???
What if one downloads over and over 20Gb file having no chance to detect mismatch early? One cannot offload downloads to p2p without partial verification supported by hash function.
So nowadays one needs to stick with Merkle trees. Gnutella (both G1 and G2) and DC++ (both NMDC and ADC) use TTH (TIGER Tree Hash) while eDonkey 2k use AICH, but it is alone to use this hash, and it is less elegant. So TTH is the de facto standard, and it would be nice if all file hashes everywhere (even when not strictly required) were TTH by default, but we are not there yet.
DC++ is not based on HTTP, but Gnutella (1 and 2) is, so you can study and/or support those HTTP headers. For instance, Shareaza can intercept downloads from browsers and offload them to p2p using Alt-Location, Content-URN, X-Thex-URI headers.
I'm writing a web service that returns a base64-encoded PDF file, so my plan is to add two headers to the response:
Content-Type: application/pdf
Content-Transfer-Encoding: base64
My question is: Is Content-Transfer-Encoding a valid HTTP header? I think it might only be for MIME. If not, how should I craft my HTTP response to represent the fact that I'm returning a base64-encoded PDF? Thanks.
EDIT:
It looks like HTTP does not support this header. From RFC2616 Section 14:
Note: while the definition of Content-MD5 is exactly the same for HTTP
as in RFC 1864 for MIME entity-bodies, there are several ways in which
the application of Content-MD5 to HTTP entity-bodies differs from its
application to MIME entity-bodies. One is that HTTP, unlike MIME, does
not use Content-Transfer-Encoding, and does use Transfer-Encoding and
Content-Encoding.
Any ideas for what I should set my headers to? Thanks.
EDIT 2
Many of the code samples found in the comments of this PHP reference manual page seem to suggest that it actually is a valid HTTP header:
http://php.net/manual/en/function.header.php
According to RFC 1341 (made obsolete by RFC 2045):
A Content-Transfer-Encoding header field, which can be used to
specify an auxiliary encoding that was applied to the data in order to
allow it to pass through mail transport mechanisms which may have
data or character set limitations.
and later:
Many Content-Types which could usefully be transported via email
are represented, in their "natural" format, as 8-bit character or
binary data. Such data cannot be transmitted over some transport
protocols. For example, RFC 821 restricts mail messages to 7-bit
US-ASCII data with 1000 character lines.
It is necessary, therefore, to define a standard mechanism for
re-encoding such data into a 7-bit short-line format. (...) The
Content-Transfer-Encoding field is used to indicate the type of
transformation that has been used in order to represent the body
in an acceptable manner for transport.
Since you have a webservice, which has nothing in common with emails, you shouldn't use this header.
You can use Content-Encoding header which indicates that transferred data has been compressed (gzip value).
I think that in your case
Content-Type: application/pdf
is enough. Additionally, you can set Content-Length header, but in my opinion, if you are building webservice (it's not http server / proxy server) Content-Type is enough. Please bear in mind that some specific headers (e.g. Transfer-Encoding) if not used appropriately, may cause unexpected communication issues, so if you are not 100% sure about usage of some header - if you really need it or not - just don't use it.
Notes in rfc2616 section 14.15 are explicit: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
"Note: while the definition of Content-MD5 is exactly the same for
HTTP as in RFC 1864 for MIME entity-bodies, there are several ways
in which the application of Content-MD5 to HTTP entity-bodies
differs from its application to MIME entity-bodies. One is that
HTTP, unlike MIME, does not use Content-Transfer-Encoding, and
does use Transfer-Encoding and Content-Encoding. Another is that
HTTP more frequently uses binary content types than MIME, so it is
worth noting that, in such cases, the byte order used to compute
the digest is the transmission byte order defined for the type.
Lastly, HTTP allows transmission of text types with any of several
line break conventions and not just the canonical form using CRLF."
As been answered before and also here, a valid Content-Transfer-Encoding HTTP response header does not exist. Also the known headers Content-Encoding and Transfer-Encoding have no appropriate value to express a Base64 encoded response body.
Starting from here, no client would expect a response declared as application/pdf to be encoded as Base64! If you wand to do so, better use a different content type like:
Content-Type: application/pdf+base64
In this case, a client would know some Base64 encoded data is coming (the basic subtype is the suffix after the plus sign) and has a hint there is PDF in there.
Even this is a little hacky (+base64 is no official media type suffix) but at least would somehow meet some standards. Better use a custom content type than misusing standard HTTP headers!
Of course no browser would be able to directly open such a response anyway. Maybe your project should consider creating another endpoint offering a binary PDF response and marking this one deprecated.
If a clients sends data in an unsupported media type to a HTTP server, the server answers with status "415 unsupported media type". But how to tell the client what media types are supported? Is there a standard or at least a recommended way to do so? Or would it just be written to the response body as text?
There is no specification at all for what to do in this case, so expect implementations to be all over the place. (What would be sensible would be if the server's response included something like an Accept: header since that has pretty much the right semantics, if currently in the wrong direction.)
I believe you can do this with the OPTIONS Http verb.
Also the status code of 300 Multiple Choices could be used if your scenario fits a certain use case. If they send a request with an Accept header of application/xml and you only support text/plain and that representation lives at a distinct URL then you can respond with a 300 and in the Location header the URL of that representation. I realize this might not exactly fit your question, but it's another possible option.
And from the HTTP Spec:
10.4.7 406 Not Acceptable
The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.
Unless it was a HEAD request, the response SHOULD include an entity containing a list of available entity characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. Depending upon the format and the capabilities of the user agent, selection of the most appropriate choice MAY be performed automatically. However, this specification does not define any standard for such automatic selection.
Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
tl;dr;
Edited the generated proxy class to inherit from Microsoft.Web.Services3.WebServicesClientProtocol**.
I came across this question when troubleshooting this error, so I thought I would help the next person who might come through here, although not sure if it answers the question as stated. I ran into this error when at some point I had to take over an existing solution which was utilizing WSE and MTOM encoding. It was a windows client calling a web service.
To the point, the client was calling the web service where it would throw that error.
Something that contributed to resolving that error for me was to check the web service proxy class that apparently is generated by default to inherit from System.Web.Services.Protocols.SoapHttpClientProtocol.
Essentially that meant that it didn't actually use WSE3.
Anyhow I manually edited the proxy and changed it to inherit from Microsoft.Web.Services3.WebServicesClientProtocol.
BTW, to see the generated proxy class in VS click on the web reference and then click the 'Show All Files' toolbar button. The reference.cs is da place of joy!
Hope it helps.
In his book "HTTP Developer's Handbook" on page 81 Chris Shiflett explains what a 415 means, and then he says, "The media type used in the content of the HTTP response should be indicated in the Content-Type entity header."
1) So is Content-Type a possible answer? It would presumably be a comma-separated list of accepted content types. The obvious problem with this possibility is that Content-Type is an entity header not a response header.
2) Or is this a typo in the book? Did he really mean to say "the HTTP request"?