I'm currently trying to optimize http-based data transfer between several applications.
Our current approach, downloading first and then creating the post-request, obviously add extra IO/memory load and latencies, which I'd like to circumvent.
The core question of all:
Is it required to send a "Content-Length" header in HTTP POST requests?
IIRC, HTTP 2616 declares that it's optional, but I'm not sure how applications actually behave at this point.
Depends what you mean by optional. If you mean that you can just omit the header anytime you like then no, it is not optional. The HTTP spec has very specific rules when to use that header. There are different ways of sending the data if you don't know the length. Chunked encoding for example.
4.4 Message Length
Related
I'm writing a web service that returns a base64-encoded PDF file, so my plan is to add two headers to the response:
Content-Type: application/pdf
Content-Transfer-Encoding: base64
My question is: Is Content-Transfer-Encoding a valid HTTP header? I think it might only be for MIME. If not, how should I craft my HTTP response to represent the fact that I'm returning a base64-encoded PDF? Thanks.
EDIT:
It looks like HTTP does not support this header. From RFC2616 Section 14:
Note: while the definition of Content-MD5 is exactly the same for HTTP
as in RFC 1864 for MIME entity-bodies, there are several ways in which
the application of Content-MD5 to HTTP entity-bodies differs from its
application to MIME entity-bodies. One is that HTTP, unlike MIME, does
not use Content-Transfer-Encoding, and does use Transfer-Encoding and
Content-Encoding.
Any ideas for what I should set my headers to? Thanks.
EDIT 2
Many of the code samples found in the comments of this PHP reference manual page seem to suggest that it actually is a valid HTTP header:
http://php.net/manual/en/function.header.php
According to RFC 1341 (made obsolete by RFC 2045):
A Content-Transfer-Encoding header field, which can be used to
specify an auxiliary encoding that was applied to the data in order to
allow it to pass through mail transport mechanisms which may have
data or character set limitations.
and later:
Many Content-Types which could usefully be transported via email
are represented, in their "natural" format, as 8-bit character or
binary data. Such data cannot be transmitted over some transport
protocols. For example, RFC 821 restricts mail messages to 7-bit
US-ASCII data with 1000 character lines.
It is necessary, therefore, to define a standard mechanism for
re-encoding such data into a 7-bit short-line format. (...) The
Content-Transfer-Encoding field is used to indicate the type of
transformation that has been used in order to represent the body
in an acceptable manner for transport.
Since you have a webservice, which has nothing in common with emails, you shouldn't use this header.
You can use Content-Encoding header which indicates that transferred data has been compressed (gzip value).
I think that in your case
Content-Type: application/pdf
is enough. Additionally, you can set Content-Length header, but in my opinion, if you are building webservice (it's not http server / proxy server) Content-Type is enough. Please bear in mind that some specific headers (e.g. Transfer-Encoding) if not used appropriately, may cause unexpected communication issues, so if you are not 100% sure about usage of some header - if you really need it or not - just don't use it.
Notes in rfc2616 section 14.15 are explicit: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
"Note: while the definition of Content-MD5 is exactly the same for
HTTP as in RFC 1864 for MIME entity-bodies, there are several ways
in which the application of Content-MD5 to HTTP entity-bodies
differs from its application to MIME entity-bodies. One is that
HTTP, unlike MIME, does not use Content-Transfer-Encoding, and
does use Transfer-Encoding and Content-Encoding. Another is that
HTTP more frequently uses binary content types than MIME, so it is
worth noting that, in such cases, the byte order used to compute
the digest is the transmission byte order defined for the type.
Lastly, HTTP allows transmission of text types with any of several
line break conventions and not just the canonical form using CRLF."
As been answered before and also here, a valid Content-Transfer-Encoding HTTP response header does not exist. Also the known headers Content-Encoding and Transfer-Encoding have no appropriate value to express a Base64 encoded response body.
Starting from here, no client would expect a response declared as application/pdf to be encoded as Base64! If you wand to do so, better use a different content type like:
Content-Type: application/pdf+base64
In this case, a client would know some Base64 encoded data is coming (the basic subtype is the suffix after the plus sign) and has a hint there is PDF in there.
Even this is a little hacky (+base64 is no official media type suffix) but at least would somehow meet some standards. Better use a custom content type than misusing standard HTTP headers!
Of course no browser would be able to directly open such a response anyway. Maybe your project should consider creating another endpoint offering a binary PDF response and marking this one deprecated.
I'm trying to write a program to match HTTP requests with their corresponding responses. Seems that everything is working well for most of the scenarios (when the transfer is perfectly ordered and even when its not, by using TCP sequence numbers).
The only problem I found is for when I have pipelined requests. After that, I get several responses but I don't know which packets are the answer to a specific request and which are not. I read in another post that the responses will come sequentially and combining this property with information on the Content-Length field seems to be a solution. The problem is that Content-length is not a mandatory field, so I'm not sure if I can always rely on that.
Does anyone know how the web-browsers that support this feature (btw, not most of them do) actually do it?
The information about the bodies length has to be present in the headers. It's just not always in 'content-length'. In order to work it all out you will have to study the relevant RFC 2616. Most notably section 4.4 deals with the different headers
Some more relevant rules from the RFC 2616:
When pipelining:
A server MUST send its responses to those requests in the same order that the requests were received.
From 9.2
If no response body is included, the response MUST include a Content-Length field with a field-value of "0".
From 10.2.7 206 Partial Content
The response MUST include .... Either a Content-Range header field ... or a multipart/byteranges
Content-Type including Content-Range fields for each part.
From 14.13 Content-Length
Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.
Current responses are a bit old. Need a refresh.
The new HTTP 1.1 RFC is RFC 7230. And contains more precise information on parsing the messages size.
Message Body Length
Associating a response to a request
Security Considerations
Detecting the size of a message is quite complex. You can have a Content-length, or Transfer-Encoding: chunked, or both, or none. And some sepcial codes like 100 Continue which may alter all this.
The first link contains 7 entries that should be checked in the right order to guess the right size.
And as stated in the last link, failing to detect the right message length may lead to HTTP Smuggling (splitting, cache poisoning) issues.
Pipelining support is the source of most smuggling issues. You should really take care of the whole RFC7230 document if you want to implement it.
If a clients sends data in an unsupported media type to a HTTP server, the server answers with status "415 unsupported media type". But how to tell the client what media types are supported? Is there a standard or at least a recommended way to do so? Or would it just be written to the response body as text?
There is no specification at all for what to do in this case, so expect implementations to be all over the place. (What would be sensible would be if the server's response included something like an Accept: header since that has pretty much the right semantics, if currently in the wrong direction.)
I believe you can do this with the OPTIONS Http verb.
Also the status code of 300 Multiple Choices could be used if your scenario fits a certain use case. If they send a request with an Accept header of application/xml and you only support text/plain and that representation lives at a distinct URL then you can respond with a 300 and in the Location header the URL of that representation. I realize this might not exactly fit your question, but it's another possible option.
And from the HTTP Spec:
10.4.7 406 Not Acceptable
The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.
Unless it was a HEAD request, the response SHOULD include an entity containing a list of available entity characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. Depending upon the format and the capabilities of the user agent, selection of the most appropriate choice MAY be performed automatically. However, this specification does not define any standard for such automatic selection.
Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
tl;dr;
Edited the generated proxy class to inherit from Microsoft.Web.Services3.WebServicesClientProtocol**.
I came across this question when troubleshooting this error, so I thought I would help the next person who might come through here, although not sure if it answers the question as stated. I ran into this error when at some point I had to take over an existing solution which was utilizing WSE and MTOM encoding. It was a windows client calling a web service.
To the point, the client was calling the web service where it would throw that error.
Something that contributed to resolving that error for me was to check the web service proxy class that apparently is generated by default to inherit from System.Web.Services.Protocols.SoapHttpClientProtocol.
Essentially that meant that it didn't actually use WSE3.
Anyhow I manually edited the proxy and changed it to inherit from Microsoft.Web.Services3.WebServicesClientProtocol.
BTW, to see the generated proxy class in VS click on the web reference and then click the 'Show All Files' toolbar button. The reference.cs is da place of joy!
Hope it helps.
In his book "HTTP Developer's Handbook" on page 81 Chris Shiflett explains what a 415 means, and then he says, "The media type used in the content of the HTTP response should be indicated in the Content-Type entity header."
1) So is Content-Type a possible answer? It would presumably be a comma-separated list of accepted content types. The obvious problem with this possibility is that Content-Type is an entity header not a response header.
2) Or is this a typo in the book? Did he really mean to say "the HTTP request"?
I have a program that is supposed to interact with a web server and retrieve a file containing structured data using http and cgi. I have a couple questions:
The cgi script on the server needs to specify a body right? What should the content-type be?
Should I be using POST or GET?
Could anyone tell me a good resource for reading about HTTP?
If you just want to retrieve the resource, I’d use GET. And with GET you don’t need a Content-Type since a GET request has no body. And as of HTTP, I’d suggest you to read the HTTP 1.1 specification.
The content-type specified by the server will depend on what type of data you plan to return. As Jim said if it's JSON you can use 'application/json'. The obvious payload for the request would be whatever data you're sending to the client.
From the servers prospective it shouldn't matter that much. In general if you're not expecting a lot of information from the client I'd set up the server to respond to GET requests as opposed to POST requests. An advantage I like is simply being able to specify what I want in the url (this can't be done if it's expecting a POST request).
I would point you to the rfc for HTTP...probably the best source for information..maybe not the most user friendly way to get your answers but it should have all the answers you need. link text
For (1) the Content-Type depends on the structured data. If it's XML you can use application/xml, JSON can be application/json, etc. Content-Type is set by the server. Your client would ask for that type of content using the Accept header. (Try to use existing data format standards and content types if you can.)
For (2) GET is best (you aren't sending up any data to the server).
I found RESTful Web Services by Richardson and Ruby a very interesting introduction to HTTP. It takes a very strict, but very helpful, view of HTTP.
I have to pass a meta-information in my HTTP response so I figured out that I could use the response header, for instance "X-MyData: 123456". Is that safe? I mean, there is a possibility that a client proxy remove this header?
Thanks!
For reference, X- headers are also referred to as x-token in the BNF of RFC 2045, as user-defined ("X-") in section 5 of RFC 2047 and as Experimental headers in section 4.2.2.1 of the News Article Format draft.
Deprecating Use of the "X-" Prefix in Application Protocols (BCP, June 2012):
deprecates the "X-" convention for most application protocols and
makes specific recommendations about how to proceed in a world without
the distinction between standard and non-standard parameters
A client proxy could do anything it wanted, but in general would not strip any headers.
Headers starting with an X- are typically reserved for nonstandard usage (i.e. no future standard will introduce a header starting X-) but a proxy may understand them and choose to modify them as it wants.
It is possible for proxy servers or any intermediate links in the chain to modify your headers, but this usually isn't a problem.
More often than not, specifying custom headers is fine as long as they're unique enough not to conflict with other people's headers and you don't expect anyone else to use yours.