I'm writing a small experimental http server in GO using the net/http package, and I need all my replies to have 'identity' transfer encoding. However the http server in GO always returns responses using the 'chunked' transfer.
Is there any way to disable chunked encoding on HTTP server in GO?
It's not clear to me whether responding with "Transfer-Encoding: identity" is valid under the spec (I think maybe you should just leave it out), but...
Inspecting the code here, I see this inside the WriteHeader(code int) function (it's a little bit strange, but this function actually flushes all the headers to the socket):
367 } else if hasCL {
368 w.contentLength = contentLength
369 w.header.Del("Transfer-Encoding")
370 } else if w.req.ProtoAtLeast(1, 1) {
371 // HTTP/1.1 or greater: use chunked transfer encoding
372 // to avoid closing the connection at EOF.
373 // TODO: this blows away any custom or stacked Transfer-Encoding they
374 // might have set. Deal with that as need arises once we have a valid
375 // use case.
376 w.chunking = true
377 w.header.Set("Transfer-Encoding", "chunked")
378 } else {
I believe "hasCL" in the first line above refers to having a content length available. If it is available, it removes the "Transfer-Encoding" header altogether, otherwise, if the version is 1.1 or greater, it sets the "Transfer-Encoding" to chunked. Because this is done right before writing it to the socket, I don't think there's currently going to be any way for you to change it.
Related
Its seems a http get can be performed using either .Q.hg or using the built in HTTP request like
`:http://host:port "string to send as HTTP method etc"
(from https://code.kx.com/q/kb/programming-examples/)
Is there any difference?
And do either persist/keep-alive by default?
Thank you
Using .Q.hg allows you to use a string which is formatted in a way that is consistent with a web-based url request, .e.g for requesting some csv data from a server:
t:.Q.hg`$":http://www.website.com/report1/format=csv&cols=sym&cols=price&date=20200630";
/the resulting string contains the data only (no metadata/headers) and can be parsed directly
("SF";1#csv)0:t
The GET equivalent is not like a browser url, however it does return the metadata/headers (which in turn makes it messier to parse), e.g.
t:(hsym`$"http://www.website.com") "GET /report1/format=csv&cols=sym&cols=price&date=20200630 HTTP/1.1\r\nhost:www.website.com\r\n\r\n";
/result looks like
"HTTP/1.1 200 OK\r\nDate: Fri, 03 Jul 2020 14:46:33 GMT\r\nContent-Type: application/txt\r\nContent-Length: 1345\r\nConnection: keep-alive ...."
/parsed using something like (strip away metadata to get to the data)
("SF";1#csv)0:_[;t]3+first t ss "\n\r\n"
The resulting metadata/header shows "Connection: keep-alive" in my example that I've just tested so perhaps that's the default? I'm not 100% on that.
.Q.hg also has the advantage of being compatible with HTTPS and making use of proxies as per the documentation: https://code.kx.com/q/ref/dotq/#qhg-http-get
.Q.hg and .Q.hp have a similar functionality to the example outlined in the link, without having to construct the HTTP requests as strings (these functions will construct the strings for you). The example was perhaps written before the .Q.hg/.Q.hp functions were introduced in v3.4.
I don't think either persist by default assuming they use HTTP 1.0 protocol.
We're currently working to the OData standard:
http://www.odata.org/developers/protocols/operations
According to this standard, if you perform an PUT operation against your data, you should return a 204 status code. The standard says that:
When processing a PUT request servers return status 204 (No Content)
to indicate success, no response body is needed.
Now, according to what one reads elsewhere, some places claim that if you're returning a 204 you should absolutely NOT return a response body - it's not a question of whether one is needed or not.
The problem is we kind of need to return a body. In this case we're performing an update query on a batch of items and partial success (i.e. some are updated, some are not) is possible. We'd like to inform the client of what the failures were and the response body is the only way of doing it that I can think of.
So what's the "correct" way of doing this. Should I flout the W3C and return a body? Or should I flout OData and return a different response code? Or is there another possibility?
The 204 response to PUT is not a mandatory requirement for OData servers (note that the odata.org is not a normative reference, the MS-OData document is). You can return 200 with body, in which case the body you return should be a serialization of the updated entity (after the updates were applied by the server). In fact in OData V3 we now have a support for Prefer header which does exactly this. The client can include a Prefer header with the PUT request and ask the server to respond with 200 and the updated entityt.
I think you must not return content and should consider returning a header to indicate the partially complete command.
The spec states
The 204 response MUST NOT include a message-body, and thus is always terminated by the first empty line after the header fields.
You may also wish to take the batch approach, http://www.odata.org/developers/protocols/batch
You shouldn't return 204 unless the operation succeeded completely.
Seems weird to return a 204 for a PUT. I always thought 204 paired with the DELETE. Where the text 'no content' means that the resource is no longer on the server. The PUT doesn't have anything to do with removing resources so shouldn't use 204. Also 204 is a success code so should be used when 100% successful.
When doing a PUT or DELETE with an "If-Match" header, in case the ETag sent by a client indicates staleness, rather than just returning a 412, I'd like to return the whole up-to-date entity (including its new ETag in the HTTP header), so the client does not have to perform another GET round trip, which they otherwise would certainly do - in my use-case at least they'd do in probably 100% of the cases.
I don't see anything for or against it in the docs for 412:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13
And looking at, say, status code 409, it doesn't seem to be a problem in general to do whatever one likes with the response body of a 4xx error:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10
So, does anything (especially in the HTTP specs) speak against return the full up-to-date entity and its ETag?
Should be fine:
All 1xx (informational), 204 (no content), and 304 (not modified)
responses MUST NOT include a message-body. All other responses do
include a message-body, although it MAY be of zero length.
Source: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.3
What is the request? GET with If-None-Match? In that case, the server isn't supposed to return 412 anyway.
For PUT, DELETE, you certainly can return the current representation. For large representations, it will be inconvenient for clients that don't need it though.
You may also want to label the payload as representation of the resource by using the Location header; see http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-16.html#identifying.response.associated.with.representation.
I'm having trouble finding a definite specification of this in the standard. I have an HTTP client that's not including a Content-Length: 0 header when doing a PUT request where I don't specify a body, and a server that gets confused by such requests, and I'm wondering which program I should be blaming.
HTTP requests have a body if they have a Content-Length or Transfer-Encoding header (RFC 2616 4.3). If the request has neither, it has no body, and your server should treat it as such.
That said it is unusual for a PUT request to have no body, and so if I were designing a client that really wanted to send an empty body, I'd pass Content-Length: 0. Indeed, depending on one's reading of the POST and PUT method definitions (RFC 2616 9.5, 9.6) one might argue that the body is implied to be required - but a reasonable way to handle no body would be to assume a zero-length body.
Not answering the question, but asserting how jaxrs allows me to frequent use of bodyless PUTs:
Example of bodyless put:
Give user an additional permission.
PUT /admin/users/{username}/permission/{permission}
A body is not required by the IETF standard, though the content-length should be 0 if there's no body. Use the method that's appropriate for what you're doing. If you were to put it into code, given
int x;
int f(){ return x; }
and a remote variable called r.
A post is equivalent to
r=f();
A put is equivalent to
r=x;
and a get is equivalent to
x=r;
What is being PUT (in the verb sense) onto the server if there's no content? The spec refers to the content as "the enclosed entity", but a request with no content would have no enclosed entity, and therefore nothing to put on the server.
Unless, of course, you wanted to PUT nothing onto the server, in which case you'd probably want a DELETE instead.
The content length field is required as per the following section in the HTTP/1.1 standard http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13
I have been trying to stream audio from a particular point by using the Range header values but I always get the song right from the beginning. I am doing this through a program so am not sure whether the problem lies in my code or on the server.
How can I find out whether the server supports the Range header param?
Thanks.
The way the HTTP spec defines it, if the server knows how to support the Range header, it will. That in turn, requires it to return a 206 Partial Content response code with a Content-Range header, when it returns content to you. Otherwise, it will simply ignore the Range header in your request, and return a 200 response code.
This might seem silly, but are you sure you're crafting a valid HTTP request header? All too commonly, I forget to specify HTTP/1.1 in the request, or forget to specify the Range specifier, such as "bytes".
Oh, and if all you want to do is check, then just send a HEAD request instead of a GET request. Same headers, same everything, just "HEAD" instead of "GET". If you receive a 206 response, you'll know Range is supported, and otherwise you'll get a 200 response.
This is for others searching how to do this. You can use curl:
curl -I http://exampleserver.com/example_video.mp4
In the header you should see
Accept-Ranges: bytes
You can go further and test retrieving a range
curl --header "Range: bytes=100-107" -I http://exampleserver.com/example_vide0.mp4
and in the headers you should see
HTTP/1.1 206 Partial Content
and
Content-Range: bytes 100-107/10000000
Content-Length: 8
[instead of 10000000 you'll see the length of the file]
Although I am a bit late in answering this question, I think my answer will help future visitors. Here is a python method that detects whether a server supports range queries or not.
def accepts_byte_ranges(self, effective_url):
"""Test if the server supports multi-part file download. Method expects effective (absolute) url."""
import pycurl
import cStringIO
import re
c = pycurl.Curl()
header = cStringIO.StringIO()
# Get http header
c.setopt(c.URL, effective_url)
c.setopt(c.NOBODY, 1)
c.setopt(c.HEADERFUNCTION, header.write)
c.perform()
c.close()
header_text = header.getvalue()
header.close()
verbose_print(header_text)
# Check if server accepts byte-ranges
match = re.search('Accept-Ranges:\s+bytes', header_text)
if match:
return True
else:
# If server explicitly specifies "Accept-Ranges: none" in the header, we do not attempt partial download.
match = re.search('Accept-Ranges:\s+none', header_text)
if match:
return False
else:
c = pycurl.Curl()
# There is still hope, try a simple byte range query
c.setopt(c.RANGE, '0-0') # First byte
c.setopt(c.URL, effective_url)
c.setopt(c.NOBODY, 1)
c.perform()
http_code = c.getinfo(c.HTTP_CODE)
c.close()
if http_code == 206: # Http status code 206 means byte-ranges are accepted
return True
else:
return False
One way is just to try, and check the response. In your case, it appears the server doesn't support ranges.
Alternatively, do a GET or HEAD on the URI, and check for the Accept-Ranges response header.
You can use GET method with 0-0 Range request header, and check whether the response code is 206 or not, which will respond with
the first and last bytes of the response body
You also can use HEAD method do the same thing as the first session which will get the same response header and code without response body
Furthermore, you can check Accept-Ranges on the response header to judge whether it can support range, but please notice if the value is none on Accept-Ranges field, it means it can't support range, and if the response header doesn't have Accept-Ranges field you also can't finger out it can't support range from it.
There is another thing you have to know if you are using 0- Range on the request header with GET method to check the response code, the response body message will be cached automatically on the TCP receive window until the cache is full.