While writing my HTTP/1.1 server, I get stuck dealing multiple ranges request.
Section 14.35.1 of RFC 2616 refers some examples but doesn't clarify server behaviour.
For instance:
GET /some/resource HTTP/1.1
...
Range: bytes=200-400,100-300,500-600
...
Should I return this exact sequence of bytes?
Or should I merge all ranges, sending 100-400,500-600?
Or sending all in between, 100-600?
Worst, when checking Content-Range response header (Section 14.16), only a single range may be returned, so I wonder how would a server response to example in Section 14.35.1 bytes=0-0,-1!!!
How should my server handle such requests?
I just had a look at how other servers that support the Range header field might respond and did a quick curl to example.com:
~# curl -s -D - -H "Range: bytes=100-200, 300-400" http://www.example.com
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-Type: multipart/byteranges; boundary=3d6b6a416f9b5
Content-Length: 385
Server: ECS (fll/0761)
--3d6b6a416f9b5
Content-Type: text/html
Content-Range: bytes 100-200/1270
eta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="vieport" content
--3d6b6a416f9b5
Content-Type: text/html
Content-Range: bytes 300-400/1270
-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: "Open Sans", "Helvetica
--3d6b6a416f9b5--
Apparently, what your looking for is the Content-Type: multipart/byteranges; boundary response header. Googling exactly that turned up a W3C document with appendices to RFC 2616
When an HTTP 206 (Partial Content) response message includes the content of multiple ranges (a response to a request for multiple non-overlapping ranges), these are transmitted as a multipart message-body. The media type for this purpose is called "multipart/byteranges".
The multipart/byteranges media type includes two or more parts, each with its own Content-Type and Content-Range fields. The required boundary parameter specifies the boundary string used to separate each body-part.
So there you go.
By the way, the server at example.com does not check for overlapping byte ranges and sends you exactly the ranges that you requested...
Related
I'm trying to test writing correct HTTP headers to understand
the syntax. Here I'm trying to PUT some text into httpbin.org/put and I expect the response body content to be the same.
PUT /HTTP/1.1
Host: httpbin.org
Accept-Language: en-us
Connection: Keep-Alive
Content-type: text/plain
Content-Length: 12
Hello jerome
However I'm getting the following bad request 400 response:
HTTP/1.1 400 Bad Request
Server: nginx
Date: Tue, 01 Mar 2016 12:34:02 GMT
Content-Type: text/html
Content-Length: 166
Connection: close
Response:
<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
What syntactical errors have I done?
NOTE: newlines are \r\n not \n in the request.
Apparently the correct syntax goes like this for PUT:
PUT /put HTTP/1.1\r\n
Content-Length: 11\r\n
Content-Type: text/plain\r\n
Host: httpbin.org\r\n\r\n
hello lala\n
I believe I didn't say much on how I connected to httpbin.org; it was via sockets in C. So the connection was already established before sending the header + message.
You miss the destination url following the PUT verb, the first line must be:
PUT http://httpbin.org/ HTTP/1.1
This will probably also fail, you need one of their handler urls so they know what to reply with:
PUT http://httpbin.org/put HTTP/1.1
The general form of the first line, or Request Line, in an HTTP request is as follows:
<method> <path component of URL, or absolute URL> HTTP/<Version>\r\n
Where for your example, the method is PUT. Including an absolute URL (so, starting with http:// or https:// is only necessary when connecting to a proxy, because the proxy will then attempt to retrieve that URL, rather than attempt to serve a local resource (as found by the path component).
As presented, the only change you should have needed to make was ensuring there was a space between the / and HTTP/1.1. Otherwise, the path would be "/HTTP/1.1"... which would be a 404, if it weren't already a badly formed request. /HTTP/1.1 being interpreted as a path means the HTTP server that's parsing your request line doesn't find the protocol specifier (the HTTP/1.1 bit) before the terminating \r\n... and that's one example of how 400 response codes are born.
Hope that helped. Consult the HTTP 1.1 RFC (2616), section 5.1 for more information and the official definitions.
When receiving a response back with a netty client object, I run into a FrameTooLongException. After taking a tcpdump, found that the response received is a large Mutlipart Mime response with about 200 parts (each with some short headers), but the actual HTTP Header for the response is quite small and are listed as;
> Host: foobar.com:20804
> Accept: */*
>
< HTTP/1.1 207 Multi-Status
< Date: Tue, 04 Aug 2015 19:44:09 GMT
< Vary: Accept
< Content-Type: multipart/mixed; boundary="63602357878446117"
< Content-Length: 33023
I couldn't find anything in the documentation about this, but are Mime part headers used in the HTTP Header size calculation, and does Netty parse it as such?
The exception I get is as follows:
io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.
at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.newException(HttpObjectDecoder.java:787)
at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.process(HttpObjectDecoder.java:779)
at io.netty.buffer.AbstractByteBuf.forEachByteAsc0(AbstractByteBuf.java:1022)
at io.netty.buffer.AbstractByteBuf.forEachByte(AbstractByteBuf.java:1000)
at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:751)
at io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:545)
at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:221)
at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:136)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1044)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:934)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
Http header terminates with 2 cr/lf (such as between Accept and HTTP in your example), and header shall start with a "start line" (HTTP/1.1...).
Therefore I see 2 issues with your example:
Your header does not start correctly : HTTP/1.1 should be the first line, followed later on by your accept and other host header params.
Probably there is something wrong in your response such that there is no 2 cr/of between your header and the body, thus leading to the decoding of the body as if it was part of the header, so the exception...
In the header of an HTTP request or response will the header keys be constant in terms of capitalization, between servers.
I ask so I can expect in my code: (Using Fake Function names)
Safe Precise Python Code
for hdr in header.keys():
if 'content-length' == hdr.lower():
recv_more_data( header[hdr] ) # header[hdr] == Content-Length (5388) bytes
break # Exit for loop when if statement is met.
Code I Would Like To Use
recv_more_data (header['Content-Length'])
# I know to expect 'Content-Length' not 'content-Length' or some other variation
Meaning will a server ever return a header with the keys like so.
Standard Request
GET / HTTP/1.1
Host: www.example-host.com
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
Connection: closed
Content-Length: 0
A Bad But Possible Response?
HTTP/1.1 200 OK
Server: nginx/1.0.15
date: Thu, 23 Oct 2014 00:25:37 GMT
content-Type: text/html; charset=iso-8859-1
transfer-encoding: chunked
Connection: close
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Encoding: gzip
Clarification will help my code neatness.
HTTP header names are case-insensitive, per the HTTP specification.
RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1
Section 4.2 - Message Headers
HTTP header fields, which include general-header (section 4.5),
request-header (section 5.3), response-header (section 6.2), and
entity-header (section 7.1) fields, follow the same generic format as
that given in Section 3.1 of RFC 822 [9]. Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive. The field value MAY be preceded by any amount
of LWS, though a single SP is preferred. Header fields can be
extended over multiple lines by preceding each extra line with at
least one SP or HT. Applications ought to follow "common form", where
one is known or indicated, when generating HTTP constructs, since
there might exist some implementations that fail to accept anything
beyond the common forms.
RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
Section 3.2 - Header Fields:
Each header field consists of a case-insensitive field name followed
by a colon (":"), optional leading whitespace, the field value, and
optional trailing whitespace.
HTTP header names are case insensitive.
It looks like you're using python. Check out the requests library. It'll make your life much easier: http://docs.python-requests.org/en/latest/
Bear in mind that, even though most major servers will have consistent capitalization, any Joe PHP Developer can set the response headers manually in their code - and there is no way to police what that guy uses as a capitalization standard.
I am attempting to upload a file to Google Drive using the "upload" URL with a type of "multipart". I'm trying to do this without a library and using basic HTTP with a multipart POST. With a body like the following, I am constantly getting the error "Invalid multipart request with 0 mime parts."
The HTTP message looks valid to me. Is there something obvious that I'm missing or doing wrong?
Is there a protocol tester that can verify if my POST body is valid or not?
POST /upload/drive/v2/files?uploadType=multipart HTTP/1.1
Authentiction: Bearer {valid auth_token}
Content-Type: multipart/mixed; boundary="--314159265358979323846"
host: localhost:3004
content-length: 254
Connection: keep-alive
--314159265358979323846
Content-Type: application/json
{"title":"Now","mimeType":"text/plain"}
--314159265358979323846
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
Mon Jun 17 2013 20:59:02 GMT-0400 (EDT)
--314159265358979323846--
(The segments look like they have double newlines. I think this is an artifact of the pasting, they are CRLF pairs in the code and appear as a newline when testing, but I guess this could theoretically be the problem, but I'd like proof.)
boundary attribute on the Content-Type header should not include double dashes. Use the following as your Content-Type:
Content-Type: multipart/mixed; boundary="314159265358979323846"
How can I calculate content length for example of:
POST /Upload/ HTTP/1.1
Host: test.lan
User-Agent: Shockwave Flash
Connection: Keep-Alive
Cache-Control: no-cache
Accept: text/*
Content-Length: ?????
Content-Type: multipart/form-data; boundary=----------------------------4d2179e6b3c0
------------------------------4d2179e6b3c0
Content-Disposition: form-data; name="Filename"
phpinfo.php
------------------------------4d2179e6b3c0
Content-Disposition: form-data; name="ASPSESSID"
6e223eb1c7561e9c599f03cc04e9444b
------------------------------4d2179e6b3c0
Content-Disposition: form-data; name="Filedata"; filename="phpinfo.php"
Content-Type: application/octet-stream
<? phpinfo(); ?>
------------------------------4d2179e6b3c0
Content-Disposition: form-data; name="Upload"
Submit Query
------------------------------4d2179e6b3c0--
The Content-Length value should be calculated by totaling all data after the termination of the message headers. In the case of your example, this is everything after this point (with CRLF characters included for readability):
...
Content-Length: ?????\r\n
Content-Type: multipart/form-data; boundary=--------------------4d2179e6b3c0\r\n
\r\n
Everything coming after the first empty line (\r\n) -- including your boundary delimiters -- should be counted in the total length. In practice, this usually means that you'll need to tabulate the Content-Length header value after generating the full message entity body. Once you have the full body of the message you can prepend it with your headers to create the full HTTP message.
According to the HTTP spec you aren't technically required to specify the Content-Length header. From RFC 2616 14.13:
Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.
However, this is a pretty standard requirement for most servers which will generally send back an error response if the Content-Length is missing or incorrectly specified.