How to extract Data from a multipart Http Request? - http

I am writing a HTTP Webserver. My server has to handle Http multipart requests. In my previous implementation, I was extracting the data with the help of content length header present in every part of request. The client which I was using give content-length header with every part part(file) in the multipart request.
But another client is not giving content-length of each file. In my implementation I use content-length header to extract that much bytes and save them into a file.
Please tell me how can I extract data now.
The Headers which I am getting now are:
POST xxxxxxxxxxxxxxxxxxxxxxx&currentTab=PHOTOxxxxxxxxxxxxxxxx HTTP/1.1
Content-Length: 6829
Content-Type: multipart/form-data; boundary=SnlCg9JqTpQIl6t_mPzByTjZ8bD24kUj; charset=UTF-8
Host: host
Connection: Keep-Alive
User-Agent: Apache-HttpClient/xxxxxxxx
Accept-Encoding: gzip
--SnlCg9JqTpQIl6t_mPzByTjZ8bD24kUj
Content-Disposition: form-data; name="file"; filename="imagesCA5L2CL6_jpg(2)_jpg.jpg"
Content-Type: photo/jpg
**Some Data byte array**
--SnlCg9JqTpQIl6t_mPzByTjZ8bD24kUj--
In this request, there is now content-length header in part data.
EDIT:
Earlier this client used to send content-length header in every part. But for some reason it is not sending it any more. Can anybody suggest any reason for that.
thanks

Like this : Reading file input from a multipart/form-data POST
Reading file input from a multipart/form-data POST

Take a look at RFC 2616 if you want to implement a HTTP/1.1 server. See section 4.4 on how to determine message length. See RFC 2388 on how to implement multipart/form-data.
The real answer is: don't reinvent the wheel, or you'll have to reimplmement a few hundred pages of RFC's. There are tons of libraries and servers out there.
If you do want to write your own web server, for example as an exercise, you would have found those RFC's already, right?

Related

What is a chunked request in OpenTSDB?

I'm using Qt4 to post some data points to a OpenTSDB server, which doesn't supports chunked HTTP requests.
The code is basically this:
QNetworkRequest request(m_url);
request.setHeader(QNetworkRequest::ContentTypeHeader, QString("application/json"));
request.setHeader(QNetworkRequest::ContentLengthHeader, jsonRequest.toAscii().size());
m_networkAccessManager.post(request, jsonRequest.toAscii());
jsonRequest is a QString containing the data points. This code is called from time to time to upload data to the server, and it usually works fine. However, sometimes I receive an error from openTSDB stating that "Chunked request not supported.".
This seems to happen when the request gets a little bigger (and by bigger, I mean some KB of data).
edit:
I've done a tcpdump of the request when the problem arises, and in fact it doesn't seen to be chunked:
POST /api/put HTTP/1.1
Content-Type: application/json
Content-Length: 14073
Connection: Keep-Alive
Accept-Encoding: gzip
Accept-Language: en,*
User-Agent: Mozilla/5.0
Host: 192.168.xx.xxx:xxxx
[{"metric":"slt.reader.temperature","timestamp":1420736269427,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736280628,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736291637,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736302748,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736313840,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736325011,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736336039,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736347182,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736358210,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736369372,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736380401,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736385286,"value":0,"tags":{"sltId":"5036","readerId":"1","antenna":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736385286,"value":10,"tags":{"sltId":"5036","readerId":"1","antenna":"2","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736385286,"value":7,"tags":{"sltId":"5036","readerId":"1","antenna":"3","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736385287,"value":6,"tags":{"sltId":"5036","readerId":"1","antenna":"4","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736385287,"value":13,"tags":{"sltId":"5036","readerId":"1","antenna":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736385287,"value":99,"tags":{"sltId":"5036","readerId":"1","antenna":"2","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736385287,"value":102,"tags":{"sltId":"5036","readerId":"1","antenna":"3","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736385287,"value":93,"tags":{"sltId":"5036","readerId":"1","antenna":"4","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.transactionsNotDeciphered","timestamp":1420736385287,"value":0,"tags":{"sltId":"5036","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736391436,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736402608,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736413642,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736424676,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736435823,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736446850,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736458007,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736469060,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736480207,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736491418,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736502620,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736513638,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736524682,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736535712,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736546742,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736557834,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736568858,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736579932,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736590966,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736601993,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736613183,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736624357,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736635387,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736646414,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736657493,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736668624,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736679743,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736685286,"value":0,"tags":{"sltId":"5036","readerId":"1","antenna":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736685286,"value":8,"tags":{"sltId":"5036","readerId":"1","antenna":"2","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736685286,"value":9,"tags":{"sltId":"5036","readerId":"1","antenna":"3","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736685295,"value":5,"tags":{"sltId":"5036","readerId":"1","antenna":"4","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736685295,"value":4,"tags":{"sltId":"5036","readerId":"1","antenna":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736685295,"value":88,"tags":{"sltId":"5036","readerId":"1","antenna":"2","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736685295,"value":130,"tags":{"sltId":"5036","readerId":"1","antenna":"3","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736685296,"value":123,"tags":{"sltId":"5036","readerId":"1","antenna":"4","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.transactionsNotDeciphered","timestamp":1420736685296,"value":0,"tags":{"sltId":"5036","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736690786,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736701910,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736712968,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736723999,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736735075,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736746106,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736757266,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736768455,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736779473,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736790606,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736801633,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736812713,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736823740,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736834856,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736845958,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736857103,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736868216,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736879292,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736890320,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736901503,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736912608,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736923761,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736934850,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736946033,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736957061,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736968223,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736979256,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736985284,"value":0,"tags":{"sltId":"5036","readerId":"1","antenna":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736985285,"value":16,"tags":{"sltId":"5036","readerId":"1","antenna":"2","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736985285,"value":9,"tags":{"sltId":"5036","readerId":"1","antenna":"3","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tags_read","timestamp":1420736985285,"value":11,"tags":{"sltId":"5036","readerId":"1","antenna":"4","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736985285,"value":9,"tags":{"sltId":"5036","readerId":"1","antenna":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736985285,"value":162,"tags":{"sltId":"5036","readerId":"1","antenna":"2","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736985285,"value":166,"tags":{"sltId":"5036","readerId":"1","antenna":"3","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.tag_transactions","timestamp":1420736985285,"value":157,"tags":{"sltId":"5036","readerId":"1","antenna":"4","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.transactionsNotDeciphered","timestamp":1420736985286,"value":0,"tags":{"sltId":"5036","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420736990353,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737001532,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737012658,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737023691,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737034823,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737045906,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737056942,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}},{"metric":"slt.reader.temperature","timestamp":1420737068032,"value":56,"tags":{"sltId":"5036","readerId":"1","host":"xxxxxxxxxxxxxxxxx"}}]HTTP/1.1 400 Bad Request
Content-Length: 1080
Content-Type: text/html; charset=UTF-8
Date: Thu, 08 Jan 2015 17:18:43 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta http-equiv=content-type content="text/html;charset=utf-8"><title>Bad Request</title>
<style><!--
body{font-family:arial,sans-serif;margin-left:2em}A.l:link{color:#6f6f6f}A.u:link{color:green}.subg{background-color:#e2f4f7}.fwf{font-family:monospace;white-space:pre-wrap}//--></style></head>
<body text=#000000 bgcolor=#ffffff><table border=0 cellpadding=2 cellspacing=0 width=100%><tr><td rowspan=3 width=1% nowrap><b><font color=#c71a32 size=10>T</font><font color=#00a189 size=10>S</font><font color=#1a65b7 size=10>D</font> </b><td> </td></tr><tr><td class=subg><font color=#507e9b><b>Looks like it's your fault this time</b></td></tr><tr><td> </td></tr></table><blockquote><h1>Bad Request</h1>Sorry but your request was rejected as being invalid.<br/><br/>The reason provided was:<blockquote>Chunked request not supported.</blockquote></blockquote><table width=100% cellpadding=0 cellspacing=0><tr><td class=subg><img alt="" width=1 height=6></td></tr></table></body></html>
I was thinking Qt was changing to use a chunked request when the request got larger, but it's not the case. So the current question is: what is a chunked request in openTSDB?
I've finally found the reason it thinks my request is chunked: OpenTSDB internally uses Netty for networking, and if Netty reads a block that doesn't contain the complete request, then it's flagged as chunked, even if there's no Transfer-Encoding header in the request.
There are two possible solutions for this:
adding "tsd.http.request.enable_chunked = true" to the config file, so that it doesn't reject chunked requests and adding "tsd.http.request.max_chunk = " to the config file, and setting it to a sensible size for your requests
If you can change the client, force it to send smaller requests
Please note that enabling chunked requests for public facing servers may not be a good idea, as a malicious client can send unbounded requests, overloading the server.
Credits goes to irc users jaylr and manolama, who provided me the info on irc channel #opentsdb#freenode.net

HTTP HEAD chunked-encoded

I have implemented a HTTP 1.1 server. It is an embedded server so I only support the mandatory features of the RFC. All responses are sent chunked-encoded.
As HEAD is mandatory it is also supported.
HEAD is a GET without body. So the server is sending a response like following in response to a HEAD request:
HTTP/1.1 200 OK
Server: testServer
Connection: keep-alive
Transfer-Encoding: chunked
What I am wondering is do have to add a "0\r\n" as it required to signal the end of chunks:
HTTP/1.1 200 OK
Server: testServer
Connection: keep-alive
Transfer-Encoding: chunked
0
I have tried to collect the relevant parts in the RFC:
"The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response."
"All responses to the HEAD request method MUST NOT include a message-body, even though the presence of entity- header fields might lead one to believe they do."
"1.Any response message which "MUST NOT" include a message-body (such as the 1xx, 204, and 304 responses and any response to a HEAD request) is always terminated by the first empty line after the header fields, regardless of the entity-header fields present in the message."
So far I understand it that my first solution (without 0) is the correct one. But it seems to be strange to send a message with Transfer-encoding: chunked which does not terminate with the the chunk style 0\r\n.
I guess the problem stated is solved as its quite old,
Recently I developed a test http server for streaming video and I wanted to put forth my learnings so that it might help others.
First, "0\r\n" is not the marker for end of chunks, the chunked trailer is '0\r\n\r\n'.
And for the header the string should look like this
"HTTP/1.1 200 OK\r\n" \
"Server: testServer\r\n" \
"Connection: keep-alive\r\n" \
"Transfer-Encoding: chunked\r\n" \
"\r\n"
Note the last CRLF, its indicating end of header.
Hope this help.

File transfer over HTTP

As far as I know it is possible to transfer binary files over HTTP protocol. But HTTP is a text-based protocol, the typical HTTP response frame looks as follows:
HTTP/1.1 200 OK
Date: Wed, 23 May 2012 22:38:34 GMT
Content-Length: 438
Content-Type: text/html; charset=UTF-8
Here goes content
If so, how should binary file be encoded in this frame? What is the Content-Type? Is the content encoded with base64 - same as attachments in POP3 protocol? Or it is raw data (is it possible not to cause problems if so?)
The header fields are text based, but the actual payload is binary. You can transfer whatever you want.
And no, it doesn't have anything to do with the Content-Type. That is just a label so that the recipient knows how to process the data; it does not affect the format in the protocol itself.
Binary files are usually transferred with the Application/octet-stream mimetype (unless they match another more specific mimetype of course). For transmission you use the raw data - no base64 needed.

Chunked Transfer Encoding problem with Apache Abdera

I'm using Apache Abdera to POST atom multipart data to my server, and am having some odd problems that I can't pin down.
It looks like an issue with chunked transfer encoding, but I'm insufficiently experienced to be certain. The problem manifests as the server throwing an error indicating that the request I sent it contains only one mime part, not two as required. I attached Wireshark to the interface and captured the conversation, and it went like this:
POST /sss/col-uri/2ee98ea1-f9ad-4f01-9b1c-cfa3c4a6dc3c HTTP/1.1
Host: localhost
Expect: 100-continue
Transfer-Encoding: chunked
Content-Type: multipart/related; boundary="1306399868259";type="application/atom+xml;type=entry"
The server's response:
HTTP/1.1 100 Continue
My client continues:
198
--1306399868259
Content-Type: application/atom+xml;type=entry
Content-Disposition: attachment; name="atom"
<entry xmlns="http://www.w3.org/2005/Atom"><title xmlns="http://purl.org/dc/terms/">Richard Woz Ere</title><bibliographicCitation xmlns="http://purl.org/dc/terms/">this is my citation</bibliographicCitation><content type="application/zip" src="cid:48bd9436-e8b6-4f68-aa83-5c88eda52fd4" /></entry>
0
b0e9
--1306399868259
Content-Type: application/zip
Content-Disposition: attachment; name="payload"; filename="example.zip"
Content-ID: <48bd9436-e8b6-4f68-aa83-5c88eda52fd4>
Packaging: http://purl.org/net/sword/package/SimpleZip
And at this point the server responds with:
HTTP/1.1 400 Bad Request
Date: Thu, 26 May 2011 08:51:08 GMT
Server: Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8l DAV/2 mod_wsgi/3.3 Python/2.6.1
Connection: close
Transfer-Encoding: chunked
Content-Type: text/xml
Indicating the error (which is well understood). My server goes on to stream a pile of base64 encoded bits onto the output stream, but in the mean time the server is not listening, it has already decided that the request was erroneous.
Unfortunately, I'm not in charge of the HTTP layer - this is all handled by Abdera using Apache httpclient. My code that does this looks like this:
client.execute("POST", url.toString(), new SWORDMultipartRequestEntity(deposit), options);
Here, the SWORDMultipartRequestEntity is a copy of the standard Abdera MultipartRequestEntity class, with a few extra headers thrown in (see, for example, Packaging in the above snippet); the "deposit" argument is just an object holding the atom part and the inputstream.
When attaching a debugger I get to this line of code fine, and then it disappears into a rat hole and then I get this error back.
Any hints or tips? I've pretty much exhausted my angles of attack!
The only thing that stands out for me is that immediately after the atom:entry document, there is a newline with "0" on it alone, which appears to be chunked transfer encoding speak for "I'm finished". Not sure how it got there, or whether it really has any effect. Help much appreciated.
Cheers,
Richard
The lonely 0 may indeed be a problem. My uninformed guess is that it results from some call to flush(), which then writes the whole buffer as another HTTP chunk. Unfortunately at the point where flush is called, the buffer had already been flushed and its size is therefore zero. So the HttpChunkedOutputFilter (or however it is called) should be taught than an empty buffer does not need to be flushed.
[update:] You should set a breakpoint in the ChunkedOutputStream class, especially the flush method. I just looked at its code and it seems to be ok, but maybe I missed something.

Difference between Content-Range and Range headers?

What is the difference between HTTP headers Content-Range and Range? When should each be used?
I am trying to stream an audio file from a particular byte offset. Should I use Content-Range or Range header?
Actually, the accepted answer is not complete. Content-Range is not only used in responses. It is also legal in requests that provide an entity body.
For example, an HTTP PUT provides an entity body, it might provide only a portion of an entity. Thus the PUT request can include a Content-Range header indicating to the server where the partial entity body should be merged into the entity.
For example, let's first create and then append to a file using HTTP:
Request 1:
PUT /file HTTP/1.1
Host: server
Content-Length: 1
a
Request 2:
PUT /file HTTP/1.1
Host: server
Content-Range: bytes 1-2/*
Content-Length: 1
a
How, let's see the file's contents...
Request 3:
GET /file HTTP/1.1
Host: server
HTTP/1.1 200 OK
Content-Length: 2
aa
This allows random file access, both READING and WRITING over HTTP. I just wanted to clarify, as I was researching the use of Content-Range in a WebDAV client I am developing, so perhaps this expanded information will prove useful to somebody else.
Range is used in the request, to ask for a particular range (or ranges) of bytes. Content-Range is used in the response, to indicate which bytes the server is giving you (which may be different than the range you requested), as well as how long the entire content is (if known).

Resources