Although HTTP is ubiquitous it comes with its baggage of Headers which in my case is becoming more of a problem.
My data to be transferred is an iota of the HTTP header size.
Is there another protocol that I can
use which is still understood by the
browsers and other networks and doesn't come with the
baggage of HTTP?
Any other way to skip headers and add it at the destination so only a miniscule of data is transferred over the network?
No.
No.
Many HTTP headers are optional. A typical browser request is much larger than a minimal request, which might look like:
GET /doc HTTP/1.1
Host: example.com
Connection: close
(I can say with confidence that requests of this form work because I use them all the time when testing Web server response via telnet example.com 80.)
Possibly you can get useful results simply by omitting some headers.
HTTP requests can be quite small. As chaos points out in his answer, you don't really need to send many headers with a request. The only header that's essential is Host. I can simplify chaos' example a bit more by using HTTP 1.0, which doesn't feature persistent connections.
GET / HTTP/1.0
Host: example.com
(blank line is necessary)
The reply can be similarly simple
HTTP/1.0 200 OK
Content-Type: text/html
data content
In this case, the overhead of HTTP is about 40 bytes in the request and the response. A standard TCP packet is 1500 bytes so you have plenty of room left over in the response packet for the actual data.
There are other HTTP headers, and they do have value. You can include cache information and do conditional GETs. You can use an HTTP/1.1 persistent socket to make subsequent requests faster. Etc, etc. You don't have to use any of this stuff if you don't want, but one nice thing about HTTP is there's a standard way to do more complicated protocols when you need it.
As for doing minimal HTTP in JavaME, if you really care about every byte you may be best off writing your own simple HTTP client by working with a plain TCP socket. If you're talking to a known server you don't need to implement much at all. (If you're talking to arbitrary servers, you need to pay more attention to error handling, redirects, etc).
WebSockets are coming in HTML5 and should suit your needs. A standard HTTP connection can be renegotiated to change protocol to websockets. But I suspect the specification might be a bit young, but it might fit the bill.
Related
I would like to understand the usage of 'Transfer-encoding: Chunked' in case of HTTP requests.
Is it common for requests to be chunked?
My thinking is no since requests need to be completely read before processing, it does not make sense to be sending chunked requests.
It is not that common, but it can be very useful for large request bodies.
My thinking is no since requests need to be completely read before processing, it does not make sense to be sending chunked requests.
(1) No, they don't need to be read completely.
(2) ...and the main reason to compress it to save bytes on the wire anyway.
For an HTTP agent acting as a reverse proxy or a forward proxy, so taking a message from one side and sending it on the other side, using a chunked transmission means you can send the parts of the message you have without storing it locally. You avoid the 'buffering' problems, slowdown and storage.
You also have some optimizations based on each actor preferred size of data blocks, like you could have an actor which likes sending packets of 8000 bytes, because that's the good number for his own kernel settings (tcp windows, internal http server buffer size, etc), while another actor on the message transmission using smaller chunks of 2048 bytes.
Finally, you do not need to compute the size of the message, the message will end on the end-of-stream marker, that's all. Which is also usefull if you are sending something which is compressed on the fly, you may not know the final size until everything is compressed.
Chunked transmission is used a lot. It is the default mode of most HTTP servers if you ask for HTTP/1.1 mode and not HTTP/1.0.
I'm a bit rusty on nuances of the HTTP protocol and I'm wondering if it can support publish/subscribe directly?
HTTP is a request reponse protocol. So client sends a request and the server sends back a response.
In HTTP 1.0 a new connection was made for each request.
Now HTTP 1.1 improved on HTTP 1.0 by allowing the client to keep the connection open and make multiple requests.
I realise you can upgrade an HTTP connection to a websocket for fast 2 way communications. What I'm curious about is whether this is strictly necessary?
For example if I request a resource "http://somewhere.com/fetch/me/slowly"
Is the server free to reply directly twice?
Such as first with a 202 accepted
and then shortly later with the content when it is ready,
but without the client sending an additional request first?
i.e.
Client: GET http://somewhere.com/fetch/me/slowly
Server: 202 "please wait..."
Server: 200 "here's your document"
Would it be correct to implement a publish/subscribe service this way?
For example:
Client: http://somewhere.com/subscribe
Server: item 1
...
Server: item 2
I get the impression that this 'might' work because clients will typically have an event loop watching the connection but is technically wrong (because a client following the protocol need not be implemented that way).
However, if you use chunked transfer encoding this would work.
HTTP/2 seems to allow this as well but I'm not clear whether something changed to make it possible.
I haven't seen much discussion of this in relation to pub/sub so what if anything is wrong with using plain HTTP/1.1 with or without chunked encoding?
If this works why do you need things like RSS or ATOM?
A HTTP request can have multiple 'responses', but the responses all have statuscodes in the 1xx range, such as 102 Processing.
However, these responses are only headers, never bodies.
HTTP/1.1 (like 1.0 before it) is a request/response protocol. Sending a response unsolicited is not allowed. HTTP/2 is a frames protocol which adds server push which allows the server to offer extra data and handle multiple requests in parallel but doesn't change its request/response nature.
It is possible to keep a HTTP connection open and keep sending more data though. Many (audio, video) streaming services will use this.
However, this just looks like a continuous body that keeps on streaming, rather than many multiple HTTP responses.
If this works why do you need things like RSS or ATOM
Because keeping a TCP connection open is not free.
I was reading reading an article on launching of HTTP/2. It was said that HTTP/2 is based on SPDY (speedy) protocol and it can provide faster browsing speed compared to HTTP/1.1 by using "header field compression" and "multiplexing". How does these terms exactly work?
Am I supposed to believe that in HTTP/1.1 requests are processed in a 'one after the other' manner?
Multiplexing
With HTTP 1.1 a lot of time is spent just waiting. A browser sends requests and waits for the response to come back and then sends another GET etc. An inefficient use of the bandwidth. At times it would use Pipelining but that too suffers that sometimes requests need to wait for the requests done prior. The head of line blocking problem.
With multiplexing, there's virtually no waiting but the browsers can just ask for hundreds of things at once and they will be delivered in whatever order they can be delivered and without individual streams or objects having to wait for each other. (With prioritization and flow control to help control them properly.)
This will be most notable on high-latency connections. For a visible and clear demo what it can do, see the golang's gophertiles demo at https://http2.golang.org/gophertiles?latency=1000 (requires a HTTP/2 enabled browser)
Header compression
Additionally, HTTP/2 offers header compression that makes a client able to squeeze in more requests earlier in a TCP connection lifetime. In the early slow-start period of a new TCP connection it can be valuable to cram in more requests so that the responses come back earlier. HTTP headers are extremely repetitive in their nature.
Server push
A HTTP/2 server can send data to the client as if the client asked for it, before the client asks for it! If the server thinks the client is likely to want/need that too, and thus a half RTT can be saved.
Starting with HTTP/2, both headers and HTTP response content could be compressed. With HTTP/1.1, headers are never compressed contrary to the content (specified with Content-Encoding header).
Multiplexing is related to server-push. Actually, when the server sends an HTML page, it can use the same connection to push additional resources like css and javascript files. If the HTML page needs to load those additional scripts, no more request will be sent to the server as they were already sent previously.
I've heared that you (in some cases) can prevent timeouts by sending the HTTP-header back to the client before the whole HTTP-body is prepared.
I know that this is impossible using gzip ... but is this possible using HTTPS?
I read in some posts that the secure part of HTTPS is done in the transport-layer (TLS/SSL) - therefore it should be possible, right?
Sorry for mixing gzip in here - it's a completely different level - I know ... and it may is more confusing than giving an example ;)
In HTTP 1.1 it's possible to send the response header before preparing of the body of the response is completed . To do this one normally uses chunked encoding.
Some servers also stream the data as is by not specifying the content length and indicating the end of stream by closing connection, but this is quite a brutal way to do things (chunked encoding was designed exactly for sending the data before it's completely available).
As HTTP(S) is HTTP running over SSL/TLS channel, TLS doesn't affect the above behavior in any way.
Yes, you can do this. HTTPS is just HTTP over an TLS/SSL transport, the HTTP protocol is exactly the same.
HTTP 1.1 supports keep alive connections, connections are not closed until "Connection: close" is sent.
So, if the browser, in this case firefox has network.http.pipelining enabled and network.http.pipelining.maxrequests increased isn't the same effect in the end?
I know that these settings are disabled because for some websites this could increase load but I think a simple http header flag could tell the browser that is ok tu use multiplexing and this problem can be solved easier.
Wouldn't be easier to change default settings in browsers than invent a new protocol that increases complexity especially in the http servers?
SPDY has a number of advantages that go beyond what HTTP pipelining can offer, which are described in the SPDY whitepaper:
With pipelining, the server still has to return the responses one at a time in the order they were requested. This can be a problem if the client requests a resource that's dynamically generated before one that is static: the server cannot send any of the "easy" static responses until the dynamically generated one has been generated and sent. With SPDY, responses can be returned out of order or in parallel as they are generated, lowering the total time to receive all resources.
As you noted in your question, not all servers are able to deal with pipelining: it's not just load, some servers actually behave incorrectly when the client requests pipelining. Using a header to indicate that it's okay to do pipelining is too late to get the maximum benefit: you are already receiving the first response at that point, so while you can use it on future connections it's already too late for this one.
SPDY compresses headers using an algorithm which is specific to that task (stateful and with knowledge of what is normally in HTTP headers); while yes, SSL already includes compression, just compressing them with deflate is not as efficient. Most HTTP requests have no bodies and only a short GET line, so the headers make up virtually the entire request: any compression you can get is an improvement. Many responses are also small compared to their headers.
SPDY allows servers to send back additional responses without the client asking for them. For example, a server might start sending back the CSS for a page along with the original HTML, before the client has had a chance to receive and parse the HTML to determine the stylesheet URL. This can speed up page loads even further by eliminating the need for the client to actually parse the HTML before requesting other resources needed to render the page. It also supports a less bandwidth-heavy version of this feature where it can "hint" about which resources might be needed, and allow the client to decide: this allows, for example, clients that don't care about images to not bother to request them, but clients that want to display images can still request the images using the given URLs without needing to wait for the HTML.
Other things too: see William Chan's answer for even more.
HTTP pipelining is susceptible to head of line blocking (http://en.wikipedia.org/wiki/Head-of-line_blocking) at the HTTP transaction level whereas SPDY only has head of line blocking at the transport level, due to its use of multiplexing.
HTTP pipelining has deployability issues. See https://datatracker.ietf.org/doc/html/draft-nottingham-http-pipeline-01 which describes a number of different workarounds and heuristics to mitigate this. SPDY as deployed in the wild does not have this problem since it is generally deployed over SSL (port 443) using NPN (http://technotes.googlecode.com/git/nextprotoneg.html) to negotiate SPDY support. SSL is key, since it prevents intermediaries from interfering.
SPDY has header compression. See http://dev.chromium.org/spdy/spdy-whitepaper which discusses some benchmark results of the benefits of header compression. Now, it's useful to note that bandwidth is less and less of an issue (see http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/), but it's also useful to remember that bandwidth is still key for mobile. Check out https://developers.google.com/speed/articles/spdy-for-mobile which shows how beneficial SPDY is for mobile.
SPDY supports features like server push. See http://dev.chromium.org/spdy/spdy-best-practices for ways to use server push to improve cacheability of content and still reduce roundtrips.
HTTP pipelining has ill-defined failure semantics. When the server closes the connection, how do you know which requests have been successfully processed? This is a major reason why POST and other non-idempotent requests are not allowed over pipelined connections. SPDY provides semantics to cancel individual streams on the same connection, and also has a GOAWAY frame which indicates the last stream to be successfully processed.
HTTP pipelining has difficulty, often due to intermediaries, in allowing deep pipelines. This (in addition to many other reasons like HoL blocking) means that you still need to utilize multiple TCP connections to achieve maximal parallelization. Using multiple TCP connections means that congestion control information cannot be shared, that compression contexts cannot be shared (like SPDY does with headers), is worse for the internet (more costly for intermediaries and servers).
I could go on and on about HTTP pipelining vs SPDY. But I'd recommend just reading up on SPDY. Check out http://dev.chromium.org/spdy and our tech talk on SPDY at http://www.youtube.com/watch?v=TNBkxA313kk&list=PLE0E03DF19D90B5F4&index=2&feature=plpp_video.
See Difference between HTTP pipeling and HTTP multiplexing with SPDY