Suppose when we request a resource over HTTP, we get a response as shown below:
GET / HTTP/1.1
Host: www.google.co.in
HTTP/1.1 200 OK
Date: Thu, 20 Apr 2017 10:03:16 GMT
...
But when a browser requests many resources at a time, how can it identify which request got which response?
when a browser requests many resources at a time, how can it identify which request got which response?
A browser can open one or more connections to a web server in order to request resources. For each of those connections the rules regarding HTTP keep-alive are the same and apply to both HTTP 1.0 and 1.1:
If HTTP keep-alive is off, the request is sent by the client, the response is sent by the server, the connection is closed:
Connection 1: [Open][Request1][Response1][Close]
If HTTP keep-alive is on, one "persistent" connection can be reused for succeeding requests. The requests are still issued serially over the same connection, so:
Connection 1: [Open][Request1][Response1][Request3][Response3][Close]
Connection 2: [Open][Request2][Response2][Request4][Response4][Close]
With HTTP Pipelining, introduced with HTTP 1.1, if it is enabled (on most browsers it is by default disabled, because of buggy servers), browsers can issue requests after each other without waiting for the response, but the responses are still returned in the same order as they were requested.
This can happen simultaneously over multiple (persistent) connections:
Connection 1: [Open][Request1][Request2][Response1][Response2][Close]
Connection 2: [Open][Request3][Request4][Response3][Response4][Close]
Both approaches (keep-alive and pipelining) still utilize the default "request-response" mechanism of HTTP: each response will arrive in the order of the requests on that same connection. They also have the "head of line blocking" problem: if [Response1] is slow and/or big, it holds up all responses that follow on that connection.
Enter HTTP 2 multiplexing: What is the difference between HTTP/1.1 pipelining and HTTP/2 multiplexing?. Here, a response can be fragmented, allowing a single TCP connection to transmit fragments of different requests and responses intermingled:
Connection 1: [Open][Rq1][Rq2][Resp1P1][Resp2P1][Rep2P2][Resp1P2][Close]
It does this by giving each fragment an identifier to indicate to which request-response pair it belongs, so the receiver can recompose the message.
I think you are really asking for HTTP Pipelining here. This is a technique introduced in HTTP/1.1, through which all requests would be sent out by the client in order and be responded by the server in the very same order. All the gory details are now in RFC 7230, sec. 6.3.2.
HTTP/1.0 had (or has) a comparable method known as Keep Alive. This would allow a client to issue a new request right after the previous has been answered. The benefit of this approach is that client and server no longer need to negotiate through another TCP handshake for a new request/response cycle.
The important part is that in both methods the order of the responses matches the order of the issued requests over one connection. Therefore, responses can be uniquely mapped to the issuing requests by the order in which the client is receiving them: First response matches, first request, second response matches second request, … and so forth.
I think the answer you are looking for is TCP,
HTTP is a protocol that relies on TCP to establish connection between the Client and the Host
In HTTP/1.0 a different TCP connection is created for each request/response pair,
HTTP/1.1 introduced pipelining, wich allowed mutiple request/response pair, to reuse a single TCP connection, to boost performance (Didnt work very well)
So the request and the corresponding response are linked by the TCP connection they rely on,
It's then easy to associate a specific request with the response it produced,
PS: HTTP is not bound to use TCP forever, for example google is experimenting with other transport protocols like QUIC, that might end up being more efficient than TCP for the needs of HTTP
In addition to the explanations above consider a browser can open many parallel connections, usually up to 6 to the same server. For each connection it uses a different socket. For each request-response in each socket it is easy to determine the correlation.
In each TCP connection, request and response are sequential. A TCP connection can be re-used after finishing a request-response cycle.
With HTTP pipelining, a single connection can be multiplexed for
multiple overlapping requests.
In theory, there can be any number[*1] of simultaneous TCP connections, enabling parallel requests and responses.
In practice, the number of simultaneous connections is usually limited on the browser and often on the server as well.
[*1] The number of simultaneous connections is limited by the number of ephemeral TCP ports the browser can allocate on a given system. Depending on the operating system, ephemeral ports start at 1024 (RFC 6056), 49152 (IANA), or 32768 (some Linux versions).
So, that may allow up to 65,535 - 1023 = 64,512 TCP source ports for an application. A TCP socket connection is defined by its local port number, the local IP address, the remote port number and the remote IP address. Assuming the server uses a single IP address and port number, the limit is the number of local ports you can use.
I have two questions on HTTP Connection close:
If a client sends a HTTP request with Connection: close to HTTP Server, Is it the HTTP Server or client responsibility to send TCP FIN after response is received by client?
If a client sends a bad formatted HTTP request, and server sends a 400 BAD REQUEST, is it best practice to close the connection by server (even though the HTTP request has connection: keep-alive) or is it good practice to keep the connection still active?
Thanks in advance for answering my queries?
When the server receives a 400 Bad Request, it is going to send the response with the keep-alive header because if the client feels like sending another request, then they can use a pre-existing connection (this connection is shut down within a certain amount of time, it has an expiration date). The Keep-Alive Header is more about not saturating the network with TCP connection demands. You basically say "I am going to talk to you, for 2 minutes, whatever you send me, I'll answer you though this connection"
The server is only an object that receives commands from an user. You ask him, he does or not. The TCP FIN is something you send to the server to shut down the connection, but you choose when you don't want to communicate with him anymore. The client transmits the first FIN, and receives an ACK to ensure that the server got it. Then the server launches its own FIN, and waits for the ACK. If everything is okay, you and your server are no longer friends.
Suppose I initiate a TCP connection for HTTP request and response. After the completion of three way handshaking process, I now send a request to the server for a HTTP packet with SYN flag set. The server responses to my request with ACK flag. Upon completion of the response, it will send me a FIN flag. Now how can I determine the size of the packet that the server send me during the request and response? Is there any process to do so?
Suppose I initiate a TCP connection for HTTP request and response. After the completion of three way handshaking process, I now send a request to the server for a HTTP packet with SYN flag set.
No you don't. The SYN was in the first packet sent in the three-way handshake. That's done. At best you will be piggy-backing the HTTP request on the final ACK packet of the connect handshake.
The server responses to my request with ACK flag.
No, the server responds to your HTTP request with an HTTP response. The ACK is irrelevant at the HTTP level.
Upon completion of the response, it will send me a FIN flag.
No it won't, unless you're speaking HTTP 1.0 or you have set the Connection: close header. Otherwise it will keep the connection open for HTTP keepalive.
Now how can I determine the size of the packet that the server send me during the request and response? Is there any process to do so?
The server didn't send you anything during the request. The size of the response body is given by the Content-length header or the sum of the chunked-encoding section sizes if that's used. You need to read the HTTP 1.1 RFC about this, it's non-trivial.
You also need to stop confusing HTTP with TCP. HTTP is an application protocol layered on top.
I noted that fiddler sends "connection:close" header when the client sends a CONNECT request to initiate secure connection along with the "200 connection established" message.
CONNECT request to a forward HTTP proxy over an SSL connection?
As explained in above question, the connection should be kept-alive between the client and the proxy so that the client can subsequently sent the actual request.
Why does fiddler sends the close header? wouldn't the client close the connection because of the header instead?.
Any Connection header in the successful response to the CONNECT request does not make any sense and gets ignored. CONNECT will establish a tunnel, which only ends with the end of the TCP connection. But a Connection header would make sense with an unsuccessful CONNECT, because with close the client would need to start a new TCP connection and with keep-alive (implicit with HTTP/1.1 response) it can reuse it with another request.
Connection: Close means that the connection will be closed after the request completes. Since the request in this scenario only completes when the HTTPS connection is closed, this is exactly the behavior you want for this sort of request.
Arguably, using Connection: keep-alive on a CONNECT request is invalid, since there's no legal way for the connection to be kept-alive after the tunnel is closed.
Protocol used: HTTP 1.1
Header includes 'Transfer-Encoding: chunked'
On sending subsequent requests (same request to the same server), containing 'Transfer-Encoding: chunked', I see new TCP socket being opened for each request.
Should the same socket not be used for requests? Why is the socket being terminated each time?
Note:
Requests sent using fiddler