I wanted to know what all factors do I need to check while analysing latency issue on the firewall Wireshark's capture.?
I know about timestamps (before previous packet reached).. But nothing after than that
If you are talking about latency of HTTP transaction, you can consider 3 aspects:
roundtrip time, typically it's the time from your HTTP request to the TCP ACK for the request
Initial response time: that's the time between your HTTP request and first packet in the HTTP response.
Total response time: that's the time between your HTTP request and last packet of HTTP response (Wireshark will tell you the last packet of response since that's when you see the full http response)
Good luck.
Related
I'm an application developer, looking into trying to understand how HTTP request/responses are passed at the transport layer, and found some good resources that have been helping me, but there are still some questions I can't track down
How is HTTP converted to TCP?
This thread helped me understand that as a request is passed down each layer, the layer is "encapsulating" the payload with headers until it's sent over the wire
So my current understanding is that the HTTP Request is really a payload, which then the TCP layer will throw some headers on. These headers seem to mostly be used to track the order of the packets of the messages that will be going back-and-forth
TCP Headers:
And then IP headers are slapped on (https://www.thegeekstuff.com/2012/03/ip-protocol-header/)
Which just track the source/destination and size of the packet
It's clear that a every HTTP request does not map 1:1 to a packet, based on what I have found it looks like the requests can be broken up into pieces to be streamed over TCP
If the request needs to be broken up, where is the code that does that?
I'm having troubles with understanding how SPDY can solve HOL blocking.
Quote from: http://chimera.labs.oreilly.com/books/1230000000545/ch02.html#TCP_HOL
To understand why that is the case, recall that every TCP packet carries a unique sequence number when put on the wire, and the data must be passed to the receiver in-order (Figure 2-8). If one of the packets is lost en route to the receiver, then all subsequent packets must be held in the receiver’s TCP buffer until the lost packet is retransmitted and arrives at the receiver. Because this work is done within the TCP layer, our application has no visibility into the TCP retransmissions or the queued packet buffers, and must wait for the full sequence before it is able to access the data. Instead, it simply sees a delivery delay when it tries to read the data from the socket. This effect is known as TCP head-of-line (HOL) blocking.
So HOL blocking exists because TCP guarantees in-order delivery. But here the user igrigorik says that SPDY allows for the packet to come in different order. But isn't SPDY just a HTTP replacement? Meaning it still runs over TCP (from here).
HOLB has several causes, of which packet retransmission is one, but it's not the one relevant to HTTP and SPDY.
The one relevant to HTTP and SPDY is the fact that in HTTP 1.x multiple requests must be responded in order.
Imagine a HTTP client that sends to a server 2 requests over the same TCP connection, and that the first response is "large" in content length while the second response is "small" in content length.
Due to the nature of the HTTP 1.x protocol, the second response must wait for the first response to complete. The second response is head-of-line blocked by the first response.
With multiplexed protocols like SPDY and HTTP 2, instead, this type of HOLB does not exist, because the second "small" response can arrive to the client well before the first "large" response (they can even be interleaved).
The diagram of the question you referenced above explains it graphically.
Ilya, in his response, was not referring to TCP packets, but to HTTP "packets" when he was saying that they can be out of order. Imagine a "packet" made of the HTTP headers, and a "packet" made of POST data to be uploaded to the server (or, in a response, "packets" made of the data to be downloaded to the client).
In HTTP 1.x, these HTTP "packets" must be in order (first all the HTTP "packets" of request 1, then all the HTTP "packets" of request 2; or first all the HTTP "packets" of response 1 and then all the HTTP "packets" of response2), while in SPDY and HTTP 2 they may be out of order or even interleaved.
The lack of this kind of HOLB in SPDY and HTTP 2 makes these protocols more efficient than HTTP 1.x.
The HOLB caused by TCP retransmissions affects any TCP based protocol, included multiplexed protocols like SPDY and HTTP 2, and duplex protocols like HTTP 1.x.
I was just reading this Wikipedia article on HTTP pipelining and from the diagram it appears that responses can be sent concurrently on one connection. Am I misinterpreting the diagram or is this allowed?
Section 8.1.2.2 of RFC 2616 states:
A server MUST send its responses to those requests in the same order
that the requests were received.
Whilst that stops short of explicitly ruling out concurrent responses, it does not mention a need to ensure that responses must not only start in the correct order with relation to requests, but also finish in the correct order.
I also cannot imagine the practicalities of dealing with concurrent responses - how would the client know to which response the received data applies?
Therefore my interpretation of the RFC is that whilst additional requests can be made whilst the response to the first request is being processed, it is not allowedfor the client to send concurrent requests or the server to send concurrent responses on the same connection.
Is this correct? I've attached a diagram below to illustrate my interpretation.
It would prevent the problems I mentioned from occurring, but it does not appear to completely align with the diagram in Wikipedia.
Short answer: Yes, clients and servers can send requests and responses concurrently.
However, a server cannot send multiple responses to one request, i.e. the request response pattern still applies. RFC 2616 (and the Wikipedia article you are refering to) simply state that a client does not need to wait for the server's response to send an additional request on the same connection. So the requests in your diagram look good :).
But the server doesn't have to wait for each of its responses to finish before it can start transmission of the next response. It can just send the responses to the client as it receives the client's requests. (Which results in the diagram shown in the Wikipedia article.)
How does the client know to which request a response applies?
Well, let's ignore that whole network delay stuff for a minute here and assume that pipelined request or response messages arrive at once but only after all of them have been sent.
The client sends its requests in a certain order (without waiting for responses inbetween requests).
The server receives the requests in the same order (TCP guarantees that) all at once.
The server takes the first request message, processes it, and stores the response in a queue.
The server takes the second request message, processes it, and stores the response in a queue.
(You get the idea...)
The server sends the contents of that queue to the client. The responses are stored in order so the response to the first request is at the beginning of that queue followed by the response to the second request and so on...
The client receives the responses in the same order (TCP guarantees that) and associates the first response with the first request it made and so on.
This still works even if we don't assume that we receive all the messages at once because TCP guarantees that the data that was sent is received in the same order.
We could also ignore the network completely and just look at the messages that are transferred between server and client.
Client -> Server
GET /request1.html HTTP/1.1
Host: example.com
...
GET /request2.html HTTP/1.1
Host: example.com
...
GET /request3.html HTTP/1.1
Host: example.com
...
Server -> Client
HTTP/1.1 200 OK
Content-Length: 234
...
HTTP/1.1 200 OK
Content-Length: 123
...
HTTP/1.1 200 OK
Content-Length: 345
...
The great thing about TCP is that this particular stream of messages always looks the same. You can send all of the requests first and then receive the responses; you can send request 1 first, receive the first response, send the remaining requests, and receive the remaining responses; you can send the first and part of the second request, receive part of the first response, send the remaining requests, receive the remaining responses; etc. Because TCP guarantees to keep the order of the transmitted messages, we can always associate the first request with the first response and so on.
I hope this answers your question...
I'm sifting through some network traces and noticed on my own machine that when I connect over HTTP, packets look something like:
client --> server: GET
server --> client: tcp ack
server --> client: HTTP response
client --> server: tcp ack
However, I looked at some CIFS (SMB) traces I have saved from a few years back. I see things like:
client --> server: Create Request
server --> client: Create response (This packet also acks the request)
At a high level, I'm wondering why the difference - what is causing the different behaviors? What is controlling whether the application response is placed on the request ack or another packet: the application or OS?
This behavior is dependent on both the OS and the application. In linux, the kernel doesn't send an ACK directly, but instead waits a fixed number of milliseconds (around 200), hoping that is has some data to send back and can let the ACK piggyback the data.
If the timer goes off, then the ACK is sent immediately.
Example 1.
Client sends the GET request.
Server tries to create a http response, but before it does that 200ms are gone
and it must send the ACK before the http response.
Example 2.
Client sends the GET request.
Server creates a http response within the timer limit, and the ACK can piggyback
the data.
Meaning, if your application got slower at generating that response, the ACK will be send without piggybacking on the data. And also depending on the OS, the delay timer can be higher / lower and once again changing how ACK's are sent.
If I make multiple HTTP Get Requests to the same server and get HTTP 200 OK responses to each one how do I tell which request maps to which response using Wireshark?
Currently it looks like an http request is made, and the next HTTP 200 OK response is quickly received so everything is in a the proper sequence. I have seen things to the contrary however. For example using the Google Maps API v2 I've made several requests for location information and then the information is received in an arbitrary order (closely resembling the order in which I requested it, but not necessarily perfect.)
So my intuition is I cannot assume that my responses will be received in a specific order, even though they may be in order most of the time. So I'm wondering how I can determine this order from the response.
Update: Clarification as to what I need. I just need to know that the server has received the request. It seems like I need to do this by looking at sequence numbers and perhaps even ACKS. The reasoning behind this approach is I'm basically observing a web app and checking it is sending the information and the information is being received.
Update: This has nothing to do with wireshark specifically. I believe it is confusing people so I removing it from the title. It has to do with the HTTP protocol on top of the TCP/IP protocol and how we map responses to requests.
Thanks.
After you have stopped capturing packets follow this steps:
position the cursor on a GET request
Open the Analyze menu
click "Follow TCP Stream"
You get a new window with requests and responses in sequence.
While I was googling for a complete different question, I saw this one and I think I can provide a more complete answer :
HTTP dictates that responses must arrive in the order they were requested, Therefore, if you are looking at a single TCP connection at a given time you should be seeing :
Request ; Response ; Request ; Response ...
Also in HTTP/1.1, there is support for "Pipeline" where the client doesn't have to wait for responses to arrive in order to issue the next request. What could be observed in such cases is :
Request ; Response ; Request ; Request ; Response ; Response ; Request ; Response
In the HTTP response itself, there is no reference to the specific request that triggered it.
Filipo's suggestion is classic when debugging / observing a single TCP connection, but, when observing multiple TCP connections, you can't click the follow TCP Stream because you'd have to do it for each connection.
If you have many TCP connections, and many requests/responses you will have to look at TCP Source port in the request packet, and the TCP dest port in the response packet to know which response is related to each tcp connection, and then apply the HTTP request/response order rules.
Also, Wireshark CAN decompress the response body, and it will do it automatically if all the response body has arrived, but it will do so NOT in the Follow TCP Stream.
I always use Wireshark to debug HTTP.
Seems like this ability is not provided by the HTTP protocol at the application layer so I must go down to the transportation layer to determine this. In my case the TCP/IP layer using sequence numbers.
HTTP only presumes a reliable
transport; any protocol that provides
such guarantees can be used; the
mapping of the HTTP/1.1 request and
response structures onto the
transport data units of the protocol
in question is outside the scope of
this specification.
Read more:
http://www.faqs.org/rfcs/rfc2616.html#ixzz0e20kxKcz
Don't use Wireshark to debug HTTP, use an HTTP debugger such as Fiddler2