I am trying to estimate bandwidth usage of a XMPP application.
The application is receiving a 150-bytes ping each second, and answering with a ping of the same size.(*)
However, when I measure the data usage, I get something like 900 bytes per ping (and not the 300 expected)
I have a suspicion this might relate to something layer bellow (TCP? IP?) and datagram sizes. But, so far, reading the TCP/IP guide did not lead me anywhere.
Another hypothesis would be that this overhead comes from XMPP itself, somehow.
Can anyone enlighten me ?
(*) to get this "150 bytes" I counted the number of chars in the <iq> (the xml representation of the ping)
I am using TLS, but not BOSH (actually, BOSH on the other connection: I am measuring results in the android client, and the pings are coming from a web application, but I think that should not matter)
The client is Xabber, running on android
Lets try to calculate the worst-case overhead down to the IP level.
For TLS we have:
With TLS 1.1 and up, in CBC mode: an IV of 16 bytes.
Again, in CBC mode: TLS padding. TLS uses blocks of 16 bytes, so it may need to add 15 bytes of padding.
(Technically TLS allows for up to 255 bytes of padding, but in practice I think that's rare.)
With SHA-384: A MAC of 48 bytes.
TLS header of 5 bytes.
That's 84 extra bytes.
TCP and IP headers are 40 bytes (from this answer) if no extra options are used, but for IPv6 this would be 60 bytes.
So you could be seeing 84 + 60 + 150 = 294 bytes per ping.
However, on the TCP level we also need ACKs. If you are pinging a different client (especially over BOSH), then the pong will likely be too late to piggyback the TCP ACK for the ping. So the server must send a 60 byte ACK for the ping and the client also needs to send a 60 byte ACK for the pong.
That brings us to:
294 + 60 + 294 + 60 = 708
900 still sounds a lot too large. Are you sure the ping and the pong are both 150 bytes?
Related
When transferring, say, 1GB worth of data over the internet this data is split into packets, each packet containing a small piece of data and aach of these packets are part of a frame.
Eg. Windows reports that you are transferring the file in 100kb/s over a TCP connection, but this appears to be the amount of data from the file being transferred per second, and does not seem to include the ip or tcp header, or ethernet frame.
What is the actual amount of traffic on the network needed to transfer at this speed? Or is that data actually already included in the transfer speed, but just small enough that it makes no significant difference?
Also, IP supports up to 1500 byte / packet (I think?), but what is the common size of data packets when loading, say, an HD image on reddit?
Sorry for the rather basic questions I probably should have figured out myself by now...
It depends on where you look at the transfer rate:
Task Manager will report all the transferred bytes (i.e. the sum of all the packets including their headers).
A file transfer program will report the transmitted payload.
Task Manager
If you look at Task Manger / Network, you can see the transmitted bytes together with the number of transmitted packets (unicast or non-unicast).
That data comes from the network driver (or at least something close to it), so it makes sense to report the total amount of data here (otherwise each packet would need to be inspected to calculate the payload).
There is also a graph showing the transfer rate. Those numbers could easily be compared with the reported numbers in file transfer software.
File transfer program
A file transfer program on the other hand, does not know the details about the packets being created in the lower layers (those could be any size). So the only option here is to report the amount of transmitted payload data / part of the file, which also makes more sense to the user.
Network packets
On normal networks (there could also be jumbo frames), a TCP-packet (full ethernet frame) is around 1500 bytes when fully loaded (on my system (IPv4) the packets are 1514 bytes with a total header size of 54 bytes -- 14 for the Ethernet header, 20 for the IP header and 20 for the TCP header). Those could be split in smaller packets along the way in the network, but in most cases they won't.
Transfer rate
When transferring a file (or other large datastream), on average 2 full packets (1514 bytes) will be sent each time, and 1 small packet (54 bytes) is received (the [ACK] packet). In this optimal case we have 2 x 1460 payload, and 2 x 54 bytes of overhead on the sending side + 54 bytes on the receiving side. When comparing to the maximum transfer rate of the internet connection, we also have to consider some latency.
Not all transmissions are optimal:
There could be packets that never arrived or where the checksum was wrong, so a retransmit would be needed.
In some cases data could be sent in smaller parts, causing a higher overhead/payload ratio (but with small chunks Nagle's algorithm could take care of that).
Certain software could be reading the file contents into small buffers (say 4096 bytes). Those could then be split in 2 x 1460 and 1 x 1176, introducing some extra overhead.
Conclusion
It's hard to tell or calculate the exact ratio transferred_bytes/payload. It depends on the quality of the internet connection (lost packets, retransmits), the software or API calls used to transfer the data, and even the underlying network (small frames vs jumbo frames for example).
A typical full-size TCP/IPv4 packet on the Internet is of size 1500B (maximum transmission unit (MTU)), of which (minimum) 20B are of TCP header and (minimum) 20B are of IPv4. This MTU was chosen to be compatible with Ethernet. Furthermore, there are application headers (e.g., HTTP for web, SIP/RTP/RTCP for voice call, etc.) included in this packet. The minimum MTU is 576B for IPv4 and 1280B for IPv6. One can see MTU on Linux with ifconfig command.
The best way to figure these values is by using a pcap tool/network analyzer such as Wireshark. Also refer to wiki pages or a good networking book for headers and fields of the protocols.
I'm pretty sure that the reported transer rate doen't include all the headers and overhead of the different layers in the protocal stack, since the reported thruput usually comes from some user-space application which would only get the actual data from the network stream object. It would need to do additional work to find out about all the headers and frames and other overhead that occurred in the different layers and affected the actual physical transmission.
I am building an app where my phone frequently send data to my server. Since I would be using my mobile data, I was wondering how much data it cost to set up (and tear down?) a TCP connection to my server.
TCP Three-way handshake
Device 1 sends its TCP sequence number and maximum segment size to Device 2.
Device 2 responds by sending its sequence number and maximum segment size to Device 1.
Device 1 acknowledges receipt of the sequence number and segment size information.
Each packet is composed of an IP header and data (payload). In this case, the data section contains TCP. The TCP header contains various fields including the source and destination ports, sequence and acknowledgment numbers, window size, TCP flags, urgent pointer, and reserved bits.
Like the IP header, the TCP header may also contain options. (Note that TCP options and IP options are two different things.) Because the TCP options change the length of the TCP header, the length is set in the header.
IPv4 header is five 4-byte chunks, or 20 bytes total.
TCP typically usually uses 24 bytes of the header for handshake (first two packets) and about 20 for normal packet transmission.
Maximum Segment Size (MSS): 4 bytes
Window Scale (WSCALE): 3 bytes
Timestamp (TS): 10 bytes
No Operation (NOP): 1 byte
Selective Acknowledgment Permitted (SackOK): 2 bytes
Selective Acknowledgment Data: 10 bytes (plus 8 bytes for each additional pair of sequence numbers)
Terminating a Connection
Even though establishing a connection using 3-way handshake requires only 3 packets to be transmitted, tearing down one requires 4!
In the first frame the client sends a FIN that is accompanied by an ACK. The FIN parameter is set, it will inform the server that it has no more data to send.
The response (2nd frame) would be simply the server acknowledging the FIN sent from the client.
Even though TCP has established connections between the two computers, the connections are still independent of one another. Therefore, the server will also transmit a FIN to the client.
You guessed it right ... the client would ACK the FIN of the server in the last forth packet.
The offset of each of the frames is typically 20 bytes.
To sum it up.
Establishing a connection: ~ 128-136 bytes
Tearing down a connection: ~ 160 bytes
If you plan to use TLS / SSL handshake, this is estimated to be between 4.5k-6.5k.
Note: Please also take a look at TCP/IP Header Compression
Sources:
Inside the TCP Handshake
Explanation of the Three-Way Handshake via TCP/IP
Studying Normal Traffic, Part Three: TCP Headers | Symantec Connect
I was reading rfc791 and trying to understand the relations with MTU as well as with the minimum packet size for IPv4. Here are two quotes from the rfc:
"All hosts must be prepared to accept datagrams of up to 576 octets (whether they arrive whole or in fragments)."
And
"Every internet module must be able to forward a datagram of 68 octets without further fragmentation. This is because an internet header may be up to 60 octets, and the minimum fragment is 8 octets."
Do I understand correctly that first related only to hosts, i.e. only hosts must be able to process minimum packet size of 576 bytes, while the second statement defines the mi packet size for a router? But of so, then it is possible to have a router not being able to receive a packet of 68 bytes for himself ?
Or I'm missing something very fundamental?
Thanks.
Mark
The 576 octet is a "least maximum". In other words, a host needs to be prepared to have a maximum packet size of no less than 576 octets. It can be bigger than that, such as the 1518 limit used by most (non-jumbo) Ethernet devices, but not any smaller.
Anything that is set up to forward packets must not split them into chunks of less than 68 octets.
By standard, 576 Bytes is "minimum MTU" supported over IP infrastructure. This means, any host/router must support this value and any IP packets can be smaller than 576 Bytes (atleast 68 Bytes) which can move IP world without fragmentation.
HTH
The first relates to accepting; the second relates to forwarding.
then it is possible to have a router not being able to receive a packet of 68 bytes for himself
That doesn't make any sense. A host must be able to accept datagrams of up to 576 octets.
My application connects to the server using TCP and system is working fine. Using Wireshark I noticed that some TCP packets of 60 bytes are sent with no data. Is this normal?
as part of TCP transmissions and handshake are there some packets sent with no data?
Thanks
Kumar
There are the ACK packets that carry no data and only acknowledge received data. When using Wireshark it should display these "flags" on the empty packets.
To be more accurate you should show a screenshot of the wireshark capture, so we know what the size you mean is.
Meanwhile I dissected an ACK packet of IPv4 TCP traffic here and these are my results:
Protocol Size Description
Ethernet protocol 14 byte 2 MAC adresses (6 byte), Protocoll type (2 byte)
Internet protocol 20 byte The default header size
TC protocol 32 byte The default header size (20 byte) + options (12 byte)
_____________________________________________________________________________________
Total 66 byte
without EP 52 byte (Probably the size the OP is talking about)
without EP, Opts 40 byte (The size talked about in the comments)
The options in my case were 2 NOPs (each 1 byte) and two timestamps (each 5 byte??). The order of the protocols is the same as in Wireshark.
Wireshark splits the information down to each field, so you can see what takes up how much space.
I am analyzing wireshark log files, when I make a request to a web page using firefox through a proxy server.
Following are details of connection establishment:
I have noted "maximum segment size" when I open options branch in the TCP segment details of the [SYN] message from my PC to the proxy server - it says 1460 bytes
Similarly, maximum segment size eof the [SYN,ACK] message from the proxy server to my PC - it says 1460 bytes
After establishing the TCP connection, should not each of the TCP frames sent from proxy server to my PC be of 1460 bytes? I am puzzled that why are they 590 bytes. Please advice how the 590 size is being set
A plausible explanation is that 590 turns out to be the Path MTU for the particular connection.
In other words whereby the client (one of the end nodes of the connection)accepts packets of a maximum of of 1460 bytes payload, some node(s) on the way accepts smaller packets. For efficiency purposes, the Path MTU Discovery allows the originator of a packet to size it so that it would fit the smaller MTU encountered on the path, and hence avoid fragmentation.
BTW:
1460 is a very common MTU (well MSS), because it it corresponds to 1500, Ethernet v2's maximum, minus 20+20= 40 bytes for the IP header overhead)
See the following Wikipedia entry for an overview of MTU (Maximum Transmission Unit) and a basic description of the Path MTU Discovery method (Basically setting the the DF i.e. do-not-fragment flag and relying on the ICMP ""Destination Unreachable (Datagram Too Big)" messages to detect that some node on the way couldn't handle the packet, and hence try with smaller size until it goes through).
Also, I suggest inspecting the packets when the connection is to a different host, maybe a peer on the very same network segment, without going through the proxy mentioned. Chances are you will then start seeing 1460 bytes frames.