I am using raw sockets to communicate with a TCP server. For the purposes of my project, I need to emulate a TCP timeout.
Whenever a timeout occurs, server re-transmits the first lost packet. On receiving ACK for this packet, the sever re-transmits the second packet and also sends a packet that was previously unseen (due to F-RTO algorithm). In order to stop F-RTO, I need to send duplicate ACK for the later packet.
Lets says the congestion window is 20 at the time of time out. Server will send packet 1 and I will ACK packet 1. Server will then send packet 2 and packet 21. I will ACK packet 2 and send duplicate ACK for packet 21 to stop F-RTO. The problem that I am having is that although client is sending 2 ACKs, for some unknown reasons server is only getting one ACK. As a results it gets stuck in F-RTO.
Wireshark shows client sends multiple duplicate ACKs but from server side I can only see a single ACK. Since the second ACK is duplicate to first one, their fields and checksums are same. Can some one please help me out?
Related
I am working on a high-performance TCP server, and I see the server not processing fast enough on and off when I pump high traffic using a TCP client. Upon close inspection, I see spikes in "delta time" on the TCP server. And, I see the server sending an ACK and 0.8 seconds later sending PSH,ACK for the same seqno. I am seeing this pattern multiple times in the pcap. Can experts comment on why the server is sending an ACK followed by a PSH,ACK with a delay in between?
TCP SERVER PCAP
To simplify what ACK and PSH means
ACK will always be present, it simply informs the client what was the last received byte by the server.
PSH tells the client/server to push the bytes to the application layer (the bytes forms a full message).
The usual scenario you are used to, is more or less the following:
The OS has a buffer where it stores received data from the client.
As soon as a packet is received, it is added to the buffer.
The application calls the socket receive method and takes the data out of the buffer
The application writes back data into the socket (response)
the OS sends a packet with flags PSH,ACK
Now imagine those scenarios:
step 4 does not happen (application does not write back any data, or takes too long to write it)
=> OS acknowledge the reception with just an ACK (the packet will not have any data in it), if the application decides later on to send something, it will be sent with PSH,ACK.
the message/data sent by the server is too big to fit in one packet:
the first packets will not have PSH flag, and will only have the ACK flag
the the last packet will have the flags PSH,ACK, to inform the end of the message.
I am very new to networking, so this might sound simple. Though I have tried to look here and here and here and have got few basics of TCP, there are few questions whose answers I am not certain about.
Is a request and response part of 2 different TCP establishments. To explain that :
Is a connection established, kept alive until all packets are delivered, request sent and connection closed for each request and same happens for its response.
or
A connection is opened, request sent, connection kept alive, response arrives and connection closed.
Is the ACK number always 1 + sequence number of sent segment.
Is a request and response part of 2 different TCP establishments
You need just 3 packets to handshake and establish a bidirectional TCP connection. So no, you do not establish TCP connection for receiving and sending parts.
On the other hand, there is a sutdown() system call which allows to shutdown a part of the bidirectional connection. See man shutdown(2). So there is a possibility to establish a unidirectional connection by opening a bidirectional and then shutdown one of the sides.
Is the ACK number always 1 + sequence number of sent segment.
We usually do not send ACK for every received packet. There are also selective ACKs, retransmissions etc. So in general, the answer is no, the ACK number is not always seq + 1.
On the other hand, if you are sending a small amount of data and waiting for the confirmation, no errors or packet loss occurred, most probably there will be just one packet with that data and one ACK with seq + 1.
Hope that helps.
We're seeing this pattern happen a lot between two RHEL 6 boxes that are transferring data via a TCP connection. The client issues a TCP Window Full, 0.2s later the client sends TCP Keep-Alives, to which the server responds with what look like correctly shaped responses. The client is unsatisfied by this however and continues sending TCP Keep-Alives until it finally closes the connection with an RST nearly 9s later.
This is despite the RHEL boxes having the default TCP Keep-Alive configuration:
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
...which declares that this should only occur until 2hrs of silence. Am I reading my PCAP wrong (relevant packets available on request)?
Below is Wireshark screenshot of the pattern, with my own packet notes in the middle.
Actually, these "keep-alive" packets are not used for TCP keep-alive! They are used for window size updates detection.
Wireshark treats them as keep-alive packets just because these packets look like keep-alive packet.
A TCP keep-alive packet is simply an ACK with the sequence number set to one less than the current sequence number for the connection.
(We assume that ip 10.120.67.113 refers to host A, 10.120.67.132 refers to host B.) In packet No.249511, A acks seq 24507484. In next packet(No.249512), B send seq 24507483(24507484-1).
Why there are so many "keep-alive" packets, what are they used for?
A sends data to B, and B replies zero-window size to tell A that he temporarily can't receive data anymore. In order to assure that A knows when B can receive data again, A send "keep-alive" packet to B again and again with persistence timer, B replies to A with his window size info (In our case, B's window size has always been zero).
And the normal TCP exponential backoff is used when calculating the persist timer. So we can see that A send its first "keep-alive" packet after 0.2s, send its second packet after 0.4s, the third is sent after 0.8, the fouth is sent after 1.6s...
This phenomenon is related to TCP flow control.
The source and destination IP addresses in the packets originating from client do not match the destination and source IP addresses in the response packets, which indicates that there is some device in between the boxes doing NAT. It is also important to understand where the packets have been captured. Probably a packet capture on the client itself will help understand the issue.
Please note that the client can generate TCP keepalive if it does not receive a data packet for two hours or more. As per RFC 1122, the client retries keepalive if it does not receive a keepalive response from the peer. It eventually disconnects after continuous retry failure.
The NAT devices typically implement connection caches to maintain the state of ongoing connections. If the size of the connection reaches limit, the NAT devices drops old connections in order to service the new connections. This could also lead to such a scenario.
The given packet capture indicates that there is a high probability that packets are not reaching the client, so it will be helpful to capture packets on client machine.
I read the trace slightly differently:
Sender sends more data than receiver can handle and gets zerowindow response
Sender sends window probes (not keepalives it is way to soon for that) and the application gives up after 10 seconds with no progress and closes the connection, the reset indicates there is data pending in the TCP sendbuffer.
If the application uses a large blocksize writing to the socket it may have seen no progress for more than the 10 seconds seen in the tcpdump.
If this is a straight connection (no proxies etc.) the most likely reason is that the receiving up stop receiving (or is slower than the sender & data transmission)
It looks to me like packet number 249522 provoked the application on 10.120.67.113 to abort the connection. All the window probes get a zero window response from .132 (with no payload) and then .132 sends (unsolicited) packet 249522 with 63 bytes (and still showing 0 window). The PSH flag suggests that this 63 bytes is the entire data written by the app on .132. Then .113 in the same millisecond responds with an RST. I can't think of any reason why the TCP stack would send a RST immediately after receiving data (sequence numbers are correct). In my view it is almost certain that the app on .113 decided to give up based on the 63 byte message sent by .132.
In TCP 3-way handshake, 3 segments will be sent (SYN, SYN ACK, ACK). What if the third segment(ACK) is lost? Is the sender going to resend the segment or give up establishing the connection? And how do the two hosts know the segment is lost?
TCP has a sequence number in all packets. Hence it's easy to know if a packet was lost or not. If a host doesn't get an ACK on a packet he just resends it.
In most cases though, even if that ACK was lost, there will be no resending for a very simple reason. Directly after the ACK, the host that opened the TCP protocol is likely to start sending data. That data will, as all TCP packets, have an ACK number, so the recipient would get an ACK that way. Hence, the sender of the SYN-ACK should reasonably not care that it didn't get the ACK, because it gets an "implicit" ACK in the following package.
The re-send of the SYN-ACK is only necessary of there no data is received at all.
Update: I found the place in the RFC that specified exactly this:
If our SYN has been acknowledged (perhaps in this
incoming segment) the precedence level of the incoming segment must
match the local precedence level exactly, if it does not a reset
must be sent.
In other words, if the ACK is dropped but the next packet is not dropped, then everything is fine. Otherwise, the connection must be reset. Which makes perfect sense.
I am not an expert on this particular situation, but I suspect what will happen is the client will think it is connected but the server will not. If the client tries to send data to the server, the server will reject it and send a RST packet to the client so it can reset its "connection".
Say our client is sending the packets at a constant rate. Now, if server goes down temporarily there can be two situations
(We are using the TCP protocol)
1) The packet won't be delivered to the server. Consequently, the other packets in the line have to wait for the server to respond. And the communication can be carried out from there.
2) The packet won't be delivered and will be tried again, but the other packages won't be affected by this packet.
Say, packets A, B and C are to be transferred. While I am sending packet A the server goes down temporarily, then the packets B and C will be sent at the time they were initially scheduled to be or they will be sent once A is received by the server.
TCP is a stream-oriented protocol. This means that if, on a single TCP connection, you send A followed by B then the reciever will never see B until after it has seen A.
If you send A and B over separate TCP connections, then it is possible for B to arrive before A.
When you say "goes down temporarily", what do you mean? I can see two different scenarios.
Scenario 1: The connection between Server and Client is interrupted.
Packet A is sent on its way. Unfortunately, as it is winding its ways through he cables, one cable breaks and A is lost. Meanwhile, depending on the exact state of the TCP windowing algorithm, packets B and C may or may not be sent (as that would depend on the window size, the size of A/B7C and the amount of as-yet unacknowledged bytes sent). I guess that is saying both your "1" and "2" may be right?
If B and/or C have been sent, there will be no ack of A, until it has been resent. If they have been sent, once A has arrived, the server will ack up until the end of the last frame received in sequence (so, C, if taht is the case).
Scenario 2: The sever goes down
If this happens, all TCP state will be lost and connections will have to be re-established after the server has finished rebooting.