(Original title: "Weird TCP connection close behavior")
I am troubleshooting TCP connection process using Wireshark. Client opens connection to server (I tried two different servers), and starts receiving long stream of data. At some point in time client wants to stop and sends server [FIN, ACK] packet, but server does not stop sending data, it continues till its own full stream end, and then sends its own completion packet [FIN, PSH, ACK]. I figured it out keeping reading data from the client's socket after client sent FIN packet. Also, after client sent this FIN packet, its state is FIN_WAIT, thus waiting for FIN response from server...
Why servers do not stop sending data and respond to FIN packet with acknowledgment with FIN set?
I would expect, after client sends FIN packet, server will still send several packets which were on the fly before it received FIN, but not the whole pack of long data stream!
Edit: reading this I think that web server is stuck in stage "CLOSE-WAIT: The server waits for the application process on its end to signal that it is ready to close" (third row), and its data sending process "is done" when it flushed all contents to the socket at its end, and this process can not be terminated. Weird.
Edit1: it appears my question is a little different one. I need to totally terminate connection at client's side, so that server stops sending data, and it (server) would not go crazy about forceful termination from client's side, and aborted its data sending thread at its side being ready for next connection.
Edit2: environment is HTTP servers.
The client has only shutdown the connection for output, not closed it. So the server is fully entitled to keep sending.
If the client had closed the connection, it would issue an RST in response to any further data received, which would stop the server from sending any more, modulo buffering.
Why servers do not stop sending data and respond to FIN packet with acknowledgment with FIN set?
Why should they? The client has said it won't send another request, but that doesn't mean it isn't interested in the response to any requests it has already sent.
Most protocols, such as HTTP, specify that the server should complete the response to the current request and only then close the connection. This is not an abnormal abort, it's just a promise not to send anything else.
Related
I am working on a high-performance TCP server, and I see the server not processing fast enough on and off when I pump high traffic using a TCP client. Upon close inspection, I see spikes in "delta time" on the TCP server. And, I see the server sending an ACK and 0.8 seconds later sending PSH,ACK for the same seqno. I am seeing this pattern multiple times in the pcap. Can experts comment on why the server is sending an ACK followed by a PSH,ACK with a delay in between?
TCP SERVER PCAP
To simplify what ACK and PSH means
ACK will always be present, it simply informs the client what was the last received byte by the server.
PSH tells the client/server to push the bytes to the application layer (the bytes forms a full message).
The usual scenario you are used to, is more or less the following:
The OS has a buffer where it stores received data from the client.
As soon as a packet is received, it is added to the buffer.
The application calls the socket receive method and takes the data out of the buffer
The application writes back data into the socket (response)
the OS sends a packet with flags PSH,ACK
Now imagine those scenarios:
step 4 does not happen (application does not write back any data, or takes too long to write it)
=> OS acknowledge the reception with just an ACK (the packet will not have any data in it), if the application decides later on to send something, it will be sent with PSH,ACK.
the message/data sent by the server is too big to fit in one packet:
the first packets will not have PSH flag, and will only have the ACK flag
the the last packet will have the flags PSH,ACK, to inform the end of the message.
So I have a remote device using a Lantronics XPort module connecting to a VPS. They establish a TCP connection and everything is great. The server ACKs everything.
At some point the remote device stops transmitting data. 30 seconds goes by.
The device then starts sending SYN packets as if trying to establish a new connection. The device is configured to maintain a connection to the server, and it always uses the same source port. (I realize this is bad, but it is hard for me to change)
The server sees a SYN packet from the same (source ip, source port), so the server thinks the connection is ESTABLISHED. The server does not respond to the SYN packet.
Why does the server not respond with ACK as described in Figure 10 in RFC 793? ( https://www.ietf.org/rfc/rfc793.txt )
How can I get the server to kill the connection or respond with an ACK?
It could be the case that during that 30 second silence, the device is waiting for an ACK from the server, and that ACK was dropped somewhere along the line. In this case, I think it should retransmit.
Server is running Ubuntu with kernel 3.12.9-x86_64-linode37
Thank you for any help!
My first suggestion is change the client to use the same connection or to gracefully close the connection before re-opening.
As you DO NOT have control over client and all that can do is on server, you can try this:
Configure keep-alive to be sent after 10 secs of silence and probe only once. If client does not respond, server closes the connection. By doing this, the server should be in listening mode again within 10 seconds of silence without client responding. You can play with the following sysctl's and arrive at optimal values.
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 1
======
Also, regarding missing-ack that you have mentioned in your question, TCP takes care of those things. 30 seconds is too long a time for the first re-transmission from sender. If client/device does not get an ack after 30 seconds, it will/should not try to open a new connection. If you are seeing that, it is an insane-TCP stack at the client. What is that device and which OS/TCP-stack is it using?
it is kernel version has different behavior, ignore any syn packet in kernel 3.12.9-x86_64. but server ack a ack packet, client receive the ack resent rst, and sent new syn in kernel 4.9.0.
incoming-tcp-syns-possibilities
TCP packets ignored on ESTABLISHED connection
Sending a FIN is an indication that the end won't send any data. But can it send TCP keepalive?
If it can then it contradicts the statement of "sending stuff after FIN" even though it is keepalive but not data.
If it doesn't send keepalive, it won't detect if the other end is gone.
For ex:
Client send FIN and got ACK back. Server then sent a reset which was lost.
Only way the client can detect that server has gone away is by sending keepalive which will be responded with reset and then the client closes.
Does this happen?
According to RFC-1122:
Keep-alive packets MUST only be sent when no data or acknowledgement packets have been received for the connection within an interval.
Sending FIN is only an indication that no more data will be sent, not that no more data can be received. If the application after sending FIN is still expecting data, the RFC indicates it can send a keep-alive packet to make sure the other end is still there.
We are facing random RST packet problem in our environments, which causes some unexpected behaviors, following image is snapshot of the tcp data generated by wireshark, which shows the problem:
Client (117.136.2.181) successfully sets up the connection with the server (192.168.40.16)
Client sends some data to the server, as well the KEEP_ALIVE signal.
Server receives the data, process it and sends the result back to client.
Server close the socket.
Server does not receive the ACK signal from client, so it re-transmits the result data as well as the FIN signal, this is automatically done by TCP protocol. However, server still does not receive the ACK signal from client.
Server sends a RST signal to client so connection is closed.
After some analysis, we think some network problem happens after step 3, so all the result data and FIN signal sent from server are not ack'd by client, but we are very confused about the RST signal sent from the server. Based on our understanding, a RST signal is sent if a half-closed socket receives some data, or if there is data in the receive queue when closes a socket. But both these seem not be the root cause of our case.
Can some one help to elaborate why this is happening?
RST usually happens when close is called on the socket without shutdown, or after a shutdown while the other party is still trying to send data (still has not replied with an FIN).
Some programming languages have a socket.close(timeout) for example .NET, that calls shutdown then close after timeout has passed.
So the client have up to timeout to finish sending and closing the connection with FIN, if it fails to do so, the connection will be forcibly closed by RST.
See https://stackoverflow.com/a/23483487/1438522 for a more thorough explanation about difference between close and shutdown.
please see the following pic:
the client initiate a half-close request and get acknowledged,if i dont misunderstand what half-close means,the client should not be able to send any data afterwards, then how can it send back an ACK later.
It's not the client data that is being sent, it the acknowledgement for the server data received after client does shutdown( fd, SHUT_WR ), which means client will not send anymore, but can still receive. The ACK itself is part of the transport protocol, not the application.