I have an application that is sending web requests and after some variable period of time we start getting connection timeout errors. Analysis of the TCP/IP packets reveals the following occurring for each connection attempted:
No. Time Source Destination Protocol Info
98635 2011-04-21 10:13:47.192237 172.22.3.250 172.22.3.100 TCP [TCP Port numbers reused] httpx > http-alt [SYN] Seq=0 Win=65535 Len=0 MSS=1460
98639 2011-04-21 10:13:47.192397 172.22.3.100 172.22.3.250 TCP http-alt > httpx [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460
98641 2011-04-21 10:13:47.192425 172.22.3.250 172.22.3.100 TCP httpx > http-alt [ACK] Seq=1 Ack=1 Win=65535 Len=0
98646 2011-04-21 10:13:47.192502 172.22.3.250 172.22.3.100 TCP httpx > http-alt [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
Once this happens the client machine must be rebooted to fix the issue. Restarting the client or server application, or rebooting the server have no effect. Comparison of the packet details between working and non-working messages reveal no differences for flags, etc.
Thank you,
Neil
TCP Port numbers reused is a clue. It might be your client is re-using the port numbers of the old connection which is supposed to be closed.
Does your client bind to a static source port? If yes it might be that the clients operating system is rejecting the packets from the server because it thinks they belong to an old connection. The OS is rejecting the packets, that's why only a client reboot solves the problem. (Alternatively try waiting a couple of minutes).
When opening the connection on the client, don't assign a source port.
Related
I have an application deployed on a Kubernetes cluster. I am using Istio/Envoy in this deployment to control inbound/outbound traffics. I have collected some TCP packets using TCPdump for investigating some issue.
To my understanding, a TCP handshake should only involves a pair of 5-tuple (src-IP, src-Port, dst-IP, dst-Port, protocol).
For example
IP: 198.168.1.100 Port: 52312 ----SYN----> IP: 198.168.1.101 Port: 80
IP: 198.168.1.100 Port: 52312 <--SYN ACK-- IP: 198.168.1.101 Port: 80
IP: 198.168.1.100 Port: 52312 ----ACK----> IP: 198.168.1.101 Port: 80
But in the packets I collected, what I don't understand is this:
10.X.X.X 127.0.0.1 TCP 76 33500 → 15001 [SYN] Seq=3333992218
X.X.X.X 10.X.X.X TCP 76 80 → 33500 [SYN, ACK] Seq=2228273021 Ack=3333992219
10.X.X.X 127.0.0.1 TCP 68 33500 → 15001 [ACK] Seq=3333992219 Ack=2228273022
Notice the SYN ACK was returned from a port 80. First, I thought there could be missing packets and there was actually two handshakes, but looking at the sequence number and acknowledgment number, it seems to be single handshake.
If this is a single handshake, how would you explain this? Is there a technique that does the TCP handshake differently?
If this is a single handshake, how would you explain this? Is there a technique that does the TCP handshake differently?
According to this blog:
This is a single handshake, just there are 2 separate connections, first goes to envoy sidecar, then envoy sidecar as a middleman send it to your pod.
So this is the magic: the connection is not established between client and server directly, but split into 2 separate connections:
connection between client and sidecar
connection between sidecar and server
Those two connections are independently handshaked, thus even if the latter failed, the former could still be succesful.
Actual view of the two sides: a middleman sits between client and server
If you're looking for more informations about the 15001 port itself you can visit istio documentation.
It is explained in more details here.
Context -
I am making a request from source(IP - x) to target (IP - y)
curl y
intercepting it in wireshark -
x -> y [SYN] , sync request to target
y -> x [SYN, ACK] , target acknowledges client's sync request and also makes a sync request to client, because it also needs to communicate to client
x -> y [ACK] , client acknowledges target's sync request
x -> y HTTP GET [PSH, ACK], client makes HTTP GET request (why send ACK?)
y -> x [ACK] , target acknowledges client's request
y -> x [PSH, ACK] , target pushes the content (and acknowldges previous ACK? which one?)
y -> x HTTP 302 FOUND [FIN, ACK] , target send FIN to finalize and ACK
x -> y [ACK] , client acknowledges target's FIN
x -> y [ACK] , client acknowledges previous ACK? which one?
x -> y [FIN, ACK] , client sends FIN to finalize communication with target and send additional ACK
y -> x [ACK] , server acknowledges client's FIN
I have posted my question with each tcp request/response.
Can someone please explain what those extra ACKs are for?
As Steffen pointed out in comments, Even after handshake, It is recommended that ACK is sent with each packet from client/server.
I want to scan udp ports with SYN flag. But how would udp server react to the SYN packet, when the udp port is open and not open?
There are not flags field (including SYN flag) in the UDP header. So, technically speaking, it is not possible to send SYN packet using UDP protocol.
If you try to send TCP-formatted packet with protocol value 17 (UDP), the checksum will not match, and the packet will be discarded.
My testing environment
client
IP 192.168.0.2/24
gateway 192.168.0.1
server
IP 192.168.0.1/24
http service run on port 80
When I try getting web page hosted on server, everything goes fine.
Then I write a kernel module using netfilter on server, which will change dest IP to 192.168.0.1 if the origin dest IP is 192.168.1.1, and will change source IP to 192.168.1.1 if origin source IP is 192.168.0.1. I think you can understand that I'm just pretending the server to be 192.168.1.1 for the client.(IP header Checksum and TCP Checksum are changed properly)
I use the web browser(chrome, firefox...) on client to visit 192.168.1.1 and capture the packets on the client, results are like:
192.168.0.2:someport_1 -> 192.168.1.1:80 [SYN]
192.168.1.1:80 -> 192.168.0.2:someport_1 [SYN, ACK]
192.168.0.2:someport_2 -> 192.168.1.1:80 [SYN]
192.168.1.1:80 -> 192.168.0.2:someport_2 [SYN, ACK]
192.168.0.2:someport_3 -> 192.168.1.1:80 [SYN]
192.168.1.1:80 -> 192.168.0.2:someport_3 [SYN, ACK]
I don't know why the client will never send the last ACK of TCP handshaking, any ideas?
Edit1:
Now I think that the browser didn't get the [SYN, ACK] packet from the server although wireshark can see it, so maybe it's because that the OS(Windows7) dropped the [SYN, ACK] packet from the server. Now the question becomes that why would windows drop a correct [SYN, ACK] packet?
You said the IP checksum is OK, but what about the TCP checksum, which is computed from a pseudo-header which includes source and destination IP ?
I've made three mistakes.
The first one is that skb can be nonlinear, which will cause the checksum got from csum_partial() be incorrect.
The second one is that I use csum_tcpudp_magic() to get the checksum, but forgot to change skb->ip_summed, so the NIC will use my correct checksum as the partial checksum of tcp pseudo-header to recalculate the checksum, leading checksum incorrect in the packet.
The third mistake is that my wireshark seems to be set to ignore the tcp checksum, and it always shows the packets with wrong checksum as good ones, while tcpdump will tell me incorrect checksums.
Please help me decode the following. I have a problem in the network, unable to figure out where it is. Follow is the decode summary we got from the capture by running the OPNET software.
I think Connection 2 is successful, where the client ends with FIN, and connection 1 ends prematurely issuing RST, what could be the reasons..?
Connection 1
Frame Source Destination Decode Summary
480 Server Client D=3590 S=443 FIN ACK=3832271994 SEQ=1338775336 LEN=0 WIN=11088
481 Client Server D=443 S=3590 ACK=1338775337 SEQ=3832271994 LEN=0 WIN=64512
502 Client Server D=443 S=3590 RST ACK=1338775337 SEQ=3832271994 LEN=0 WIN=0
Connection 2
855 Server Client D=3600 S=443 FIN ACK=3778329063 SEQ=3650187663 LEN=0 WIN=26328
856 Client Server D=443 S=3600 ACK=3650187664 SEQ=3778329063 LEN=0 WIN=63297
857 Client Server D=443 S=3600 FIN ACK=3650187664 SEQ=3778329063 LEN=0 WIN=63297
859 Server Client D=3600 S=443 ACK=3778329064 SEQ=3650187664 LEN=0 WIN=26328
In summary the connection endings are,
1 - Server: FIN, Client: RST
2 - Server: FIN, Client: FIN
From the Wikipedia page on TCP:
Some host TCP stacks may implement a half-duplex close sequence, as Linux or HP-UX do. If such a host actively closes a connection but still has not read all the incoming data the stack already received from the link, this host sends a RST instead of a FIN (Section 4.2.2.13 in RFC 1122).
Could it be something like this you are seeing?