I have around 20 clients communicating together with a central server in the same LAN. The clients can make transaction simultaneously with the server. The server forward each transaction to external appliance in the network. Sometimes it works, sometimes my application shows a "time out" message in a client screen (randomly)
I mirrored all traffic and found TCP Retransmission after TCP Reset packets for the first TCP Sequence. I immediately thought about packet loss but all my cables/NIC are fine, and I do not see DUP ACK in the capture.
It seems that RST packets may have different significations.
What causes those TCP Reset?
Where should I focus my investigation: network or application design ?
I would appreciate any help. Thanks in advance.
Judging by the capture, I assume your central server is 137.56.64.31. What's happening is the clients are initiating a connection to the server with a SYN packet and the server responds with a RST. This is typical if the server has no application listening on that particular port e.g. the webserver application isn't running and a client tries to connect to port 80.
The clients are all connecting to different ports on the server, which is unusual for an central server, but not unheard of. The destination ports the clients are connecting to on the server are: 11007, 11012, 11014, 11108, and 11115. Is that normal for the application? If not, the clients should be connecting to whatever port the application server is listening on.
The reason for the retransmits is that instead of giving up on the connection upon receiving a RST from the server, the client tries to initiate the connection again so Wireshark considers it a retransmission.
Related
The goal is to make an http request from the client browser to my server. Simple stuff; however I'm hitting a wall with the networking portion. In order to expose my server to WAN I have used one of my public IPs and NAT to translate to the private ip of my server on inbound traffic and to my public IP on outbound traffic.
The issue is that I can't make a connection. Specifically I can't get the last part of the TCP handshake. Using a test setup with Wireshark on the client and server I can see that the client send the SYN -> the server receives the SYN -> the server sends a SYN/ACK -> the client receives a SYN/ACK -> the the client send an ACK -> the server DOES NOT receive the ACK. It waits for a moment then does a retransmission. Eventually resetting.
I have tried adding various firewall rules even though I don't think it could be the firewall because the first packets make a successful round trip.
I've turned windows firewall off(the server)
I've tried disabling TCP checksum offloading
I've looked for network anti virus settings on the server and on the sonic wall(the router)
I would expect the the tcp connection to complete. I can't for the life of me think of a reason why the ACK would consistently go missing.
That is another thing. The behavior is consistent.
pings also work just fine.
NOTE: The server is actually a VM and the physical server that manages it is in my network.
Any guidance on what to try and where to look would be very much appreciated. Thanks.
UPDATE: I can make connection using port 5000(It's another port I have opened on the firewall). Port 80 still doesn't work though.
In my case this was caused by COX not allowing inbound traffic to port 80. I'm not sure why the first portions of the tcp handshake were getting through. If anyone can explain that part leave a comment.
Ive build a local multiplayer game (multiplayer over wlan network). Now, I want to add an online multiplayer feature..
Currently, the network communications consist mostly of "signals" (tcp/udp packets sent from game-host peer to the game-client peers). I would like to use this mostly signal based communication for my online multiplayer (because of performance and efficiency ), too . But, since the host peer is now replaced by a server there will be a lot problems with sending signals (NAT, firewall,...).
So is there good solution to implement these signals?
regards
there will be a lot problems with sending signals (NAT, firewall,...)_
What problems exactly?
Normally, the clients establish a TCP connection to the server and the server uses this TCP connection to communicate with the clients.
For UDP-based communication the clients use Internet Gateway Device Protocol to forward ports on the router, so that the server can send UDP datagrams to the clients.
Assuming your server is in public internet, not behind any NAT. All the clients must initiate the connection. Otherwise the server can't know clients credential and can't connect. As the server has no NAT it will accept connection from client. And this connection client must keep alive. So when server needs to send some data there should be no problem.
This will work for both UDP and TCP.
when a browser sends a request to a web server, the web server has to send a response.
from what i have understood from reading so far, the server than dispatches the packets of response data with dest-port/dest-ip parts being the client browser's.
1) If the above is right, than doesn't it mean that the browser has to always be listening to a port for incoming traffic from the server?
2) And if the client is listening for incoming connections on a port, isn't that a security concern?
3) If 2 is right, than how are most corporate firewalls for employees be configured? (seeing as they probably need to browse the net) - a quick overview, details unnecessary.
doesn't it mean that the browser has to always be listening to a port for incoming traffic from the server?
No. Layman's explanation: a browser initiates a TCP connection to the web server. This connection is recognized by source ip and port, dest ip and port and protocol by all intermediate level 3 machines (e.g. routers, firewalls).
In a TCP connection, one party listens (the web server) while the other party connects (the browser). Traffic can flow over this connection in both directions, until either party (or intermediate machine) closes the connection.
Corporate firewalls allow outbound connections over port 80 (and 443), so their employees can browse the web over HTTP(S). The data the server returns is sent over the connection initiated by the client.
Of course if an outside attacker knows of a connection, they can send packets with a spoofed IP, so they can send data pretending to be the server. Those packets will be dropped if anything is wrong, like the sequence number, so they won't end up in the user's browser.
Given that you have multiple web browsers running, all which obviously listen on port 80, how would a browser figure if an incoming HTTP response was originated by itself? And whether or not catch the response and show it?
As part of the connection process a TCP/IP connection is assigned a client port. Browsers do not "listen on port 80"; rather a browser/clients initiate a request to port 80 on the server and waits for a reply on the client port from the server's IP.
After the client port is assigned (locally), each client [TCP/IP] connection is uniquely identified by (server IP, server port, client IP, client port) and the connection (and response sent over such) can be "connected back" to the correct browser. This same connection-identifying tuple is how a server doesn't confuse multiple requests coming from the same client/IP1
HTTP sits on top of the TCP/IP layer and doesn't have to concern itself with mixing up connection streams. (HTTP/2 introduces multiplexing, but that is a different beast and only affects connection from the same browser.)
See The Ephemeral Port Range for an overview:
A TCP/IPv4 connection consists of two endpoints, and each endpoint consists of an IP address and a port number. Therefore, when a client user connects to a server computer, an established connection can be thought of as the 4-tuple of (server IP, server port, client IP, client port). Usually three of the four are readily known -- client machine uses its own IP address and when connecting to a remote service, the server machine's IP address and service port number are required [leaving only the client port unknown and to be automatically assigned].
What is not immediately evident is that when a connection is established that the client side of the connection uses a port number. Unless a client program explicitly requests a specific port number, the port number used is an ephemeral port number. Ephemeral ports are temporary ports assigned by a machine's IP stack, and are assigned from a designated range of ports for this purpose. When the connection terminates, the ephemeral port is available for reuse, although most IP stacks won't reuse that port number until the entire pool of ephemeral ports have been used. So, if the client program reconnects, it will be assigned a different ephemeral port number for its side of the new connection.
See TCP/IP Client (Ephemeral) Ports and Client/Server Application Port Use for an additional gentle explanation:
To know where to send the reply, the server must know the port number the client is using. This [client port] is supplied by the client as the Source Port in the request, and then used by the server as the destination port to send the reply. Client processes don't use well-known or registered ports. Instead, each client process is assigned a temporary port number for its use. This is commonly called an ephemeral port number.
1 If there are multiple client computers (ie. different TCP/IP stacks each assigning possibly-duplicate ephemeral ports) using the same external IP then something like Network Address Translation must be used so the server still has a unique tuple per connection:
Network address translation (NAT) is a methodology of modifying network address information in Internet Protocol (IP) datagram packet headers while they are in transit across a traffic routing device for the purpose of remapping one IP address space into another.
thank you all for answers.
the hole listening thing over port 80 was my bad,I must have been dizzy last night :D
anyway,as I have read HTTP is connectionless.
browser initiates an HTTP request and after a request is made, the client disconnects from >the server and waits for a response. The server process the request and re-establish the >connection with the client to send response back.
therefor the browser does not maintain connection waiting for a response.so the answer is not that easy to just send the response back to the open socket.
here's the source
Pay attention browesers aren't listening on specific port to receive HTTP response. Web server listening on specific ports (usually 80 or 443). Browser open connection to web server, and send HTTP request to web server. Browser don't close connection before receive HTTP response. Web server writes HTTP response on opened connection.
Given that you have multiple web browsers running, all which obviously listen on port 80
Not obvious: just wrong. The HTTP server listens on port 80. The browsers connect to port 80.
how would a browser figure if an incoming HTTP response was originated by itself?
Because it comes back on the same connection and socket that was used to send the request.
And whether or not catch the response and show it?
Anything that comes back on the connected socket belongs to the guy who connected the socket.
And in any case all this is the function of TCP, not the browser.
Some things look strange to me:
What is the distinction between 0.0.0.0, 127.0.0.1, and [::]?
How should each part of the foreign address be read (part1:part2)?
What does a state Time_Wait, Close_Wait mean?
etc.
Could someone give a quick overview of how to interpret these results?
0.0.0.0 usually refers to stuff listening on all interfaces.
127.0.0.1 = localhost (only your local interface)
I'm not sure about [::]
TIME_WAIT means both sides have agreed to close and TCP
must now wait a prescribed time before taking the connection
down.
CLOSE_WAIT means the remote system has finished sending
and your system has yet to say it's finished.
I understand the answer has been accepted but here is some additional information:
If it says 0.0.0.0 on the Local Address column, it means that port is listening on all 'network interfaces' (i.e. your computer, your modem(s) and your network card(s)).
If it says 127.0.0.1 on the Local Address column, it means that port is ONLY listening for connections from your PC itself, not from the Internet or network. No danger there.
If it displays your online IP on the Local Address column, it means that port is ONLY listening for connections from the Internet.
If it displays your local network IP on the Local Address column, it means that port is ONLY listening for connections from the local network.
Foreign Address - The IP address and port number of the remote computer to which the socket is connected. The names that corresponds to the IP address and the port are shown unless the -n parameter is specified. If the port is not yet established, the port number is shown as an asterisk (*). (from wikipedia)
What is the distinction between 0.0.0.0, 127.0.0.1, and [::]?
0.0.0.0 indicates something that is listening on all interfaces on the machine.
127.0.0.1 indicates your own machine.
[::] is the IPv6 version of 0.0.0.0
My machine also shows *:\* for UDP which shows that UDP connections don't really have a foreign address - they receive packets from any where. That is the nature of UDP.
How should each part of the foreign address be read (part1:part2)?
part1 is the hostname or IP addresspart2 is the port
127.0.0.1 is your loopback address also known as 'localhost' if set in your HOSTS file. See here for more info: http://en.wikipedia.org/wiki/Localhost
0.0.0.0 means that an app has bound to all ip addresses using a specific port. MS info here: http://support.microsoft.com/default.aspx?scid=kb;en-us;175952
'::' is ipv6 shorthand for ipv4 0.0.0.0.
Send-Q is the amount of data sent by the application, but not yet acknowledged by the other side of the socket.
Recv-Q is the amount of data received from the NIC, but not yet consumed by the application.
Both of these queues reside in kernel memory.
There are guides to help you tweak these kernel buffers, if you are so inclined. Although, you may find the default params do quite well.
This link has helped me a lot to interpret netstat -a
A copy from there -
TCP Connection States
Following is a brief explanation of this handshake. In this context the "client" is the peer requesting a connection and the "server" is the peer accepting a connection. Note that this notation does not reflect Client/Server relationships as an architectural principal.
Connection Establishment
The client sends a SYN message which contains the server's port and the client's Initial Sequence Number (ISN) to the server (active open).
The server sends back its own SYN and ACK (which consists of the client's ISN + 1).
The Client sends an ACK (which consists of the server's ISN + 1).
Connection Tear-down (modified three way handshake).
The client sends a FIN (active close). This is a now a half-closed connection. The client no longer sends data, but is still able to receive data from the server. Upon receiving this FIN, the server enters a passive close state.
The server sends an ACK (which is the clients FIN sequence + 1)
The server sends its own FIN.
The client sends an ACK (which is server's FIN sequence + 1). Upon receiving this ACK, the server closes the connection.
A half-closed connection can be used to terminate sending data while sill receiving data. Socket applications can call shutdown with the second argument set to 1 to enter this state.
State explanations as shown in Netstat:
State Explanation
SYN_SEND Indicates active open.
SYN_RECEIVED Server just received SYN from the client.
ESTABLISHED Client received server's SYN and session is established.
LISTEN Server is ready to accept connection.
NOTE: See documentation for listen() socket call. TCP sockets in listening state are not shown - this is a limitation of NETSTAT. For additional information, please see the following article in the Microsoft Knowledge Base:
134404 NETSTAT.EXE Does Not Show TCP Listen Sockets
FIN_WAIT_1 Indicates active close.
TIMED_WAIT Client enters this state after active close.
CLOSE_WAIT Indicates passive close. Server just received first FIN from a client.
FIN_WAIT_2 Client just received acknowledgment of its first FIN from the server.
LAST_ACK Server is in this state when it sends its own FIN.
CLOSED Server received ACK from client and connection is closed.
For those seeing [::] in their netstat output, I'm betting your machine is running IPv6; that would be equivalent to 0.0.0.0, i.e. listen on any IPv6 address.