Problem
I'm trying to develop a communication system where:
A, B are machines under NAT, A is the server B is the client
S is the STUN server
S is running on a machine reachable on the Internet
The flow is as follows:
A hits S with an opcode saying he's the server
S registers A as server
B hits S with an opcode saying he's the client
S sends to A B's external infos (IP, PORT)
S sends to B A's external infos (IP, PORT)
A starts sending B an opcode saying he's the server every 500ms
and meanwhile listens for packets saying he's got a client
B starts sending A an opcode saying he's the client every 500ms
and meanwhile listen for packets saying he's got the server
Trouble
Here's where the troubles start, the STUN server does its job, since both ends receive correct infos about the other.
But then I do never receive the other end's message, so both ends keep listening without receiving the handshake opcode nor anything else.
NAT's Behaviour
I did examine this NAT's behaviour and seems it does like this
A is at 192.168.X.X, on port 4444
connects to the outside exposing N.N.N.N:4444
so the port number is kept as long as it's free, gets a new (random ?) one if not available.
Tests
The tests I run have seen both ends (A, B) hosted on the same machine, both bound to the machine's internal IP, tried to bind to 127.0.0.1, 0.0.0.0, nothing changed.
If while they're listening for handshakes I echo something with nc to localhost, it is received and displayed (as an unrecognised message) without any problem. The connection routed via the NAT doesn't wotk tough, every packet is discarded.
Also tried with A hosted on the machine, B on an Android phone under mobile data, with a simple app written ad-hoc. Still locks waiting for something, like the nodejs tests.
Update:
Another thing I tried to do is to open an hole with nc
On two different machines under the same NAT I ran:
echo "GREET UNKOWN PEER" | nc -u <NAT IP> 4567 -p 4568
echo "GREET UNKOWN PEER" | nc -u <NAT IP> 4568 -p 4567
Different times for each machine. From my understanding this should punch an hole in the NAT with the first packets discarded and the subsequent forwarded. But nothing happened, no end got the message.
I've also tried:
from local machine
echo "GREET UNKOWN PEER" | nc -u <PUBLIC IP> 4567 -p 4568
from public machine
echo "GREET UNKOWN PEER" | nc -u <NAT IP> 4568 -p 4567
this one works, the local machine under NAT contacts the public one and after the first discarded packet is able to receive and send on the assigned port. I wonder why this doesn't work on two machines under the same NAT (???)
Code
I didn't show any code because I think there is some kind logic flaw in this, however here's the github project for that.
index.js contains the STUN server, the tests folder contains the test cases: test.js starts the stun server, PeerClientTest.js and PeerServerTest.js are mockups of the client and server.
Run node tests/test.js to start the server on a public machine (change IPs in config.js and tests/config.js)
then node tests/PeerServerTest.js to start the server ("A") and node tests/PeerClientTest.js to start the client ("B"). Both will recognize each other via STUN, then listen for the other end's handshake opcode while sending their own handshake opcode. This never happens so they just keep sending/listening forever.
Node is not required, so if there are better solutions in other languages just tell, will be appreciated.
B's NAT is filtering A's packets and is not letting them through. NAT filters unknown packets sent to it. Your server A is sending packet to client B. But client B previously never sent a packet through NAT to A. So to B's NAT A's packets are unknown and being discarded.
You need to punch a hole in B's NAT for the NAT to allow the incoming packets. Send a packet from B to A NAT's IP:Port. After that when you send a packet from A to B, B's NAT won't discard A's packet.
This won't work if A and B's NAT has a combination like Symmetric and Symmetric/PRC NAT. In this case you will have to use a TURN relay server.
Related
Sorry, my networking skills are subpar,
I have a scenario where I have 2 different networks and have 2 servers in those networks, server A is in network A, server B in network B
When someone tries to connect to an external server (over internet), we have a virtual interface configured between A & B (IPIP tunnel)
1)
If I try to netcat/telnet external server (x.x.100.3), I have a route on serverA that routes to my virtual inteface (tunnel1)
2) tunnel1 sends packets over IPIP tunnel to its peer "tunnel2"
3) I need to route any requests to x.x.100.3 over to another internet-facing p1p1 interface
4) serverB sends packets to external serverC over p1p1 interface (I tested running tcpdump -i p1p1 and can see the packets going out
5) reply from C comes back to p1p1
If I do the same netcat cmd from B, I get connected. Yet from A, I get a timeout.
SO my question is, how do I configure this setup so the user who initiates Netcat on server A, gets the reply from server C? Right now the connection stops at server B (p1p1) because the packet header is changing after the Route on B.
I tried playing with iproute2 table rules but cannot get it to work. I basically need to use Server B as a "proxy" network server for any requests going to external server C.
Thanks!
figured out the problem, I didnt have a gateway route in place. This was preventing the back-forth flow of packets.
heres the write up how to setup IPIP tunnel between 2 networks,
https://sites.google.com/site/mrxpalmeiras/linux/create-ipip-tunnel-between-networks
I thought I understood the whole thing about NAT etc but now I came to a problem.
First what I assumed:
Because there are not enough IPv4 addresses available we need another system.
The devices of today at home for connecting to the internet are a combination of:
1) A modem at the physical-level to change the type of signals on the wire.
2) A switch at link-level so you can connect multiple computers to the device
3) A router to connect all the computers to the internet and go beyond your home-subnet etc.
4) A NAT to allow all the internal computers to connect to the outside
5) A portforwarder to let connections from the outside to the internal network
What I call a NAT:
When making a request to the outside: the NAT-part of the device changes the source-port and the source-ip of the request coming from an internal computer. The new source-ip will be your public-ip. The NAT-part will hold a record in a table with this mapping: "original-ip, original-port, new-port".
When a reponse comes back, the NAT will check the destination-port and compare this with the new-ports is in his table. If it finds a match the NAT will replace the destination-ip with original-ip and new-port with original-port. As a consequence the response will be forwarded to the internal computer that made the request.
So, the NAT-part is for when a connection is initialized from the inside. When this request traverses the NAT, 2 things are changed: source-ip and source-port.
Then the portforwarder:
This part of the device will accept connections initialized in the outside-world to your network. It will look at the destination-port of the incoming request and by making a rule for that port-number it may change the destination-port and the destination-ip of the request to an internal ip. With these rules a request from the outside can connect to a computer on your internal network and thus the portforwarder changes 2 things: the destination-ip and the destination-port.
A: Before I ask my question, how is this explanation?
Now my problem is with the response after a request came from the outside through the portforwarder. Assume the right rules are made and a request came through portforwarding on an internal computer. So in the portforwarder the destination-ip was changed to the internal-ip of the computer and the destination-port was changed to the port where the service is running on. If this internal-computer is a webserver it will generate a response. So the destination-ip will be the request's source-ip and the destination-port will be the request's source-port. The source-ip will be the internal-ip of the computer and the source-port will be the port of the service.
Now that response has to go to the outside. So I assume it goes through the NAT to the outside?
So after passing the NAT, the source-ip will be the public-ip and the source-port will be random. Now I tested this with wireshark. I contacted a webserver behind a NAT and I saw the reponse was coming from port 80 ?! How is this possible? This indicates that the response of the forwarded request did not pass the NAT?
I rethought the concept and my new hypothesis is that when a connection is initialized from the outside, it will pass the portforwarder and reach the right computer. This will create a response and when this response reaches our "all-in-one"device, this device can recognize it forwarded the request of the response and will not change the source-port.
B: Is this indeed the case or is it done in another way?
Wikipedia says about portforwarding: "The source address and port are, in this case, left unchanged. When used on machines that are not the default gateway of the network, the source address must be changed to be the address of the translating machine, or packets will bypass the translator and the connection will fail." (http://en.wikipedia.org/wiki/Port_forwarding)
This confirms that the response of a forwarded request MUST go through the portforwarder again and not through the NAT so the source-port wont be changed. The portforwarder will change the source-ip to the public-ip.
Can someone verify this or give me another explanation than mine?
Now I tested this with wireshark. I contacted a webserver behind a NAT
and I saw the reponse was coming from port 80 ?! How is this possible?
This indicates that the response of the forwarded request did not pass
the NAT?
The webserver inside the NAT does not have to be running on port 80. It certainly is set up at the NAT to port forward and respond as if it were at port 80, but that doesn't mean much about the port the web server is actually running on.
Here is some ASCII "art" that may help.
**Internal Network** **NAT Router** **External Computer**
Web Server running at IP 9.9.9.9 port 80 IP 20.20.20.20
IP 192.168.1.7 port 4567
Request web page at 9.9.9.9:80
Forwards port 80 traffic
to 192.168.1.7:4567
Replies with the web page
Puts 9.9.9.9:80 in the
source field and sends
the page on
Gets the page from "9.9.9.9:80"
even though it actually came
from 192.168.1.7:4567
http header : Connection: Keep-Alive
After reading alot about this , I still can't understand how its working.
wiki :
A keepalive signal can
also be used to indicate to Internet infrastructure that the
connection should be preserved. Without a keepalive signal
intermediate NAT-enabled routers can drop the connection after
timeout.
I dont udnerstand :
a Server can have 1,000,000 cuncorrent connections .
John sends a request to the server.
Paul's compter is on the same lan near paul. paul also sends a request to the same server.
John's and paul organization is behind router.
How the hell the server knows how to keep connection alive for both paul and john ?
Also , when john sends request the second time , it "doesnt open a new conneciton" , so how does keep-alive is applied here ?
First of all, TCP/IP connection is not some thin wire that is temporarily connecting two computers. At the end of the day both TCP/IP and UDP are just a series of packets. It's the operating system that pretends you have a connection by putting the IP packets back together in correct order.
Now back to your question. Note that the problem is not really HTTP-specific, all of this works on TCP/IP layer. Say Paul has 192.168.0.100 and John has 192.168.0.101 internal IP addresses while NAT has public 1.2.3.4 address. When Paul connects to some server, his OS uses 192.168.0.100:54321 address (port is chosen randomly by OS). This request hits NAT which remembers that address and forwards request to external server. The external server sees 1.2.3.4:4321 (notice the different port) as the user is behind the NAT so internal IP is not visible.
When the external server (let it be web server) sends a reply, it sends it to 1.2.3.4:4321. NAT, on the other hand, remembers that 4321 port should be forwarded to 192.168.0.100:54321` - and so it is.
Now imagine John sends request to the same server. This TCP/IP connection is routed through NAT which remember that request from 192.168.0.101:32123 was made. This request is then forwarded using public 1.2.3.4:4322 (notice different port). When response arrives, NAT checks the port if it is 4322, it routes to 192.168.0.101:32123 (John). Otherwise (on port 4321) Paul will get his reply.
Note: do not confuse client ephemeral port with server port (80 in HTTP by default).
This question already has answers here:
Does the port change when a server accepts a TCP connection?
(3 answers)
Closed 4 years ago.
I understand the basics of how ports work. However, what I don't get is how multiple clients can simultaneously connect to say port 80. I know each client has a unique (for their machine) port. Does the server reply back from an available port to the client, and simply state the reply came from 80? How does this work?
First off, a "port" is just a number. All a "connection to a port" really represents is a packet which has that number specified in its "destination port" header field.
Now, there are two answers to your question, one for stateful protocols and one for stateless protocols.
For a stateless protocol (ie UDP), there is no problem because "connections" don't exist - multiple people can send packets to the same port, and their packets will arrive in whatever sequence. Nobody is ever in the "connected" state.
For a stateful protocol (like TCP), a connection is identified by a 4-tuple consisting of source and destination ports and source and destination IP addresses. So, if two different machines connect to the same port on a third machine, there are two distinct connections because the source IPs differ. If the same machine (or two behind NAT or otherwise sharing the same IP address) connects twice to a single remote end, the connections are differentiated by source port (which is generally a random high-numbered port).
Simply, if I connect to the same web server twice from my client, the two connections will have different source ports from my perspective and destination ports from the web server's. So there is no ambiguity, even though both connections have the same source and destination IP addresses.
Ports are a way to multiplex IP addresses so that different applications can listen on the same IP address/protocol pair. Unless an application defines its own higher-level protocol, there is no way to multiplex a port. If two connections using the same protocol simultaneously have identical source and destination IPs and identical source and destination ports, they must be the same connection.
Important:
I'm sorry to say that the response from "Borealid" is imprecise and somewhat incorrect - firstly there is no relation to statefulness or statelessness to answer this question, and most importantly the definition of the tuple for a socket is incorrect.
First remember below two rules:
Primary key of a socket: A socket is identified by {SRC-IP, SRC-PORT, DEST-IP, DEST-PORT, PROTOCOL} not by {SRC-IP, SRC-PORT, DEST-IP, DEST-PORT} - Protocol is an important part of a socket's definition.
OS Process & Socket mapping: A process can be associated with (can open/can listen to) multiple sockets which might be obvious to many readers.
Example 1: Two clients connecting to same server port means: socket1 {SRC-A, 100, DEST-X,80, TCP} and socket2{SRC-B, 100, DEST-X,80, TCP}. This means host A connects to server X's port 80 and another host B also connects to the same server X to the same port 80. Now, how the server handles these two sockets depends on if the server is single-threaded or multiple-threaded (I'll explain this later). What is important is that one server can listen to multiple sockets simultaneously.
To answer the original question of the post:
Irrespective of stateful or stateless protocols, two clients can connect to the same server port because for each client we can assign a different socket (as the client IP will definitely differ). The same client can also have two sockets connecting to the same server port - since such sockets differ by SRC-PORT. With all fairness, "Borealid" essentially mentioned the same correct answer but the reference to state-less/full was kind of unnecessary/confusing.
To answer the second part of the question on how a server knows which socket to answer. First understand that for a single server process that is listening to the same port, there could be more than one socket (maybe from the same client or from different clients). Now as long as a server knows which request is associated with which socket, it can always respond to the appropriate client using the same socket. Thus a server never needs to open another port in its own node than the original one on which the client initially tried to connect. If any server allocates different server ports after a socket is bound, then in my opinion the server is wasting its resource and it must be needing the client to connect again to the new port assigned.
A bit more for completeness:
Example 2: It's a very interesting question: "can two different processes on a server listen to the same port". If you do not consider protocol as one of the parameters defining sockets then the answer is no. This is so because we can say that in such a case, a single client trying to connect to a server port will not have any mechanism to mention which of the two listening processes the client intends to connect to. This is the same theme asserted by rule (2). However, this is the WRONG answer because 'protocol' is also a part of the socket definition. Thus two processes in the same node can listen to the same port only if they are using different protocols. For example, two unrelated clients (say one is using TCP and another is using UDP) can connect and communicate to the same server node and to the same port but they must be served by two different server processes.
Server Types - single & multiple:
When a server processes listening to a port that means multiple sockets can simultaneously connect and communicate with the same server process. If a server uses only a single child process to serve all the sockets then the server is called single-process/threaded and if the server uses many sub-processes to serve each socket by one sub-process then the server is called a multi-process/threaded server. Note that irrespective of the server's type a server can/should always use the same initial socket to respond back (no need to allocate another server port).
Suggested Books and the rest of the two volumes if you can.
A Note on Parent/Child Process (in response to query/comment of 'Ioan Alexandru Cucu')
Wherever I mentioned any concept in relation to two processes say A and B, consider that they are not related by the parent-child relationship. OS's (especially UNIX) by design allows a child process to inherit all File-descriptors (FD) from parents. Thus all the sockets (in UNIX like OS are also part of FD) that process A listening to can be listened to by many more processes A1, A2, .. as long as they are related by parent-child relation to A. But an independent process B (i.e. having no parent-child relation to A) cannot listen to the same socket. In addition, also note that this rule of disallowing two independent processes to listen to the same socket lies on an OS (or its network libraries), and by far it's obeyed by most OS's. However, one can create own OS which can very well violate this restriction.
TCP / HTTP Listening On Ports: How Can Many Users Share the Same Port
So, what happens when a server listen for incoming connections on a TCP port? For example, let's say you have a web-server on port 80. Let's assume that your computer has the public IP address of 24.14.181.229 and the person that tries to connect to you has IP address 10.1.2.3. This person can connect to you by opening a TCP socket to 24.14.181.229:80. Simple enough.
Intuitively (and wrongly), most people assume that it looks something like this:
Local Computer | Remote Computer
--------------------------------
<local_ip>:80 | <foreign_ip>:80
^^ not actually what happens, but this is the conceptual model a lot of people have in mind.
This is intuitive, because from the standpoint of the client, he has an IP address, and connects to a server at IP:PORT. Since the client connects to port 80, then his port must be 80 too? This is a sensible thing to think, but actually not what happens. If that were to be correct, we could only serve one user per foreign IP address. Once a remote computer connects, then he would hog the port 80 to port 80 connection, and no one else could connect.
Three things must be understood:
1.) On a server, a process is listening on a port. Once it gets a connection, it hands it off to another thread. The communication never hogs the listening port.
2.) Connections are uniquely identified by the OS by the following 5-tuple: (local-IP, local-port, remote-IP, remote-port, protocol). If any element in the tuple is different, then this is a completely independent connection.
3.) When a client connects to a server, it picks a random, unused high-order source port. This way, a single client can have up to ~64k connections to the server for the same destination port.
So, this is really what gets created when a client connects to a server:
Local Computer | Remote Computer | Role
-----------------------------------------------------------
0.0.0.0:80 | <none> | LISTENING
127.0.0.1:80 | 10.1.2.3:<random_port> | ESTABLISHED
Looking at What Actually Happens
First, let's use netstat to see what is happening on this computer. We will use port 500 instead of 80 (because a whole bunch of stuff is happening on port 80 as it is a common port, but functionally it does not make a difference).
netstat -atnp | grep -i ":500 "
As expected, the output is blank. Now let's start a web server:
sudo python3 -m http.server 500
Now, here is the output of running netstat again:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
So now there is one process that is actively listening (State: LISTEN) on port 500. The local address is 0.0.0.0, which is code for "listening for all". An easy mistake to make is to listen on address 127.0.0.1, which will only accept connections from the current computer. So this is not a connection, this just means that a process requested to bind() to port IP, and that process is responsible for handling all connections to that port. This hints to the limitation that there can only be one process per computer listening on a port (there are ways to get around that using multiplexing, but this is a much more complicated topic). If a web-server is listening on port 80, it cannot share that port with other web-servers.
So now, let's connect a user to our machine:
quicknet -m tcp -t localhost:500 -p Test payload.
This is a simple script (https://github.com/grokit/dcore/tree/master/apps/quicknet) that opens a TCP socket, sends the payload ("Test payload." in this case), waits a few seconds and disconnects. Doing netstat again while this is happening displays the following:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.10:500 192.168.1.13:54240 ESTABLISHED -
If you connect with another client and do netstat again, you will see the following:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.10:500 192.168.1.13:26813 ESTABLISHED -
... that is, the client used another random port for the connection. So there is never confusion between the IP addresses.
Normally, for every connecting client the server forks a child process that communicates with the client (TCP). The parent server hands off to the child process an established socket that communicates back to the client.
When you send the data to a socket from your child server, the TCP stack in the OS creates a packet going back to the client and sets the "from port" to 80.
Multiple clients can connect to the same port (say 80) on the server because on the server side, after creating a socket and binding (setting local IP and port) listen is called on the socket which tells the OS to accept incoming connections.
When a client tries to connect to server on port 80, the accept call is invoked on the server socket. This creates a new socket for the client trying to connect and similarly new sockets will be created for subsequent clients using same port 80.
Words in italics are system calls.
Ref
http://www.scs.stanford.edu/07wi-cs244b/refs/net2.pdf
Some things look strange to me:
What is the distinction between 0.0.0.0, 127.0.0.1, and [::]?
How should each part of the foreign address be read (part1:part2)?
What does a state Time_Wait, Close_Wait mean?
etc.
Could someone give a quick overview of how to interpret these results?
0.0.0.0 usually refers to stuff listening on all interfaces.
127.0.0.1 = localhost (only your local interface)
I'm not sure about [::]
TIME_WAIT means both sides have agreed to close and TCP
must now wait a prescribed time before taking the connection
down.
CLOSE_WAIT means the remote system has finished sending
and your system has yet to say it's finished.
I understand the answer has been accepted but here is some additional information:
If it says 0.0.0.0 on the Local Address column, it means that port is listening on all 'network interfaces' (i.e. your computer, your modem(s) and your network card(s)).
If it says 127.0.0.1 on the Local Address column, it means that port is ONLY listening for connections from your PC itself, not from the Internet or network. No danger there.
If it displays your online IP on the Local Address column, it means that port is ONLY listening for connections from the Internet.
If it displays your local network IP on the Local Address column, it means that port is ONLY listening for connections from the local network.
Foreign Address - The IP address and port number of the remote computer to which the socket is connected. The names that corresponds to the IP address and the port are shown unless the -n parameter is specified. If the port is not yet established, the port number is shown as an asterisk (*). (from wikipedia)
What is the distinction between 0.0.0.0, 127.0.0.1, and [::]?
0.0.0.0 indicates something that is listening on all interfaces on the machine.
127.0.0.1 indicates your own machine.
[::] is the IPv6 version of 0.0.0.0
My machine also shows *:\* for UDP which shows that UDP connections don't really have a foreign address - they receive packets from any where. That is the nature of UDP.
How should each part of the foreign address be read (part1:part2)?
part1 is the hostname or IP addresspart2 is the port
127.0.0.1 is your loopback address also known as 'localhost' if set in your HOSTS file. See here for more info: http://en.wikipedia.org/wiki/Localhost
0.0.0.0 means that an app has bound to all ip addresses using a specific port. MS info here: http://support.microsoft.com/default.aspx?scid=kb;en-us;175952
'::' is ipv6 shorthand for ipv4 0.0.0.0.
Send-Q is the amount of data sent by the application, but not yet acknowledged by the other side of the socket.
Recv-Q is the amount of data received from the NIC, but not yet consumed by the application.
Both of these queues reside in kernel memory.
There are guides to help you tweak these kernel buffers, if you are so inclined. Although, you may find the default params do quite well.
This link has helped me a lot to interpret netstat -a
A copy from there -
TCP Connection States
Following is a brief explanation of this handshake. In this context the "client" is the peer requesting a connection and the "server" is the peer accepting a connection. Note that this notation does not reflect Client/Server relationships as an architectural principal.
Connection Establishment
The client sends a SYN message which contains the server's port and the client's Initial Sequence Number (ISN) to the server (active open).
The server sends back its own SYN and ACK (which consists of the client's ISN + 1).
The Client sends an ACK (which consists of the server's ISN + 1).
Connection Tear-down (modified three way handshake).
The client sends a FIN (active close). This is a now a half-closed connection. The client no longer sends data, but is still able to receive data from the server. Upon receiving this FIN, the server enters a passive close state.
The server sends an ACK (which is the clients FIN sequence + 1)
The server sends its own FIN.
The client sends an ACK (which is server's FIN sequence + 1). Upon receiving this ACK, the server closes the connection.
A half-closed connection can be used to terminate sending data while sill receiving data. Socket applications can call shutdown with the second argument set to 1 to enter this state.
State explanations as shown in Netstat:
State Explanation
SYN_SEND Indicates active open.
SYN_RECEIVED Server just received SYN from the client.
ESTABLISHED Client received server's SYN and session is established.
LISTEN Server is ready to accept connection.
NOTE: See documentation for listen() socket call. TCP sockets in listening state are not shown - this is a limitation of NETSTAT. For additional information, please see the following article in the Microsoft Knowledge Base:
134404 NETSTAT.EXE Does Not Show TCP Listen Sockets
FIN_WAIT_1 Indicates active close.
TIMED_WAIT Client enters this state after active close.
CLOSE_WAIT Indicates passive close. Server just received first FIN from a client.
FIN_WAIT_2 Client just received acknowledgment of its first FIN from the server.
LAST_ACK Server is in this state when it sends its own FIN.
CLOSED Server received ACK from client and connection is closed.
For those seeing [::] in their netstat output, I'm betting your machine is running IPv6; that would be equivalent to 0.0.0.0, i.e. listen on any IPv6 address.