I'm looking to build a bittorrent client in Ruby (although language is not important over here).
I read a BEP specification which says querying at /announce (without DHT support) to tracker would give a list of peers currently connected for a given info-hash.
To examine this. I created a torrent file and I found that tracker return the IP of my machine along with the port (which i confirmed is the running port of Bittorrent client on my machine i.e Utorrent)
But here is the problem the Peers info returned the IP of my ISP i.e 111.125.209.41 (the public facing IP since I'm behind the NAT).
Now I cant connect (via TCP) to the Process running on my machine with the public facing IP of my ISP.
Can anyone suggest how does bitorrent work on this and what should I do to solve this.
It could that over UDP Bittorrent would be using UDP hold punching not sure what happen on TCP.
There are two separate concerns.
How do peers on the internet (aka 99.99999% of the world) connect to your NATed node
How do peers inside your network connect to a node on the same network.
The first is achieved by various nat traversal methods, including negotiating with the nat device or manually configuring it.
The second issue either requires a router capable of hairpin routing in combination with a forwarded port or local peers discovering the internal address through other means, such as LSD.
I am trying to write a port-forwarder in Twisted, that will forward to port 8000 if an IP address is already in the cache & to another port - say 4000 if not. I already have the cache written, but am having issues with where to add logic to the portforwarding with Twisted.
Take this simple example:
class LoggingProxyServer(portforward.ProxyServer):
def dataReceived(self, data):
portforward.ProxyServer.dataReceived(self, data)
class LoggingProxyFactory(portforward.ProxyFactory):
protocol = LoggingProxyServer
What twisted method do I override to add the cache checking?
ProxyServer.connectionMade is responsible for setting up the TCP connection that is outgoing from the proxy process. It uses the host and port attributes of its factory to decide what it's going to use as the destination of that connection attempt.
If you want to vary the behavior of the proxy, that's the code you'll need to override.
You can easily find the IP address of the client which has connected to ProxyServer. The ProxyServer instance has a transport attribute that refers to an ITransport provider (probably an ITCPTransport provider if your proxy is listening for incoming TCP connections).
Transports have methods to tell you the addresses of their two endpoints. getHost tells you the local address and getPeer tells you the remote address.
So, for example, you could write a conditional that had one behavior for all TCP clients with IP addresses starting with a 1 and something else for all TCP clients with other IP addresses:
if self.transport.getPeer().host.startswith("1"):
...
else:
...
I am trying to measure the latency between one of my machines, and an EC2 instance. EC2 instances cannot be pinged. So I tried using application level timestaps (using gettimeofday()). I send a tcp packet with a timestamp in the payload.
Upon receiving this packet, I calculate the timestamp on my machine, and obtain the difference. It always comes out to be negative. My guess was that the clocks in the two machines could be skewed. So I used ntp to synchronize both the machines, but the problem still persists.
Can someone please help.
EC2 instances can be pinged, if configured to allow it. I set one up for this today while trying to track down packet drops in us-west-2. In the security group protecting the instance, you add a rule to permit "ICMP Echo Request" from the source address of the machine where you're originating the ping.
See the AWS FAQ for this quote.
Why can't I ping my instance? Ping uses ICMP ECHO, which by default is
blocked by your firewall. You'll need to grant ICMP access to your
instances by updating the firewall restrictions that are tied to your
security group.
ec2-authorize default -P icmp -t -1:-1 -s 0.0.0.0/0
Check out the latest developer guide for details.
Section: Instance Addressing and Network Security -> Network Security
-> Examples
I am trying to achieve double NAT working with UPNP. My PC connection is as follows:
Internet -> Router 1 -> Router 2-> PC
I have successfully done port forwarding on the Router 2, but not able to do on router 1. Due to which packets from the server(which is on Internet) are not able to reach PC.
Gone through few document how to tackle but mostly found manual way like bridge the network or put the lan wire.
I am rather looking for the some recursive solution using UPNP protocol to do the port forward on both routers. I guess torrent, skype does the same.
How to get the network interface related to router 1 and then port forward on that Network ?
Any help would be highly appreciated.
Thanks,
Pawan
There is an IGD-PCP IWF specification that tries to solve similar problem, although it assumes PCP support on your "Router 1", not UPnP. So let's try to approach this issue from a pure theoretical point of view with two plain UPnP routers/NAT devices.
There are several distinct steps of UPnP communication as per UPnP Device Architecture version 2.0:
addressing
discovery
description
control
eventing
presentation
Addressing is of little interest for us, let's assume proper DHCP everywhere and be done with it. Eventing and presentation are also almost useless in our case. So the main things to be concerned about are discovery, description and control.
Discovery works via SSDP message exchange. SSDP uses UDP for its transport with port number 1900 (by default) and well-known multicast address.
Description starts with URL provided by the device at discovery phase, the control point (that is PC in our case) needs to issue an HTTP GET request on this URL and that means it uses TCP as a transport protocol with devices IP address (unicast).
Control starts with URL provided by the device in its description, and it uses SOAP on top of HTTP on top of TCP which in turn also means unicast IP for us.
So, what all of that means for double NAT is that in description and control steps of UPnP interaction we have zero problems communicating from PC to Router 1 as all of that is just standard TCP with unicast IP addresses. But to get to the description step we need to have a URL of Router 1, so let's take a closer look at how this URL is acquired the normal way.
There are two main mechanisms for discovery — advertising (when the device periodically multicasts some information about it) and search (when the control point sends multicast search message and the device answers to that with unicast response). Obviously, by default our PC behind the Router 2 can't get multicast advertisements from Router 1 and the Router 1 can't get multicast search messages from PC, so we have a problem here and the question now is whether there is a possibility for communication without multicast.
Luckily, the same architecture document says:
In addition, a control point is allowed to unicast a discovery message to a specific IP address on port 1900 or on the port specified by the optional SEARCHPORT.UPNP.ORG header field (which supersedes port 1900 for this use), searching for a UPnP device or service at that specific IP address.
...
All devices shall listen to incoming unicast search messages on port 1900 or, if provided, the port number specified in the SEARCHPORT.UPNP.ORG header field and shall respond if any of their root devices, embedded devices or services matches the search criteria in the discovery message.
And this means that if you know the Router 1 IP address (from Router 2 side, of course), you can (and most importantly, allowed by specification to) communicate with it with unicast UDP messages and that is also NAT-friendly, so not an issue to be done from PC behind the Router 2.
The only thing left is getting Router 1 IP address. Unfortunately, there is no easy standard way to do that, but you have at least two options: tracerouting (in whatever fashion you want) and brute force IP scanning (most probably, the potential set of IPs for Router 1 is limited).
Now you can communicate with the Router 1, but there is still one minor thing you should always remember while communicating with it — in any internal UPnP messages you should use Router 2 IP address (as seen from Router 1 side) and its ports. Like in NewInternalClient parameter of AddPortMapping action on Router 1 you should use Router 2 IP. This, BTW, raises a question of Router 2 IP, but you can get that via Router 2 UPnP ExternalIPAddress variable of WANIPConnection service (this service is required for IGDs to implement).
So, to summarize:
this technically can be done, although I doubt that any standard library would do that for you
two things you need are:
Router 2 "external" IP and you get that via ExternalIPAddress variable of WANIPConnection UPnP service
Router 1 "internal" IP (from Router 2 side), that requires tracerouting or scanning
given Router 1 IP you just need to use unicast messaging at discovery step instead of multicast
everything else should just work, with only caveat of using Router 2 "external" IP instead of PCs IP in the UPnP messages
With the upnpc cli tool I've been able to get Double NAT port forwarding working.
For my example, let's say I want port 6667 to be forwarded to my machine, 192.168.50.123, behind a double NAT. The first layer of NAT has the IP range 192.168.1.1-255, and then 2nd layer has IP range 192.168.50.1-255.
On a machine inside the 2nd layer of NAT, just setup port forwarding with upnpc like normal.
$ upnpc -a 192.168.50.123 6667 6667 tcp
...
$ upnpc -l # this will confirm the rule is in place
...
Now for the slightly tricky part. I now issue UPNP requests to the outer layer NAT. Since the usual SSDP discovery won't work, I have to specify the XML root description URL manually. It's often something standard like http://192.168.1.1:5000/rootDesc.xml, though on some hardware the port number is randomized. I find just port scanning the gateway address will eventually reveal it.
Now that I have the rootDesc url, http://192.168.1.1:5000/rootDesc.xml, I then issue UPNP command to the outer layer NAT:
$ upnpc -u "http://192.168.1.1:5000/rootDesc.xml" -a 192.168.1.XX 6667 6667 tcp
...
$ upnpc -u "http://192.168.1.1:5000/rootDesc.xml" -l # this will confirm the rule is in place
...
With the -u option instead of discoverying the rootDesc, it just uses the URL provided.
The address 192.168.1.XX should be the 2nd layer network's IP address on as seen by the 1st network. When running the first set of UPNP commands this will be shown since it's the "external" address from that NAT's perspective.
With that now setup, traffic will go like this:
<INTERNET> --> MY_PUBLIC_IP -> 192.168.1.XX -> 192.168.50.123
This question already has answers here:
Does the port change when a server accepts a TCP connection?
(3 answers)
Closed 4 years ago.
I understand the basics of how ports work. However, what I don't get is how multiple clients can simultaneously connect to say port 80. I know each client has a unique (for their machine) port. Does the server reply back from an available port to the client, and simply state the reply came from 80? How does this work?
First off, a "port" is just a number. All a "connection to a port" really represents is a packet which has that number specified in its "destination port" header field.
Now, there are two answers to your question, one for stateful protocols and one for stateless protocols.
For a stateless protocol (ie UDP), there is no problem because "connections" don't exist - multiple people can send packets to the same port, and their packets will arrive in whatever sequence. Nobody is ever in the "connected" state.
For a stateful protocol (like TCP), a connection is identified by a 4-tuple consisting of source and destination ports and source and destination IP addresses. So, if two different machines connect to the same port on a third machine, there are two distinct connections because the source IPs differ. If the same machine (or two behind NAT or otherwise sharing the same IP address) connects twice to a single remote end, the connections are differentiated by source port (which is generally a random high-numbered port).
Simply, if I connect to the same web server twice from my client, the two connections will have different source ports from my perspective and destination ports from the web server's. So there is no ambiguity, even though both connections have the same source and destination IP addresses.
Ports are a way to multiplex IP addresses so that different applications can listen on the same IP address/protocol pair. Unless an application defines its own higher-level protocol, there is no way to multiplex a port. If two connections using the same protocol simultaneously have identical source and destination IPs and identical source and destination ports, they must be the same connection.
Important:
I'm sorry to say that the response from "Borealid" is imprecise and somewhat incorrect - firstly there is no relation to statefulness or statelessness to answer this question, and most importantly the definition of the tuple for a socket is incorrect.
First remember below two rules:
Primary key of a socket: A socket is identified by {SRC-IP, SRC-PORT, DEST-IP, DEST-PORT, PROTOCOL} not by {SRC-IP, SRC-PORT, DEST-IP, DEST-PORT} - Protocol is an important part of a socket's definition.
OS Process & Socket mapping: A process can be associated with (can open/can listen to) multiple sockets which might be obvious to many readers.
Example 1: Two clients connecting to same server port means: socket1 {SRC-A, 100, DEST-X,80, TCP} and socket2{SRC-B, 100, DEST-X,80, TCP}. This means host A connects to server X's port 80 and another host B also connects to the same server X to the same port 80. Now, how the server handles these two sockets depends on if the server is single-threaded or multiple-threaded (I'll explain this later). What is important is that one server can listen to multiple sockets simultaneously.
To answer the original question of the post:
Irrespective of stateful or stateless protocols, two clients can connect to the same server port because for each client we can assign a different socket (as the client IP will definitely differ). The same client can also have two sockets connecting to the same server port - since such sockets differ by SRC-PORT. With all fairness, "Borealid" essentially mentioned the same correct answer but the reference to state-less/full was kind of unnecessary/confusing.
To answer the second part of the question on how a server knows which socket to answer. First understand that for a single server process that is listening to the same port, there could be more than one socket (maybe from the same client or from different clients). Now as long as a server knows which request is associated with which socket, it can always respond to the appropriate client using the same socket. Thus a server never needs to open another port in its own node than the original one on which the client initially tried to connect. If any server allocates different server ports after a socket is bound, then in my opinion the server is wasting its resource and it must be needing the client to connect again to the new port assigned.
A bit more for completeness:
Example 2: It's a very interesting question: "can two different processes on a server listen to the same port". If you do not consider protocol as one of the parameters defining sockets then the answer is no. This is so because we can say that in such a case, a single client trying to connect to a server port will not have any mechanism to mention which of the two listening processes the client intends to connect to. This is the same theme asserted by rule (2). However, this is the WRONG answer because 'protocol' is also a part of the socket definition. Thus two processes in the same node can listen to the same port only if they are using different protocols. For example, two unrelated clients (say one is using TCP and another is using UDP) can connect and communicate to the same server node and to the same port but they must be served by two different server processes.
Server Types - single & multiple:
When a server processes listening to a port that means multiple sockets can simultaneously connect and communicate with the same server process. If a server uses only a single child process to serve all the sockets then the server is called single-process/threaded and if the server uses many sub-processes to serve each socket by one sub-process then the server is called a multi-process/threaded server. Note that irrespective of the server's type a server can/should always use the same initial socket to respond back (no need to allocate another server port).
Suggested Books and the rest of the two volumes if you can.
A Note on Parent/Child Process (in response to query/comment of 'Ioan Alexandru Cucu')
Wherever I mentioned any concept in relation to two processes say A and B, consider that they are not related by the parent-child relationship. OS's (especially UNIX) by design allows a child process to inherit all File-descriptors (FD) from parents. Thus all the sockets (in UNIX like OS are also part of FD) that process A listening to can be listened to by many more processes A1, A2, .. as long as they are related by parent-child relation to A. But an independent process B (i.e. having no parent-child relation to A) cannot listen to the same socket. In addition, also note that this rule of disallowing two independent processes to listen to the same socket lies on an OS (or its network libraries), and by far it's obeyed by most OS's. However, one can create own OS which can very well violate this restriction.
TCP / HTTP Listening On Ports: How Can Many Users Share the Same Port
So, what happens when a server listen for incoming connections on a TCP port? For example, let's say you have a web-server on port 80. Let's assume that your computer has the public IP address of 24.14.181.229 and the person that tries to connect to you has IP address 10.1.2.3. This person can connect to you by opening a TCP socket to 24.14.181.229:80. Simple enough.
Intuitively (and wrongly), most people assume that it looks something like this:
Local Computer | Remote Computer
--------------------------------
<local_ip>:80 | <foreign_ip>:80
^^ not actually what happens, but this is the conceptual model a lot of people have in mind.
This is intuitive, because from the standpoint of the client, he has an IP address, and connects to a server at IP:PORT. Since the client connects to port 80, then his port must be 80 too? This is a sensible thing to think, but actually not what happens. If that were to be correct, we could only serve one user per foreign IP address. Once a remote computer connects, then he would hog the port 80 to port 80 connection, and no one else could connect.
Three things must be understood:
1.) On a server, a process is listening on a port. Once it gets a connection, it hands it off to another thread. The communication never hogs the listening port.
2.) Connections are uniquely identified by the OS by the following 5-tuple: (local-IP, local-port, remote-IP, remote-port, protocol). If any element in the tuple is different, then this is a completely independent connection.
3.) When a client connects to a server, it picks a random, unused high-order source port. This way, a single client can have up to ~64k connections to the server for the same destination port.
So, this is really what gets created when a client connects to a server:
Local Computer | Remote Computer | Role
-----------------------------------------------------------
0.0.0.0:80 | <none> | LISTENING
127.0.0.1:80 | 10.1.2.3:<random_port> | ESTABLISHED
Looking at What Actually Happens
First, let's use netstat to see what is happening on this computer. We will use port 500 instead of 80 (because a whole bunch of stuff is happening on port 80 as it is a common port, but functionally it does not make a difference).
netstat -atnp | grep -i ":500 "
As expected, the output is blank. Now let's start a web server:
sudo python3 -m http.server 500
Now, here is the output of running netstat again:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
So now there is one process that is actively listening (State: LISTEN) on port 500. The local address is 0.0.0.0, which is code for "listening for all". An easy mistake to make is to listen on address 127.0.0.1, which will only accept connections from the current computer. So this is not a connection, this just means that a process requested to bind() to port IP, and that process is responsible for handling all connections to that port. This hints to the limitation that there can only be one process per computer listening on a port (there are ways to get around that using multiplexing, but this is a much more complicated topic). If a web-server is listening on port 80, it cannot share that port with other web-servers.
So now, let's connect a user to our machine:
quicknet -m tcp -t localhost:500 -p Test payload.
This is a simple script (https://github.com/grokit/dcore/tree/master/apps/quicknet) that opens a TCP socket, sends the payload ("Test payload." in this case), waits a few seconds and disconnects. Doing netstat again while this is happening displays the following:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.10:500 192.168.1.13:54240 ESTABLISHED -
If you connect with another client and do netstat again, you will see the following:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.10:500 192.168.1.13:26813 ESTABLISHED -
... that is, the client used another random port for the connection. So there is never confusion between the IP addresses.
Normally, for every connecting client the server forks a child process that communicates with the client (TCP). The parent server hands off to the child process an established socket that communicates back to the client.
When you send the data to a socket from your child server, the TCP stack in the OS creates a packet going back to the client and sets the "from port" to 80.
Multiple clients can connect to the same port (say 80) on the server because on the server side, after creating a socket and binding (setting local IP and port) listen is called on the socket which tells the OS to accept incoming connections.
When a client tries to connect to server on port 80, the accept call is invoked on the server socket. This creates a new socket for the client trying to connect and similarly new sockets will be created for subsequent clients using same port 80.
Words in italics are system calls.
Ref
http://www.scs.stanford.edu/07wi-cs244b/refs/net2.pdf