I'm struggle with what technique to choose for a server client aspect of my application.
Defining design
Windows, C# on .net 2
On many machines there is a .net 2 service. I call that the Client.
Machines can be in different networks behind NAT's (or not) connected to Internet.
Server services are public.
Requirements
To communicate with the Clients on demand.
Client must listen for incoming connections.
The server can be or not online.
Port forwarding is not possible.
What are my choices to do something like that?
Now I'm looking in the UDP Hole punching technique. The difference between the UDP hole punching technique setup and my setup is that instead of having 2 clients behind a NAT and a mediation Server, I got only the client behind the NAT that must communicate with the server. That must be easier but I'm having hard time to understand and implement.
I'm on the right way with the this kind of NAT traversal or may be some other methods much easier to implement?
Other methods that I've taken in consideration:
When the service sees the server online, creates a connection to the server using TCP. The problem is that I have something around 200 clients, and the number is rising and I was afraid that this is a resource killer.
When the service sees the server online, checks a database table for commands then at every 30 seconds checks again. This is also a resource killer for my server.
Bottom of line is, if the UDP Hole Punching tehnique is the right way for this scenario, please provide some code ideas for de UDPServer that will run on the service behind NAT.
Thank you.
Hole punching and p2p
You might be interested in a high level discussion of UDP hole punching. Hole punching is needed if you want clients (who both might be behind a firewall) to communicate directly without an relaying server. This is how many peer 2 peer (p2p) communications work.
With p2p, typically NAT'ed clients must use some external server to determine each other's "server reflexive address." When NAT translation occurs, behind the firewall ports can be mapped to some arbitrary port to the public. A client can use a STUN server to determine its "server reflexive address." Clients then will, through an intermediary server, exchange server reflexive addresses and can initiate communication (with hole punching to initiate the session).
Often, a NAT will not behave in a manner to allow direct communication as described above. Sending packets to different destinations will cause a NAT to map ports to entirely different values depending on the destination. In this case, a TURN server is needed.
Links
Nat traversal and different types of NAT behaviors
STUN RFC
TURN RFC
Server-client communication
If your client only needs to communicate with a server, hole punching is not needed. As long as the client can communicate with the public Internet, then you can use any C# socket API (I'm not familiar with C#) to make connections to the server's public IP/port combination. Typically, clients making socket connections don't specify a source port and let the underlying socket API make that decision since it really doesn't matter.
Your server should be listening to a specific port (you make this determination), and when it receives a packet from the client, the source address of the packet will be some NAT'ed address. In other words, the source address will be the public IP of whatever firewall your client is behind. If the NAT changed the source port of the client's packet, the server will see this NAT'ed port as the source port. It really doesn't matter, since when the server sends back a response packet, the NAT on the client machine will translate the destination port (it stores translations internally) and correctly send the packet back to the correct private host (the client).
Related
I'm currently developing a "node-based" system where a server will send out a UDP broadcast on the private network (with a custom protocol), which will be received by several different clients which supports the specified protocol. The server will after the request pick between some of the clients for a more steady TCP connection.
Request for client sequence
Server broadcasting a request-for-ip message to every device/node on the network.
All available clients that supports the protocol will answer with their unique IP to the server.
Server chooses among the clients via a request-for-connection message.
Client that got choosen by the server connects to the server via TCP for a reliable connection.
My question
I've got pretty good knowledge about both TCP and UDP, but I've never designed a system like this before. Do you think this system is built in the right way or is there a more "standard" way doing something similar to this? What are your thoughts?
Thanks!
--- Edit ---
Added a diagram of the program.
There is a standard protocol to advertise services on the network, which you may like to consider: Simple Service Discovery Protocol, based on periodic UDP multicast:
The Simple Service Discovery Protocol (SSDP) is a network protocol based on the Internet protocol suite for advertisement and discovery of network services and presence information. It accomplishes this without assistance of server-based configuration mechanisms, such as Dynamic Host Configuration Protocol (DHCP) or Domain Name System (DNS), and without special static configuration of a network host. SSDP is the basis of the discovery protocol of Universal Plug and Play (UPnP) and is intended for use in residential or small office environments.
In this protocol clients join that UDP multicast group to discover local network services and initiate connections to them, if they wish to. And this is pretty much the intended use case for the protocol, which is somewhat different from your use case.
One benefit of IP/UDP multicast is that multicast packets can be dropped in the network adapter if no process on the host has joined that multicast group. Another one is that IP/UDP multicast can be routed across networks.
From the diagram you posted:
The server is the mediator (design pattern) whose location must be known to every other process of the distributed system.
The clients need to connect/register with the server.
Your master client is a control application.
It makes sense for the server to advertise itself over UDP multi-cast.
Online clients would connect to the server using TCP on start or TCP connection loss. If a client terminates for any reason that breaks the TCP connection and the server becomes immediately aware of that, unless the client was powered off or its OS crashed. You may like to enable frequent TCP keep-alives for the server to detect dead clients as soon as possible, if no data is being transmitted from the server to the clients. Same applies to the clients.
All communications between the server and the clients happen over TCP. Otherwise you would need to implement reliable messaging over UDP or use PGM, which can be a lot of work. Multicast UDP should only be used for server discovery, not bi-directional communication that requires reliable delivery.
The master client also connects to the server, possibly on another port, for control. The master client can discover all available servers (if there is more than one) and allow the user to choose which one to connect to.
Hi I am slightly confused on how UDP Hole Punching works and how I would implement it. According to this wikipedia article:
https://en.m.wikipedia.org/wiki/UDP_hole_punching#Flow
Both the clients that want to establish a p2p connection must set up a UDP conversation with the server in order to exchange ip's and punch holes. What I am confused on is lets say client a wants to initiate a p2p conversation with client b. How would client b know to connect to the sever in order for the clients to swap ip's? This is required else they would not know the other clients ip. Am I misunderstanding this concept somehow?
In the regular case the peers do not have static IP addresses and also public ports are allocated dynamically with transient routing rules, valid for typically 1 to 3 minutes.
There is no way to guess the dynamic port of the dual peer and even there is no way to establish immediate routing to it without predefined forwarding rules.
In contrast to often transcribed documentation, the hole punching through router + internet service provider is actually done with sending UDP packets to a public mediator server.
The peers contact each other by re-using exactly the public ip/port currently seen by the mediator server.
The mediator server's router necessarily has a forwarding rule to the server, so the server is publicly reachable and a public communication can be initated.
If the mediator server does not have a static address, a public DNS server is required to resolve the server's dynamic address.
For trapping local port mapping and returning packets to the right target,
all three nodes will maintain a mapping of a unique static node identifier to the current ip/port of incoming packets; each of the clients needs to send periodically alive message with client identifier to the mediator server and counterpart; the mediator server responds with an alive-acknowledge message carrying the mediator server's current public address/port. Here, optional port mapping rules of the routers are to be considered to get the actual public ports.
The dynamic adjustment of remote ports makes it difficult to have several independent communication channels, at least I'm not aware of a fork mechanism for UDP servers.
If there is a requirement for independent communication channels, e.g. like in FTP you have a command port and a streaming port, the packet protocol may be extended by a logical port and incoming packets may be dispatched according to the logical port.
Finally there are security risks:
1.) the communication could be hi-jacked, by anyone sniffing on any node of the routing path; he could send and alive message from a different address to one of the peers and so would inherit the peer's communication stream.
The minimum solution here is to add authentication to the alive messages.
2.) Certainly, encryption is mandatory for user data in any public network!
Due to uncertainty of delivery of UDP packets, encryption is just possible on packet base, as e.g. AES/ECB does, and so should be chosen strong.
I know there needs to be a STUN/ICE/TURN server to find the IP addresses of the peers involved in a WebRTC communication. However, even after IPs are found, how do the peers actually talk to each other independently without having any ports opened?
If you build a website, you usually have to open the ports on your server to have others access your site. What's the magic that is happening in WebRTC that I'm not understanding?
There are several strategies to do this: one possibility is for the client to explicitly open a port via UPnP. I'm not sure if any current WebRTC client does so, but in general networking this is a possibility.
Failing that, the STUN server kicks in. There are several hole punching techniques it can try; read the aforelinked article for the gory details. In short though, a firewall will usually open a port for outgoing traffic (because it needs to receive responses), so by establishing an outgoing connection to a known target and then making note of the port that was opened it is possible to open a port.
Failing even that, a TURN server is necessary. This server is publicly accessible from both peers, even if both peers cannot see each other. The TURN server then will act as a relay between the two. This somewhat negates the point of a P2P protocol, but is necessary in a certain percentage of situations (estimates range around 10%-20%).
The original Question is "what/who creates the sockets?"
The browsers creates the socket and bind them to a local port for you
during the "ICE gathering".
Wether you use any stun/turn server or
not, each candidate generated during the ice gathering has a
corresponding port open.
Those ports are usually open only for 30 mn
after which they are revoked to avoid an attack by someone using old
and/or spoof candidates. These 30mns are not specified in any
specification and are an arbitrary choice by the browser vendor. -
The next question is "how does the remote peer know about which ports are open".
through the ICE mechanism, which for each media will generate potential candidates and send them to the remote peer through your preferred signaling channel.
ICE candidates (which are one line of SDP, really) have a "type". if this type is HOST, then your candidate is a local candidate generated without the use of any stun or turn server. is the type is SRFLX, then you have used a STUN server to add the mapping between your local IP:port and your public IP:port. if your type is RELAY, same thing with a TURN server.
of course, using the local IP:port HOST candidate will fail unless the remote peer is on the same local network.
From the browser and local system point of view, the socket is open on the local IP:PORT anyway. Hence, opening the sockets and finding out on which port a remote peer should connect to connect to the socket are separate problems handled separately.
The Final question is: "can it really work without a STUN server"
Most probably no, unless you are on the same sub network.
Stats shows (http://webrtcstats.com) that even with a STUN server, you still fail in 8% of the case, for the general public. It's much more in enterprise, where you'd better have advanced turn (supporting tunneling through TCP/80 and TLS/443) and even support for HTTP proxy's CONNECT method.
The scenario is the following. I have two machines A and B:
A: Client (behind NAT)
B: Server (behind NAT)
I want B to be able to listen on any given port, so that A can send packets to B through that specific TCP port and receive any response. If both machines are not behind a NAT it is pretty straight foward process. However how do I make it work so that it works even when B is behind a router, without him having to go change the router configuration enable some port forwarding etc...
For example, how do peer-to-peer programs like torrent clients work without the user having anything to configure?
To answer the example of Peer to Peer programs, and in general: There is a technology called Universal Plug and Play which NAT routers can use to allow clients behind them to expose ports to the outside. That's what bittorrent clients can use so the other clients can directly connect to them.
An alternative to a proxy server is a match-making server. Instead of proxying all of the traffic, the match maker just negotiates until the peers can talk to each other. This involves finding the external public IPs of the peers and talking to each one so that the firewall/router knows that the peers wish to communicate.
This is called hole punching and it often has to be done by the match maker rather than the peers themselves. Once the hole are punched though, the match maker can tell the peers about each other and they can communicate directly.
You will have to either:
Set up port forwarding from the nat
gateway in front the server into the machine your server software is running, and have the client
connect to the IP address of that
gateway.
Create a proxy server sitting
inbetween the 2 nat gatewys so both
your server and client can connect
to that. Both your server and client
have to set up a connection to that
proxy which will mediate the data
between those 2 connections.
Hole punching is moderately well-understood for UDP communication, but it can be reliably used to set up peer-to-peer TCP streams as well. Here is the well detailed article on both TCP and UDP:
http://www.brynosaurus.com/pub/net/p2pnat/
I was wondering that how application like skype ( a popular chat client ) works in local network with one router, How it can listen on particular port?
for example:=
In one network A and B are two machines running skype , gateway of both is G1,
now how A and B will have same IP on internet that is of G1, but how can they ensure that they are listening on different ports? How can they ask to router G1 for unique port.
I want to make a simple text chat server on linux. How can I have connections between two different computers in two different networks?
Solution to your problem is to have a forwarding server somewhere in the net.
Different programs use different means to connect to each other. But every chat server, including Skype, has a server, which forwards data or information about subnet IP/port availability.
There are two types of clients: "listening" clients and "passive" ones. Listening clients have direct access to Internet via router port forwarding, and "passive" ones have to use additional tricks to get their hands on external data, line external servers or additional ports to listen.
The point is, not clients connect to each other, but they connect to a server, which then connects back to them to verify they are available, and, if at least one of them is not firewalled, direct another on to connect to the first one, excludint itself from further communication. And if both are firewalled, then is has to forward their messages through itself.
Host Discovery
Manual discovery, client A knowns who client B is
Discovery through broadcast UDP which is used by lot of games for LAN play. A client sends out a packet to the broadcast address for their subnet. The peers can choose to pick up this broadcast and respond. The downside is that this is limited to the current subnet. The more general INADDR_BROADCAST (255.255.255.255) works for all subnets on the local-link, but it cannot be routed, so won't work over internet (this is what DHCP auto-configuration uses).
Discovery through a central (Rendezvous) server. Each individual client knows the address of the server, and the latter informs them about each other. This technique is used by IRC, Voip, IMs and by most 'peer-to-peer' networks.
Communication
After the initial discovery is done you want to be able to talk to eachother. On the internet this can get tricky. Most people nowadays have their own router and sit behind a NAT, so direct connections are impossible.
Using a Rendezvous server, you can possibly talk to each other using the server itself. client A tells the server what to say, and it in turn tells client B, since both clients have an outbound connection to the server.
It is possible for the clients to talk to each other without the server proxying. This requires either DMZ, port forwarding or UPnP. DMZ will basically forward all incoming connections on all the ports to a given local IP. Port forwarding only forwards certain ports to local IPs. UPnP is a bit more advanced, the client requests that the router temporarily forwards a port to it, and you tell the other client via the rendezvous server where to connect.
Chatting app implementation
The easiest solution to your problem is most likely to use a central server, which is known by all the clients, that proxies host discovery and possibly the communication between the clients. If you want the clients to communicate directly, you can just proxy host discovery, and then let either DMz, manual port forwarding or UPnP do the rest.
Another solution would be to just have direct communication through NAT traversal techniques discussed above, and do manual host discovery.
Yet another solution would be to use a public webserver and 'abuse' its ability to insert content to chat with each other.
You need a central UDP Rendezvous Server.
After the initial connection from the client to the server the UDP clients can be redirected to talk to eachother directly even if firewalled.
The trick is to open an UDP connection from the inside.
Check out Real-Time Media Flow Protocol and how they use it.
Check out UDP Hole Punching
alt text http://labs.adobe.com/technologies/stratus/images/p2pvideo_250x215.jpg
Traditional NAT servers replace the source address and port with the address and a random port number of the external interface of the NAT server. This works well for simple protocols such as HTTP and SMTP, but it can create problems for more complex protocols that require multiple response ports on the external interface of the NAT server. NAT servers also aren’t aware of information stored in the data portion of the application layer header without the help of NAT editors and similar software fixes.
Windows XP’s answer to these problems is NAT Traversal, which can automatically allow the UPnP-enabled NAT client application to communicate with a UPnP NAT device. NAT Traversal provides methods to allow the UPnP client to learn the public IP address of the NAT server and to negotiate dynamically assigned port mappings for UPnP NAT client applications.
NAT Traversal features can be built into any hardware device or software application. Applications that commonly cause troubles for NAT devices but work well when UPnP-enabled include the following:
Multiplayer Internet games
Audio and video communications
Terminal Services clients and servers
Peer-to-peer file sharing applications
When these applications are UPnP-enabled, access through the Windows XP ICS allows them to work seamlessly.
Unless A and B are actually "listening" to the responses to outgoing requests, your router will need to be cofigured to forward the relevant port numbers to the relevant hosts. This isn't something that you can request in the code, it's something you need to configure on the router itself.