If a client inside firewall connects to a public server through TCP, once the connection is formed, the firewall allows two-way communication. This is what we normally see in our daily usage. My question is: is this also true for UDP?
Since UDP does not have connect, I will modify the question a little bit...
Suppose the client inside firewall sends a UDP packet to a public server, can the server respond back through firewall using the source address and port of incoming packet?
Yes this is called NAT traversal (or UDP punch through) and works in a similar way to TCP - the stateful NAT device is aware you recently sent a UDP packet from a certain end point internally to a certain end point outside and for a period will accept UDP packets from the same outside endpoint and forward them to the same internal end point.
I always prefer the IETF docs than the plethora of conflicting information out there (including on this site):
https://www.rfc-editor.org/rfc/rfc5128
UPDATE: There are other techniques to "UDP hole punching" such as UPnP's Internet Gateway Protocol and PCP. Each has their advantages and disadvantages and I am afraid there is no one solution works on all NATs!
Related
I'm currently developing a "node-based" system where a server will send out a UDP broadcast on the private network (with a custom protocol), which will be received by several different clients which supports the specified protocol. The server will after the request pick between some of the clients for a more steady TCP connection.
Request for client sequence
Server broadcasting a request-for-ip message to every device/node on the network.
All available clients that supports the protocol will answer with their unique IP to the server.
Server chooses among the clients via a request-for-connection message.
Client that got choosen by the server connects to the server via TCP for a reliable connection.
My question
I've got pretty good knowledge about both TCP and UDP, but I've never designed a system like this before. Do you think this system is built in the right way or is there a more "standard" way doing something similar to this? What are your thoughts?
Thanks!
--- Edit ---
Added a diagram of the program.
There is a standard protocol to advertise services on the network, which you may like to consider: Simple Service Discovery Protocol, based on periodic UDP multicast:
The Simple Service Discovery Protocol (SSDP) is a network protocol based on the Internet protocol suite for advertisement and discovery of network services and presence information. It accomplishes this without assistance of server-based configuration mechanisms, such as Dynamic Host Configuration Protocol (DHCP) or Domain Name System (DNS), and without special static configuration of a network host. SSDP is the basis of the discovery protocol of Universal Plug and Play (UPnP) and is intended for use in residential or small office environments.
In this protocol clients join that UDP multicast group to discover local network services and initiate connections to them, if they wish to. And this is pretty much the intended use case for the protocol, which is somewhat different from your use case.
One benefit of IP/UDP multicast is that multicast packets can be dropped in the network adapter if no process on the host has joined that multicast group. Another one is that IP/UDP multicast can be routed across networks.
From the diagram you posted:
The server is the mediator (design pattern) whose location must be known to every other process of the distributed system.
The clients need to connect/register with the server.
Your master client is a control application.
It makes sense for the server to advertise itself over UDP multi-cast.
Online clients would connect to the server using TCP on start or TCP connection loss. If a client terminates for any reason that breaks the TCP connection and the server becomes immediately aware of that, unless the client was powered off or its OS crashed. You may like to enable frequent TCP keep-alives for the server to detect dead clients as soon as possible, if no data is being transmitted from the server to the clients. Same applies to the clients.
All communications between the server and the clients happen over TCP. Otherwise you would need to implement reliable messaging over UDP or use PGM, which can be a lot of work. Multicast UDP should only be used for server discovery, not bi-directional communication that requires reliable delivery.
The master client also connects to the server, possibly on another port, for control. The master client can discover all available servers (if there is more than one) and allow the user to choose which one to connect to.
Let's assume that I start a server at one of the computers in my private network (192.168.10.10:9900).
Now when making a request from some other computer in the same network, how does the client computer (OS?) knows which protocol to use / which protocol the server follows ? [TCP or UDP]
EDIT: As mentioned in the answers, I was basically looking for a default protocol which will be used by the client in the absence of any transport protocol information.
TCP / UDP protocols work at the transport layer level (TCP / IP MODEL) and its main difference is that TCP has a method to ensure the arrival of messages while UDP is lighter because of its virtue is to be faster in Information delivery. The use of one protocol or another is always defined by the application that will use it.
So the reference you put on the private server with ip: port 192.168.10.10:9900 is very vague to be more precise we could say that we have an Apache web server running on the ip: port 192.168.10.10:9900 (the port for default is 80 when installing the server, but it can be changed in the configuration).
Now the web servers (apache, IIS, etc.) work using the TCP protocol because when a client (computer, cell phone, etc.) consults a page through a browser (Chrome, Firefox, etc.), the ideal thing is that all the website and not just some pieces. This is why this type of servers chose and use this protocol in the first instance since they seek that in the end the result is that the user obtains the complete page regardless of whether a few more milliseconds are sacrificed in the validations involved in using TPC.
Now going to the client side. The user when visiting a web page from any browser (Chrome, Firefox, etc.) will use TCP since this protocol is already configured in the browser to send the query messages and subsequently receive the messages with the same form Website information.
Now this behavior is going to be repeated for any client / server application. For example, to change the type of application on the UDP side, we can observe the operation of DHCP services which are used to receive an IP when connecting any device to a Wi-Fi network. In this case, this service seeks to be as fast as possible (instead of the most reliable) since you want the device to connect as quickly as possible to the network, so use the UDP protocol and in this case any equipment when connecting To a WIFI network you will send your messages using this protocol.
Finally, if you want to know promptly about the type of TCP / UDP protocol used by a specific application, you can search on the Wireshark application which allows you to scan the messages that leave the device or show the protocol used in the different layers of the application.
There is no reason any client would make a request to your server, so why would it care what protocol it follows? Clients don't just randomly connect to things to see if there's a server there. So it doesn't make any difference to any client.
Normally, the client computer will use the TCP protocol as default. If you start the server using UDP protocol mode, then when you use curl -XGET 192.168.10.10:9900/test-page, it will give you back an curl: (7) Failed to connect to 192.168.10.10 port 9900: Connection refused error. You can try it, use the nc -lvp 9900 -u, it will give you that result.
The answers here are pointing to some default protocol. Its' not that, Whenever you start an application let say HTTP server, the server's internal has code to open a socket(which can be TCP or UDP), since HTTP:80 is a TCP protocol the code creates a TCP socket. Similarly for other network application it depends on their requirement what kind of transport layer protocol to use (TCP Or UDP). Like a DNS client will create a UDP socket to connect to DNS server, since DNS:53 is mostly over UDP. Both TCP and UDP have different use cases, advantages and disadvantages. Depending on there uses cases / advantages / disadvantages of UDP/TCP decision is taken to implement server using either of them.
I'm struggle with what technique to choose for a server client aspect of my application.
Defining design
Windows, C# on .net 2
On many machines there is a .net 2 service. I call that the Client.
Machines can be in different networks behind NAT's (or not) connected to Internet.
Server services are public.
Requirements
To communicate with the Clients on demand.
Client must listen for incoming connections.
The server can be or not online.
Port forwarding is not possible.
What are my choices to do something like that?
Now I'm looking in the UDP Hole punching technique. The difference between the UDP hole punching technique setup and my setup is that instead of having 2 clients behind a NAT and a mediation Server, I got only the client behind the NAT that must communicate with the server. That must be easier but I'm having hard time to understand and implement.
I'm on the right way with the this kind of NAT traversal or may be some other methods much easier to implement?
Other methods that I've taken in consideration:
When the service sees the server online, creates a connection to the server using TCP. The problem is that I have something around 200 clients, and the number is rising and I was afraid that this is a resource killer.
When the service sees the server online, checks a database table for commands then at every 30 seconds checks again. This is also a resource killer for my server.
Bottom of line is, if the UDP Hole Punching tehnique is the right way for this scenario, please provide some code ideas for de UDPServer that will run on the service behind NAT.
Thank you.
Hole punching and p2p
You might be interested in a high level discussion of UDP hole punching. Hole punching is needed if you want clients (who both might be behind a firewall) to communicate directly without an relaying server. This is how many peer 2 peer (p2p) communications work.
With p2p, typically NAT'ed clients must use some external server to determine each other's "server reflexive address." When NAT translation occurs, behind the firewall ports can be mapped to some arbitrary port to the public. A client can use a STUN server to determine its "server reflexive address." Clients then will, through an intermediary server, exchange server reflexive addresses and can initiate communication (with hole punching to initiate the session).
Often, a NAT will not behave in a manner to allow direct communication as described above. Sending packets to different destinations will cause a NAT to map ports to entirely different values depending on the destination. In this case, a TURN server is needed.
Links
Nat traversal and different types of NAT behaviors
STUN RFC
TURN RFC
Server-client communication
If your client only needs to communicate with a server, hole punching is not needed. As long as the client can communicate with the public Internet, then you can use any C# socket API (I'm not familiar with C#) to make connections to the server's public IP/port combination. Typically, clients making socket connections don't specify a source port and let the underlying socket API make that decision since it really doesn't matter.
Your server should be listening to a specific port (you make this determination), and when it receives a packet from the client, the source address of the packet will be some NAT'ed address. In other words, the source address will be the public IP of whatever firewall your client is behind. If the NAT changed the source port of the client's packet, the server will see this NAT'ed port as the source port. It really doesn't matter, since when the server sends back a response packet, the NAT on the client machine will translate the destination port (it stores translations internally) and correctly send the packet back to the correct private host (the client).
Is there a way in which the http connection and tcp connection listeners can interact with each other?
I have two separate application modules one is working through http and other requires tcp .
I need to do an interaction between these two modules so is there way i can make my http based module interact with tcp based module.
First of all, you need to read up a little on networking concepts. HTTP is what's known as an application level protocol, whereas TCP is what's known as a transport layer protocol. Take a look at the OSI Network Model.
As an example, you can imagine that TCP is the telephone network. It gives you the basic means to connect to another person and speak with them. However, in order to actually communicate you need to speak the same language, such as English or French. That is the application level protocol, HTTP in your case.
So to answer your question, in order for your two applications to to communicate and exchange data they need to make a connection / call using TCP and both be speaking the same language / application level protocol namely HTTP.
Two distinct processes won't be able to use the same IP port on a same IP address. Thus, two processes won't be able to use the same incoming stream of data coming out of the TCP connection. If they use different ports, there's no problem.
If the two processes use the same IP port, as HTTP is a protocol that sits on top of TCP, it means that your TCP process can be used as a pipe by the HTTP process. The TCP process will connect to the IP port, do its stuff, and forward the data to the HTTP process that will handle it.
I've noticed recently that I don't have to forward ports for mmorpg's that I play.
I'm thinking about working on a game that people can play online and had a question.
Why is this the case given its a two way socket connection that is constantly sending data back and forth? Doesn't their server need to get through my firewall in order to connect to me?
TCP crash lesson: TCP is a two-way protocol. The challenge is that at least one host needs to initiate. Since within an MMORPG, your own computer is never acting as a server, nobody has to connect to it. All the information about game state is passed through the company's public facing servers that have public facing IP addresses (and hey, maybe they actually use port forwarding there, just to confuse my explanation... but you never have to see their pretty network internals, proxies, and other wizardry.).
Anyway, when you connect to Stackoverflow, you're making one outbound connection that requests data from the server, and then over that same connection you're receiving it back. Same exact scenario, only with a webserver instead of a game.
UPnP allows you to tackle many routers. There's also NAT Punch-Through if you have access to a third party that isn't behind a router.
Either way, port forwarding is only necessary if you wish to act as a server (or the sender in a P2P relationship). A client does not need to forward ports.
You don't need to forward ports to access the web either, despite data coming in as well.
When you make an outgoing TCP connection, your NAT router puts the connection in a table, so that when data comes in, it knows what machine in your LAN to send the packet to.
Everyone mentions TCP, but NAT works for UDP as well: The first outgoing UDP packet associates that source port with the internal IP address, and your NAT device will forward incoming traffic to that port to the correct host on the internal network.
In other words, if your computer requests the connection (outbound) first, the router opens up the port automatically, on the assumption that you're going to want data back. But if you want remote users to connect to your computer without your computer requesting it, the router would normally drop the packets since it wouldn't know where to send them (they were unsolicited). So instead, you need to tell the router to deliver any unsolicited packets at port N to your computer.
Sorry to add another answer so late, and I know one was already accepted, but I personally found the other answers to be more confusing than this simple explanation.