I'm trying to learn some of the ins and outs of P2P/decentralized networks. My question is the following. Say I have two machines named comp1 and comp2. Now comp1 is setup on my home network behind a router, and comp2 is located in my office at also behind a router. Is it possible for me to send UDP packets back and forth across the Internet like this assuming of course that ports are forwarded properly? To offer more insight on what I'm investigating, I'm trying to figure out how a new node would discover existing nodes without the use of a central server.
Thank you!
Assuming, as you said the ports are forwarded correctly you can send UDP packets to 2 clients behind router's.
A good way of detecting clients on a local intranet may be using Multicast, however this does not have widespread support on ISP's (At least here in the UK) so cannot be relied upon. Multicast is used by many device discovery platforms, such as mDNS (used inApple's Bonjour)
http://en.wikipedia.org/wiki/Multicast
(It basically works by clients subscribing to groups, and then sending messages to that group)
I think the best way of discovering new clients over the internet is to have one server which new clients contact to let it know they exist, then the centralised server will tell all the other clients about you. This is used for example in P2P games such as Modern Warfare 2 and this is what the "Trackers" do in the BitTorrent protocol.
This isn't entirely decentralised but it's probably the easiest to implement, and the most reliable.
To add to Dotmister's response, if the ports are not forwarded correctly (e.g. the router isn't statically configured to forward the ports), you will have to look into something like UDP hole punching. Either way, in order to discover a new node without some sort of central server you'll have to rely on some sort of Multicast.
Related
My knowledge about network programming is limited, so, all the comments are more than welcome. Essentially my question boils down to the following question:
Q1. Is there really such a thing as decentralized asynchronous cross-platform peer-to-peer communication?
Let me explain myself.
If we have two http servers running on computers with actual IP addresses, then clearly the answer is yes, assuming one writes a protocol for the interaction.
To go one step further, if one of them (or both) is (are) behind a router, then, with port forwarding the communication can still be established. However, here the problems start because if someone wants to run such a server on the background, say in a mobile phone, the app that is relying on this server really works when one is at home (we can not really expect to request port forwarding everywhere we go).
But even beyond that,
Q2. do mobile phones obtain an actual IP address from telecommunication companies when someone is not using a wi-fi?
If this is true, then clearly one can have cross-platform asynchronous peer-to-peer communication at the expense of not using wi-fi by running an http server on a smartphone. (I understand that this is not convenient, but it is certainly doable.)
Concluding, the two (perhaps there are more) relevant questions that I can think of are:
Q3. How does Skype really work?
Q4. How does Viber really work?
Based on the answer for Skype, it says: If one of the callee or both of them do not have a public IP, then they send voice traffic to another online Skype node over UDP or TCP.
So, it appears that there is no direct communication in Skype, because they have to use a man-in-the-middle for such a scenario.
Regarding Viber, I could not find a good-thorough answer to this particular question. Do people talk to each other through a Viber centralized server, or, do they establish a direct connection? Of course if they do establish a direct connection, then I really want to know how they manage such a thing since a mobile phone may or may not have a physical address. How is a Viber message routed to my cell phone from a friend of mine even when Viber is not running and I am behind a router?
I guess the answer to Viber is really push notifications, but as far as I can understand, all the variations of push notifications rely on open connections, and then the servers of the applications send the notifications to the clients through such connection(s). So, this approach gives us the feeling that it is asynchronous, but essentially it is not. We are cheating, in the sense that there is a constantly open connection to a server, and moreover, as far as I can understand, the application server has to push the notification through that server. Schematically:
A > Central App Server > Central Server w/ open connection to my cellphone > me
So, this seems to be once again a centralized approach.
Honestly, the only approach that I can think of that is both decentralized and asynchronous (on mobile phones as well) is to run an http server on every platform/device, but this comes at the expense of not using Wi-Fi and assuming that a telecommunication company really assigns a physical IP address to every mobile phone (which I do not know if it is true, do you?).
What about WASTE, darknets, F2Fs, etc? Do they offer advantages in the sense of a more direct asynchronous communication between some interested parties? Are there real-world applications (also including mobile phones) using such approaches for communication.
Really, this is not the actual problem that I would like to work on, but I would like to know what the state of the art is so that I can figure out how I can proceed from there. So, all comments are really more than welcome. If you have references for the state of the art I would like to know about them as well, but a brief description would also be nice.
I appreciate all your time and effort in advance.
You asked many questions, here is the beginning of the answers:
Q1: Yes. For example, take BitTorrent's very successful 10 million+ node network. Aside from the bootstrapping process, the protocol is entirely decentralized and asynchronous. See here for more info.
Q2: Yes! Go to www.whatismyip.com on your mobile telephone, and you will see your assigned IP. However, you are likely to be very filtered (e.g: incoming traffic on port 80 is likely to be blocked).
Q3: It has elements of P2P and clever tricks to get around NAT issues - see here for more info.
Q4: I don't know.
I am trying to write a program that scan an ip range and detect if an ip is address of a router or not.
Currently i used traceroute from my computer to all host in the network. However, i believe there must be some way to directly "ask" a host at an ip if it is a router or not?
by the way, do you know any program/ opensource already does this?
Routers are supposed to talk couple of protocols (actually a neat bunch) that regular IP nodes do not, and then there are some which are more common (i.e. even non-router nodes do).
Router-only protocols:
VRRP
IGRP / EIGRP
OSPF
BGP
RIP
You could do active-probing on those, i.e. send a packet (behaving as if you are another router, or an end-node) and checking to see what kind of response the router (if at all) sends.
Alternatively you could do passive-probing, like 'sniffing', i.e. watching out for the kind of IP packets being sent out by various nodes. There are some which are usually sent out by Routers only (again, mostly from the above list).
Common protocol, but that can actually tell you a lot:
SNMP (esply the unsecure one's like v1/v2, are easy to deal with, without having to establish a secure session)
Other ways:
Portscanning (actually can tell you a real lot), for example all routers have some management ports (although, often they are locked down due to security concerns)
What you want to do is often what many 'Network Management' software do, to "discover" capabilities / functionality of other nodes in the network. And, there isn't a single size-fits all solution. They use bunch of different methods, heuristics to finally figure out what the other node is.
Any node which is hopped to and not just an endpoint is a router. However, this doesn't allow you to detect routers with no reachable devices hooked up. (Any input as to whether my answer has merit would be great!)
We're implementing a SIP-based solution and have configured the setup to work with RTPProxy. Right now, we're routing everything through RTPProxy as we were having some issues with media transport relying on ICE. If we're not mistaken, a central relay server is necessary for relaying streaming data between two clients if they're behind symmetric NATs. In practice, is this a large percentage of all consumer users? How much bandwidth woudl we save if we implemented proper routing to skip the relay server when not necessary. Are there better solutions we're missing?
In falling order of usefulness:
There is a direct connection between the two endpoints in both directions. You just connect and you are essentially done.
There is a direct connection between the two endpoints in one direction. In that case you just connect via the right direction by trying both.
Both parties are behind NATs of some kind.
Luckily, UPnP works in one end, you can then upgrade the connection to the above scheme
UPnP doesn't work, but STUN does. Use it to punch a hole in the NAT. There are a couple of different protocols but the general trick is to negotiate via a middle man that coordinates the NAT-piercing.
You fall back to let another node on the network act as a relaying proxy.
If you implement the full list above, then you have to give up very few connections and don't have to spend much time on bandwidth utilization at proxies. The BitTorrent protocol, of which I am somewhat familiar, usually stops at UPnP, but provides a built-in test to test for connectivity through the NAT.
One really wonders why IPv6 did not get implemented earlier - this is a waste of programmers time.
Real world NAT types survey (not a huge dataset, though):
http://nattest.net.in.tum.de/results.php
According to Google, about 8% of the traffic has to be relayed: http://code.google.com/apis/talk/libjingle/important_concepts.html
A large percentage (if not the majority) of home users uses NAT, as that is what those xDSL/cable routers use to provide network access to the local network.
You can theoretically use UPnP to open ports and set-up forwarding rules on the router to go through the NAT transparently. Unfortunately (or fortunately, depending on who you are) many users disable UPnP as a matter of course on their router and may not appreciate having to add forwarding rules manually.
What you might be able to do (and what Skype does AFAIK) is to have some of the users that have clear network paths and enough bandwidth act as relay nodes. Apart from the routing and QoS issues, you would at least have to find some way to ensure the privacy of any relayed data from anyone, including the owner of the relay node. In addition, there might be legal issues to settle with this approach, apart from the technical ones.
I've seen and read a lot of similar questions, and the corresponding Wikipedia articles (NAT traversal, STUN, TURN, TCP hole punching), but the overwhelming amount of information doesn't really help me with my very simple problem:
I'm writing a P2P application, and I want two users of my application behind NAT to be able to connect to each other. The connection must be reliable (comparable to TCP's reliability) so I can't just switch to UDP. The solution should work on today's common systems without reconfiguration. If it helps, the solution may involve a connectible 3rd-party, as long as it doesn't have to proxy the entire data (for example, to get the peers' external (WAN) IP addresses).
As far as I know, my only option is to use a "reliable UDP" library + UDP hole punching. Is there a (C/C++) library for this? I found enet in a related question, but it only takes care of the first half of the solution.
Anything else? Things I've looked at:
Teredo tunnelling - requires support from the operating system and/or user configuration
UPnP port forwarding - UPnP isn't present/enabled everywhere
TCP hole punching seems to be experimental and only work in certain circumstances
SCTP is even less supported than IPv6. SCTP over UDP is just fancy reliable UDP (see above)
RUDP - nearly no mainstream support
From what I could understand of STUN, STUNT, TURN and ICE, none of them would help me here.
ICE collects a list of candidate IP/port targets to which to connect. Each peer collects these, and then each runs a connectivity check on each of the candidates in order, until either a check passes or a check fails.
When Alice tries to connect to Bob, she somehow gets a list of possible ways - determined by Bob - she may connect to Bob. ICE calls these candidates. Bob might say, for example: "my local socket's 192.168.1.1:1024/udp, my external NAT binding (found through STUN) is 196.25.1.1:4454/udp, and you can invoke a media relay (a middlebox) at 1.2.3.4:6675/udp". Bob puts that in an SDP packet (a description of these various candidates), and sends that to Alice in some way. (In SIP, the original use case for ICE, the SDP's carried in a SIP INVITE/200/ACK exchange, setting up a SIP session.)
ICE is pluggable, and you can configure the precise nature/number of candidates. You could try a direct link, followed by asking a STUN server for a binding (this punches a hole in your NAT, and tells you the external IP/port of that hole, which you put into your session description), and falling back on asking a TURN server to relay your data.
One downside to ICE is that your peers exchange SDP descriptions, which you may or may not like. Another is that TCP support's still in draft form, which may or may not be a problem for you. [UPDATE: ICE is now officially RFC 6544.]
Games often use UDP, because old data is useless. (This is why RTP usually runs over UDP.) Some P2P applications often use middleboxes or networks of middleboxes.
IRC uses a network of middleboxes: IRC servers form networks, and clients connect to a near server. Messages from one client to another may travel through the network of servers.
Failing all that, you could take a look at BitTorrent's architecture and see how they handle the NAT problem. As CodeShadow points out in the comments below, BitTorrent relies on reachable peers in the network: in a sense some peers form a network of middleboxes. If those middleboxes could act as relays, you'd have an IRC-like architecture, but one that's set up dynamically.
I recommend libjingle as it is used by some major video game companies which heavily relies on P2P network communication. (Have you heard about Steam? Vavle also uses libjingle , see the "Peer-to-peer networking" session in the page: https://partner.steamgames.com/documentation/api)
However, the always-work-solution would be using a relay server. Since there is no "standard" way to go through NAT, you should have this relay server option as a fall-back strategy if a connection has to be always established between any peers.
I'm trying to figure out how to use winsockets to be able to turn my game into a LAN-playable game. I've read some winsockets documentation but I can't figure out how a client can get all the games that were created on LAN.
Does it have to try to 'connect' to each IP on LAN, like trying to connect to 192.168.0.1, then 192.168.0.2, etc? Is there a better way?
You would use broadcasting to advertise your servers on the LAN. Clients can then listen for these broadcasts to 'find' servers.
See here for more info:
http://tangentsoft.net/wskfaq/intermediate.html#broadcast
Typically these game servers use the local UDP broadcast, which is something that all clients receive and can process so long as they are listening to it.
Here is some sample client and server code I found that may be of interest to you: http://visual-c.itags.org/visual-c-c++/29424/
I think there are two possible ways to do this.
Make a "lobby" that clients and servers connect to so they can find each other through it.
Servers broadcast UDP packets. Clients listen and update a list of servers.
If you need a quick and easy way, the 2nd option would be great but remeber most of UDP packets will be wasted as they are used only once for each client.
The 1st option is more general and extensile solution to this problem. However, it might need more time to design and implement.
First off, I suggest that you get wireshark for any networking development. It will show you what packet goes through the wire. It will allow you to see how other games do it since there are many ways of doing this.
Using the UDP broadcast is one way of doing it. Simply change the target ip's last byte to 255 and you should be ok.