I want to check NAT type and its behaviour with STUN server available (Public Stun Servers) (STUNTMAN). I have tried and found client codes (STUN CLIENT) which can communicate with STUN server to identify NAT behaviour, but all are in UDP. As per RFC-5780 we can check Mapping behaviour for UDP and TCP, Hence I want to check the same using TCP and i have implemented the same way as UDP does but both results differs. So, will behaviour/Steps to find NAT behaviour be the same as UDP or need to implement other way.
Any Sample client code will also be helpful.
STUN/TURN check the NAT type by simulating the connection of several differnt types, and step by step it can split Full Cone, Restricted NAT, Port Restricted NAT and Symmetric NAT.
We usually use UDP cause we don't need three way heandshake to establishe a connection, which may confuse the judgement of NAT types. So if you want to use TCP, you need first know how STUN check NAT types step by step for UDP, and think about what will happen for TCP three-way handshake. It can not just copy the code.
Related
I am trying to connect to peer via add_peer() function in LibTorrent. But what if the peer from which I want to download the file is behind NAT? Is there something for NAT Traversal in Libtorrent?
The NAT traversal in libtorrent is limited to:
Explicit port forwards using UPnP, NAT-PMP and PCP.
Implicit (opportunistic) attempts to reach peers via their external port
The peer receiving the connection attempt is not behind a NAT, but the initiating peer is. This is the case NATs are meant to support.
It sounds like you're mostly interested in (2), where we assume both peers are behind a NAT. This is commonly referred to UDP hole-punching.
Generally, if you don't control or have any influence over the peer you're trying to connect to, you're limited in what measures you can take.
Also, if the neither NAT is a full-cone (or let's say, p2p-friendly) it may not be possible for the peers to connect. A p2p-friendly NAT generally accepts incoming connections from IPs they have not had any interaction with previously.
The main two approaches used by libtorrent (and bittorrent clients generally) are:
commonly connected peers may introduce two NATed peers to each other via the peer exchange extension. In this mode both peers try to connect to each other simultaneously, hoping that both NATs will open up pin-holes for the ports that are being attempted. This only works if the swarm has at least one peer that's not behind a NAT. You can find more information about this in BEP 55
Sharing the UDP port for uTP, DHT and UDP trackers and having the listen port be implied by the source port of the tracker and DHT announce. With some luck, that source port can also be used by other hosts to reach the NATes client. This works because uTP connections also run over UDP.
As I understand it TCP is required for congestion control and error recovery or reliable delivery of information from one node to another and its not the fastest of protocols for delivering information.
Some routing protocols such as EIGRP and OSPF ride directly on top of IP. Even ICMP rides directly over IP.
Why is UDP even required at all? Is it only required so that developers/programmers can identify what application the inbound packet should be sent to based on the destination port number contained within the packet?
If that is the case then how is information gathered from protocols that ride directly on top of IP sent to the appropriate process when there is no port number information present?
Why are voice and video sent over UDP? Why not directly over IP?
(Note that I do understand thoroughly the use case for TCP. I am not asking why use UDP over TCP or vice versa. I am asking why use UDP at all and how can some protocols use directly the IP layer. Whats the added advantage or purpose of UDP over IP?)
Your question makes more sense in terms of why is UDP useful (than why is UDP required).
UDP is a recognized protocol by the Internet Assigned Numbers Authority. UDP can be useful if you want to write a network protocol that's datagram based and you want to play more nicely with Internet devices.
Routers can have rules to do things like drop any packet that doesn't make sense. So if you try and send packets using say an unassigned IP protocol number between hosts separated by one or more routers, the packets may well never get delivered as you've intended. The same could happen with packets from an unrecognized UDP protocol but that's at least one less door to worry about whether your packet can make it through.
Internet endpoints (like hosts) may do similar filtering too. If you want to write your own datagram based protocol and use a typical host operating system, you're more likely to need to write your software as a privileged process if not as a kernel extension if you're trying to ride it as its own IP protocol (than if you'll be using UDP).
Hope this answer is useful!
First of all, IP and UDP are protocols on the different layers, IP by definition is Internet layer when UDP is transport layer. Layers were introduced to simplify network protocols architecture and to separate concerns. Application layer protocols are supposed to be based on transport layer (with some exceptions).
Most popular transport protocols (in IP network) are UDP and TCP. While TCP is feature rich but with many tradeoffs UDP is very simple but gives a lot of freedom and so typically is a base for other protocols.
The main feature of UDP is multiplexing: ports that allow multiple protocol instances (aka sockets) to coexist on the same node. This means that implementing your own protocol over IP instead of UDP either you won't be able to have multiple instances of your protocol on the same machine or you'll have to implement multiplexing yourself.
There're other features like segmentation and checksum. These features are not mandatory.
And as was mentioned in another answer there're lots of middleware like routers, NATs and firewalls that can ruin the idea of a custom "right over IP" protocol, but it's more like a collateral damage than a feature of UDP.
I started to explore the option of connecting with other using a p2p connection, so I coded a simple socket program in JAVA for android devices in which the users can share simple messages p2p (I didn't have any idea about NAT then). I got to know about NAT, so I now need to establish a TCP connection with another user which uses a server for discovery but payload is transferred p2p. I have also looked at XMPP(a very good and detailed explanation of how protocol works is here) and UPnP but I dont know how to implement them.
Another interesting question that arises is of BitTorrent because they can work on any device and even behind a NAT. I am not able to get any explanation of how BitTorrent works.
I have researched a lot but I am stuck.
My questions are:
A detailed explanation of BitTorrent(like here, not how torrents work) and how is it able to work around NAT ?
Is there a way to make a NAT entry programmatically ?
Is socket programming sufficient for p2p ?
How difficult is it to create your own protocol and how can I build one ?
If two devices D1 and D2 want to communicate p2p and they know each other's IP. D1 sends a request to D2 and that can't get through the D2's NAT, but there should be an entry created in D1's NAT. So when D2 tries to send something D1's NAT should discover an entry with D2's IP. Then why is the packet not allowed by it ?
Another interesting question that arises is of BitTorrent because they can work on any device and even behind a NAT. I am not able to get any explanation of how BitTorrent works.
This statement looks like you assume that bittorrent needs full connectivity to operate.
That is incorrect.
Behind a NAT device you will still be able to establish outgoing TCP connections. Which generally is sufficient for bittorrent as long as there are other, non-NATed (or NATed but properly port-forwarded) clients in the network that can accept incoming connnections.
NAT has no impact on the flow direction of the data because connections are bi-directional once they are established. It only is problematic for the initial connection setup.
This works perfectly fine for bittorrent because bittorent does not care from which specific node you get your data.
Although better connectivity generally does improve performance.
If the identity of the node matters or one-on-one transfers are an important use-case then other p2p protocols usually attempt NAT traversal first and if that fails rely on 3rd party nodes relaying traffic between those nodes who cannot connect to each other directly.
Additionally, IPv6 support will become essential in the future to maintain end-to-end connectivity because more and more ISPs are starting to roll out carrier-grade NAT for IPv4 while IPv6 will remain non-NATed
One thing need to be clear is that 100% P2P between all type of NAT is impossible right now. There is no practical way to establish P2P connectivity between **Symmetric and Symmetric/PRC NAT. In this scenario connection is established through a relay server called TURN.
I am answering from your 2nd question because I don't know much about the first one.
2) Yes. You can send a packet through your NAT and there will be a mapping between your internal IP:Port to your NAT's external IP:Port. You can know these external IP:Port by sending a stun request. Note that this technique doesn't work for Symmetric NAT.
3)Yes socket programming sufficient for p2p.
4)Why do you need a protocol when there already exists several. ICE protocol is the best today for NAT traversal and I don't think it was easy to create. UPnP and NAT-PMP is really vulnerable in terms of security.
5)I think what happens is usually NAT blocks unknown packets coming to it. So when D1 sends a packet to D2, its NAT blocks all packets incoming from D1s IP:Port. That is why connection establishment fails. You have to employ hole punching technique for D1 and D2 to successfully establish P2P connectivity.
**By symmetric NAT I mean symmetric NAT with random port allocation.
There is a paper on "Peer-to-Peer Communication Across Network Address Translators" which describes the UDP hole punching method and extends it to be used over TCP as well.
Of course, you will always need a relay server for the cases where hole punching is not supported.
Recent versions of BitTorrent use µTP, which is layered above UDP, not TCP. µTorrent uses a private extension (ut_holepunch) that performs UDP hole punching, most other implementations don't bother (with the notable exception of Tixati).
Some NAT routers accept port forwarding requests using either the uPNP or the PMP protocol. Whether this is supported depends on the particular brand of router and its configuration.
Yes, socket programming is enough for P2P.
Difficult to answer. I suggest that you read the wikified and annotated BitTorrent specification for a start.
Yes, this is the principle behind UDP hole punching.
I've seen while searching that it is possible to use two different programs on the same computer communicating over the network using the same port and same network interface provided one use UDP and the other TCP. However I didn't get a good explanation, how does it actually work and why this is possible?
Is it also possible for multiple programs to use the same UDP port since UDP does not establish a real connection between the peers, but just sends the packets to an address? I understand it's not possible with TCP as it creates a synchronized connection between the server and the client, but what about UDP?
Please explain in details if possible, or link a good article on the topic.
The other answers are correct but somewhat incomplete.
An IP (aka "INET") socket "connection" (i.e. communication between two processes, possibly on different machines) is defined by a 5-tuple: protocol, source address, source port, destination address, destination port. You can see that this is not limited to a stateful connection such as TCP.
This means that you can bind different processes to any unique instance of that 5-tuple. Because the "protocol" (e.g. TCP and UDP) is part of the differentiating factor, each can have a different process.
Theoretically, you could bind different services to the same TCP port if they bind to different interfaces (network cards, loopback, etc.) though I've never tried it.
It is standard practice, however, to always use the same service on the same port number. If both UDP and TCP are supported, they're just different ways of communicating with that same service. DNS, for example, uses UDP on port 53 for lookup because they are small requests and it's faster than creating a TCP connection but DNS also uses TCP on port 53 for "transfers" which are infrequent and can have large amounts of data.
Lastly, in complete accuracy, it isn't necessarily a 5-tuple. IP uses the "protocol" to pass to the next layer such as TCP and UDP though there are others. TCP and UDP each seperately differentiate connections based on the remaining 4 items. It's possible to create other protocols on top of IP that use completely different (perhaps port-less) differentiation mechanisms.
And then there are different socket "domains", such as the "unix" socket domain, which is completely distinct from "inet" and uses the filesystem for addressing.
The destination isn't identified by IP Addr:Port alone. There is another thing - IP header has a field called Protocol which differentiates the TCP and UDP endpoint. As such it becomes possible for two process to bind to same IP:Port as long as communication protocol is different.
The endpoint of a connection is for UDP and TCP defined by IP, protocol (TCP or UDP) and port. This means as long as you use a different protocol the endpoint of the communication is different too.
Because they are not the only component of the means of address. It's the same as why you can have two houses with the same number on different streets, or why you know John Whorfin is not the same Red Lectroid as John Bigbooté.
Each IP packet contains a field that says which transport-layer protocol is to be used, and within the domain of that protocol is a set of ports that can be the same as in any other protocol because they are actually a completely separate set.
As for the second question, there are answers elsewhere.
I had a strange idea. I heard of software which from my understanding uses UDP to transfer files decreasing the overhead found in TCP packets.
If my app requires TCP and my LAN has software set up to communicate with another datacenter on the other side of the coast with software setup on their end. Would it be possible to send the actual data via UDP but than simulating TCP on both ends?
Does anyone have any ideas or information about such projects?
If you're asking if you can use UDP as a Layer 2, then the answer is yes, sort of. There are various protocols that allow you to create a tunnel to another network using a UDP transport, such as L2TP and even IPsec (with NAT traversal). You could also do it at the application layer.
If you're asking if TCP can be implemented in UDP, the answer is no. First, TCP packets and UDP packets have an incompatible format. Second, TCP and UDP have different protocol numbers (seen in the IP header) which means that TCP traffic destined for a UDP port would not be passed to the correct upper-layer protocol.
Both TCP and UDP are built on top of the IP, but the TCP uses different packet structure and at the layer-2 it is not possible to mimic the TCP using UDP packets.
Of course, if you have the control on both the source and destination, then it is possible to create a reliable UDP tunnel for the TCP packets. This would require some internal information (packet number, ack/nack flags) in the body of the UDP packet.
There is an interesting project http://udt.sourceforge.net/
It is a broadcast-capable reliable file transfer mechanism built on top the UDP.
PseudoTCP is a protocol which implements TCP algorithms on top of the UDP. It was introduced since the NAT traversal for TCP is much more complicated than UDP. But some P2P applications do need a reliable data transfer among nodes.
So far as I know, there are two PseudoTCP variations: Libjingle and Libnice.Libjingle is an open source library from google which was initially for gtalk. You could take a look at file sharing example from libjingle: https://developers.google.com/talk/libjingle/file_share. Recently, Chrome desktop also use PseudoTCP implementation from libjingle for reliable connections.
Yes, you can develop a protocol on UDP that simulates TCP. However, if you simulated TCP fully, it would technically have more overhead. Because TCP is implement as the packet and your simulated TCP is implemented in the body of the packet.
If you only need one or two features of TCP (such as basic ordering), then implementing it in UDP is useful.
Halo uses 2-3 (IIRC) UDP protocols that simulate different features of TCP, then full fledged TCP for initializing game-states. I Shot You First Networking, GDC publication
For example, in one case, they send 3 duplicate UDP packets to overcome packet loss.
If you control the software on both ends, and it is cost-effective to build your own protocol, then UDP can be versatile.
One way to do it now on Linux-3.18+ is to use Foo over UDP (FOU) which implements Generic UDP Encapsulation (GUE). Here's a good introduction to FOU, and the man page for ip-fou.
Or if you want an [open source] UDP based file transfer system there are things like UDT, UFTP, Tsunami-UDP, and even Google's QUIC (Now deprecated in favour of IETF QUIC).
Update: The QUIC protocol now has been standardised by the IETF which provides for secure reliable and unreliable transport over UDP as an alternative to TCP. There's a wide range of QUIC implementations available. There is also a growing set of protocol mappings on to QUIC such as HTTP/3, DNS over QUIC, etc
If my app requires TCP and my LAN has software setup to communicate
with another datacenter on the other side of the coast with software
setup on their end. Would it be possible to send the actual data via
UDP but than simulating TCP on both ends?
No. A UDP socket is in a different namespace from a TCP socket. You will be unable to write UDP at one end and send or receive TCP at the other end. TCP and UDP are peer protocols; both exist at the layer above IP. You can't use one to spoof the other.
Hmm, I believe so. You'd need to use a proxy at both ends, but it should be possible.
The biggest problem you are going to run into is that UDP is designed with the idea that you don't care if some of the packets don't ever make it to the other end.
Here's a link with some more info:
http://www.cyberciti.biz/faq/key-differences-between-tcp-and-udp-protocols/
IMHO, it's not a good idea to transmit files via UDP.
TCP's problems are in its algorithms, not its headers.
You certainly could implement the TCP algorithms on top of UDP. That would effectively be the same as tunneling TCP datagrams inside of UDP datagrams. But all this accomplishes is to add a few more bytes of overhead to each packet, and require another endpoint to unwrap the packets.
UDP itself is just thin shim on top of IP: its a convenient way to access IP packet switched networking without having to dive into kernels or receive special handling from routers. The main reason to implement reliable transport on top of UDP is to get away from TCP algorithms in favor of something more efficient. FileCatalyst was mentioned above as one company which does this, and my own company Data Expedition, Inc. does so as well.
So you could implement TCP algorithms on top of UDP, but you wouldn't want to.
You can simulate something like a connection over UDP, and you as well can add reliability checks and ordering and retransmission and so on. - but then, it still isn't TCP, it just acts the way.
Of course, one of the ends can be a kind of "hub" or "proxy" which does an adaption. Then you don't have a 2-end solution, but in fact a 4 end solution - one pair with "real" TCP and the other with the "self-knitted" "TCP" - which you put together with an appropriately crafted program.