We're prototyping a jgroups-based cluster node messaging system that will replace one that was JDBC-based. There are a lot of folks in my organization who are concerned about adding more multicast traffic to an already busy network, so I'm getting some pushback on a UDP/multicast solution.
I know JGroups can be configured to be TCP only, but I do not want to have to force a configuration step into the application where each node has to be identified ahead of time in a config file.
What I'd like then is to see if we can get a hybrid working here where multicast is used ONLY for group membership operations (discovery, heartbeats, failure detection), but messaging is all TCP-based.
I'm not finding examples of that in my searches, however, and am therefore questioning whether JGroups can be configured this way.
Can it, and any example configs showing how?
Thanks!
As for discovery, using MPING you can do that - it uses IP multicast to discover new nodes, although these later respond via the main transport (TCP in your case).
As for FD/FD_ALL, I don't think it's possible, the protocols are designed to use the main transport. You'd have to write your own FD protocol, shouldn't be that complicated.
However, if you can use UDP, you probably should. It's up to you whether you send a message to one node, multiple ones or all of them - for one destination there will be unicast UDP, for few destinations (if you set the anycast option) will be multiple unicasts used and only for all nodes the UDP will reduce the network load by multicasting. It's really up to the application, UDP just allows multicasts.
Related
Suppose I have a producer located on one machine and two independent consumers on the other. Consumers share the same IP. I want my data to be sent via WiFi and I want to avoid sending the same packages twice: I mean I want the packages to be replicated as late as possible -- either on router or on destination machine.
I managed to find these options:
1) Just use different ports -- in this case everything will be sent twice, but at least all consumers will get their copy of data.
2) Try to use SO_REUSEPORT -- without multicast this will make only one of two consumers get the packages.
3) Use multicast (with SO_REUSEADDR) -- I guess it is the only solution for my problem, and the packages will be replicated by router, not publisher.
I know, that there are messaging libraries and frameworks, which are supposed to be responsible for marshaling and routing, but I need "pure" UDP solution with maybe some simple logic upon it. I am not sure if I am correct with options that I mentioned, and I don't know whether there exists any more simple and suitable solution.
As people told you in the comments, one popular and efficient solution to you problem (one publisher multiple subscribers over network, aka pubsub) is to send UDP datagrams to a multicast IP address. The listeners must join that IP multicast group in order to tell the intermediate routers to forward those datagrams.
You can code it up yourself, there are plenty of examples elsewhere.
In a production setting, however, I recommend against reinventing the wheel along with associated mistakes. Have a look at existing solutions, such as ZeroMQ. It supports this messaging pattern of one publisher to multiple subscribers using UDP/IP multicast but with a nice interface, cross-language compatibility and much more. See Publish/Subscribe example in Python.
As far as I understand, ICE protocol is used for discovering the nodes/devices from the end-user device to "the outside".
I don't understand why it's needed. Isn't packet-routing is the responsibility of network devices like routers and switches? They should find the shortest path from the gateway to the end-user device (Actually, routers remembers those routes they previously discovered).
Moreover, NAT protocol is used to convert from an "internal ip" to "external ip" and vice-versa.
So again,
Why does the other user needs to be familiar with my internal network setup?
NAT is a kludge, put in place to try to conserve IPv4 addresses until IPv6 becomes ubiquitous, and it breaks the end-to-end connectivity which is the promise of IP. Because of that, some things don't work correctly through NAT. There are various kludges to work around the NAT kludge, and ICE is part of that. This is explained in RFC 5245, Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols:
Introduction
RFC 3264 [RFC3264] defines a two-phase exchange of Session Description
Protocol (SDP) messages [RFC4566] for the purposes of establishment of
multimedia sessions. This offer/answer mechanism is used by protocols
such as the Session Initiation Protocol (SIP) [RFC3261].
Protocols using offer/answer are difficult to operate through Network
Address Translators (NATs). Because their purpose is to establish a
flow of media packets, they tend to carry the IP addresses and ports
of media sources and sinks within their messages, which is known to be
problematic through NAT [RFC3235]. The protocols also seek to create
a media flow directly between participants, so that there is no
application layer intermediary between them. This is done to reduce
media latency, decrease packet loss, and reduce the operational costs
of deploying the application. However, this is difficult to
accomplish through NAT. A full treatment of the reasons for this is
beyond the scope of this specification.
Numerous solutions have been defined for allowing these protocols to
operate through NAT. These include Application Layer Gateways (ALGs),
the Middlebox Control Protocol [RFC3303], the original Simple
Traversal of UDP Through NAT (STUN) [RFC3489] specification, and Realm
Specific IP [RFC3102] [RFC3103] along with session description
extensions needed to make them work, such as the Session Description
Protocol (SDP) [RFC4566] attribute for the Real Time Control Protocol
(RTCP) [RFC3605]. Unfortunately, these techniques all have pros and
cons which, make each one optimal in some network topologies, but a
poor choice in others. The result is that administrators and
implementors are making assumptions about the topologies of the
networks in which their solutions will be deployed. This introduces
complexity and brittleness into the system. What is needed is a
single solution that is flexible enough to work well in all
situations.
This specification defines Interactive Connectivity Establishment
(ICE) as a technique for NAT traversal for UDP-based media streams
(though ICE can be extended to handle other transport protocols, such
as TCP [ICE-TCP]) established by the offer/answer model. ICE is an
extension to the offer/answer model, and works by including a
multiplicity of IP addresses and ports in SDP offers and answers,
which are then tested for connectivity by peer-to-peer connectivity
checks. The IP addresses and ports included in the SDP and the
connectivity checks are performed using the revised STUN specification
[RFC5389], now renamed to Session Traversal Utilities for NAT. The
new name and new specification reflect its new role as a tool that is
used with other NAT traversal techniques (namely ICE) rather than a
standalone NAT traversal solution, as the original STUN specification
was. ICE also makes use of Traversal Using Relays around NAT (TURN)
[RFC5766], an extension to STUN. Because ICE exchanges a multiplicity
of IP addresses and ports for each media stream, it also allows for
address selection for multihomed and dual- stack hosts, and for this
reason it deprecates RFC 4091 [RFC4091] and [RFC4092].
Firewalls. They're typically configured to bounce any unsolicited traffic from the world wide web to you. They only approve of you initiating contact with a server, which only then is allowed to back-traffic to you, and that's pretty much it. Unless your friends all own static IPs (which few people can justify) this is a hostile environment for peer to peer communication.
ICE tries to solve this, by enumerating addresses and ports at which the other side may be reached, and trying to connect to these addresses, by initiating outbound requests on both ends, or if all else fails, falling back to communicating through a TURN server, if specified.
See this WebRTCHacks article for more on the problem.
Why does the other user needs to be familiar with my internal network setup?
Because the other user is sometimes on your internal network. e.g. LAN games.
I am trying to write a program that scan an ip range and detect if an ip is address of a router or not.
Currently i used traceroute from my computer to all host in the network. However, i believe there must be some way to directly "ask" a host at an ip if it is a router or not?
by the way, do you know any program/ opensource already does this?
Routers are supposed to talk couple of protocols (actually a neat bunch) that regular IP nodes do not, and then there are some which are more common (i.e. even non-router nodes do).
Router-only protocols:
VRRP
IGRP / EIGRP
OSPF
BGP
RIP
You could do active-probing on those, i.e. send a packet (behaving as if you are another router, or an end-node) and checking to see what kind of response the router (if at all) sends.
Alternatively you could do passive-probing, like 'sniffing', i.e. watching out for the kind of IP packets being sent out by various nodes. There are some which are usually sent out by Routers only (again, mostly from the above list).
Common protocol, but that can actually tell you a lot:
SNMP (esply the unsecure one's like v1/v2, are easy to deal with, without having to establish a secure session)
Other ways:
Portscanning (actually can tell you a real lot), for example all routers have some management ports (although, often they are locked down due to security concerns)
What you want to do is often what many 'Network Management' software do, to "discover" capabilities / functionality of other nodes in the network. And, there isn't a single size-fits all solution. They use bunch of different methods, heuristics to finally figure out what the other node is.
Any node which is hopped to and not just an endpoint is a router. However, this doesn't allow you to detect routers with no reachable devices hooked up. (Any input as to whether my answer has merit would be great!)
I've seen and read a lot of similar questions, and the corresponding Wikipedia articles (NAT traversal, STUN, TURN, TCP hole punching), but the overwhelming amount of information doesn't really help me with my very simple problem:
I'm writing a P2P application, and I want two users of my application behind NAT to be able to connect to each other. The connection must be reliable (comparable to TCP's reliability) so I can't just switch to UDP. The solution should work on today's common systems without reconfiguration. If it helps, the solution may involve a connectible 3rd-party, as long as it doesn't have to proxy the entire data (for example, to get the peers' external (WAN) IP addresses).
As far as I know, my only option is to use a "reliable UDP" library + UDP hole punching. Is there a (C/C++) library for this? I found enet in a related question, but it only takes care of the first half of the solution.
Anything else? Things I've looked at:
Teredo tunnelling - requires support from the operating system and/or user configuration
UPnP port forwarding - UPnP isn't present/enabled everywhere
TCP hole punching seems to be experimental and only work in certain circumstances
SCTP is even less supported than IPv6. SCTP over UDP is just fancy reliable UDP (see above)
RUDP - nearly no mainstream support
From what I could understand of STUN, STUNT, TURN and ICE, none of them would help me here.
ICE collects a list of candidate IP/port targets to which to connect. Each peer collects these, and then each runs a connectivity check on each of the candidates in order, until either a check passes or a check fails.
When Alice tries to connect to Bob, she somehow gets a list of possible ways - determined by Bob - she may connect to Bob. ICE calls these candidates. Bob might say, for example: "my local socket's 192.168.1.1:1024/udp, my external NAT binding (found through STUN) is 196.25.1.1:4454/udp, and you can invoke a media relay (a middlebox) at 1.2.3.4:6675/udp". Bob puts that in an SDP packet (a description of these various candidates), and sends that to Alice in some way. (In SIP, the original use case for ICE, the SDP's carried in a SIP INVITE/200/ACK exchange, setting up a SIP session.)
ICE is pluggable, and you can configure the precise nature/number of candidates. You could try a direct link, followed by asking a STUN server for a binding (this punches a hole in your NAT, and tells you the external IP/port of that hole, which you put into your session description), and falling back on asking a TURN server to relay your data.
One downside to ICE is that your peers exchange SDP descriptions, which you may or may not like. Another is that TCP support's still in draft form, which may or may not be a problem for you. [UPDATE: ICE is now officially RFC 6544.]
Games often use UDP, because old data is useless. (This is why RTP usually runs over UDP.) Some P2P applications often use middleboxes or networks of middleboxes.
IRC uses a network of middleboxes: IRC servers form networks, and clients connect to a near server. Messages from one client to another may travel through the network of servers.
Failing all that, you could take a look at BitTorrent's architecture and see how they handle the NAT problem. As CodeShadow points out in the comments below, BitTorrent relies on reachable peers in the network: in a sense some peers form a network of middleboxes. If those middleboxes could act as relays, you'd have an IRC-like architecture, but one that's set up dynamically.
I recommend libjingle as it is used by some major video game companies which heavily relies on P2P network communication. (Have you heard about Steam? Vavle also uses libjingle , see the "Peer-to-peer networking" session in the page: https://partner.steamgames.com/documentation/api)
However, the always-work-solution would be using a relay server. Since there is no "standard" way to go through NAT, you should have this relay server option as a fall-back strategy if a connection has to be always established between any peers.
I am considering using SCTP instead of TCP for a p2p app written in C. Should I do it? Also how does the speed of SCTP compare to the speed of TCP?
EDIT:
I found that SCTP can be tunneled over UDP with the only problem being tunneled SCTP is not interoperable with untunneled SCTP.
Have you considered whether your target systems will all have SCTP pre-installed on them or whether your application will need to include SCTP itself? In my experience I would not expect all systems to have SCTP installed on them, and I would expect them not to if it were Windows.
If you include SCTP in the application itself then that will more than double the number of messages being passed into an out of the Kernel which will impact performance when compared with using the pre installed TCP.
Have you considered what benefits you want from SCTP? You mentioned fault tolerance but for this to work with SCTP it requires the application to have multiple ethernet ports and and IP addresses. Is this likely on your app?
As much as I love SCTP (!) I would seriously consider sticking with TCP unless you are sure SCTP is needed or unless you control the hosts your app is deployed on.
Regards
If it's for a local area network, sure go for it.
Note however that if you plan to use it on the open internet many consumer grade firewalls aren't flexible enough to permit unrecognised IP protocols through them.
How does it help you?
You're P2P, so every peer must have at least one socket open to every other peer.
If you've got a socket open, then you can do everything you need to do over that. If you've taken the approach of one socket per file and you have multiple files being tranferred concurrently between two given peers, then SCTP will save you one socket per file. However, on a normal P2P network of any size, you will almost never have multiple files being transferred concurrently between two peers.
Just have one socket and have your own little protocol; send a packet with a header, the header indicates content type, e.g. a command, or part a file - and if so, which file, and which byte range.
Of course, you get a little overhead for that, whereas if you have one socket for commands and one per file, you're more efficient. Is saving one socket per peer (assuming one download at a time) worth the time/hassle/complexity of using SCTP?