As per the docs, the NATS server design is a "server first" approach, regarding protecting against "lazy clients". Lazy clients are just booted under poor performance scenarios.
Due to this, I have internalized an assumption that anything connecting from past the edge should not connect directly to the NATS server, but should instead access some middle layer service point that manages and maps the more poorly performing external connections through internally initiated connections to the NATS server.
For example, consider a client service node on a remote customer facility that accesses back end services that are serviced over NATS in some form.
Is my stated assumption true in a VERY STRICT sense, such that it is NEVER advisable for that remote node to connect to a NATS server directly, even given a low number of possible client connections AND a stable network service to those facilities?
Or, is it ok-ish to connect directly from that remote node IF, and ONLY IF, I have a known solid infrastructure (low latency, high bandwidth, dependable, etc).
Finally, how about if the path to that remote node is not a very solid network service? Specifically, is it better to A) as I described, use an intermediary service on the back-end for managing the end point connection requests and passing them along over internal connections, or B) is it preferred to just let it connect directly to the server and let NATS boot it as needed, then use a re-connect on drops approach to keep the connection up as best as it can?
A good example here is a mobile endpoint, which could be up and down regularly for various reasons and due to no fault of the device or infrastructure at all.
Currently, I am designing every NATS solution as a "backend connections only" design. If this is overcomplicating my designs needlessly, of course I want to stop forcing that design constraint. :)
Related
While building an application by python for implementing post-processing in Quantum Key Distribution, I require a server (say Alice) and a client(say Bob) at distant locations to interact and exchange some information while doing the required calculations simultaneously (for which threading is used).
At first, the open-source NGROK service was used as a hosting server but it makes the time required for post-processing very huge mainly due to network congestion.
So, is there a way to establish a direct peer-to-peer connection between Alice and Bob via the internet wherein Alice's system itself acts as a server thereby bypassing a third-party host server? If there is a way out please suggest one otherwise any leads or suggestions for open source services better than NGROK would be highly helpful.
PS: If you need any additional information for providing help, i would be eager to respond.
I am struggling to describe simply what the IP Application layer actually “is”. Some of my terminology may be “not quite right”, so I am happy to be corrected on my terminology as well. Is the application layer a “thing”? Or is it more “conceptual” than that? Is the Application layer on a particular node solely that collection of protocols (such as FTP, POP3, HTTP) that an application needs to adhere to if it is to successfully communicate with another remote application?
Conversely, if I write a program that does not strictly conform to an application protocol (say FTP) then there is nothing in the local “application layer” that will “stop” my attempt to communicate to the remote end using an invalid command? Whereas, if the remote end does conform to the application protocol, then it will reject or ignore my invalid communication attempt?
Or again, I could define my own protocol (not necessary as I guess all scenarios are already covered), and as long as my protocol was “robust and consistent” I could use it to communicate between two remote applications and everything would work? Or, is the Application layer a “thing” separate from the protocols?
The application layer is everything that's inside the application - that is, not within the IP stack (which covers network and transport layers).
If you want your application to communicate with an FTP server it needs to be able to speak FTP in order to work. If you write your own client-server protocol you can do whatever you want.
The network layers are there to make your life easier and to make applications compatible with each other. You can use or not use whatever you seem fit.
“An application needs to communicate with another application on a remote system. Any communication can be thought of as the exchange of messages and data between two partners. There are rules (protocols) that define for an application: what commands are available, the sequence that they can be exchanged between partners and any limitations on the data exchanged. For example: communication with an HTTP server uses commands: GET, POST and PUT (and others), whereas communication with an SMTP (mail) server uses: MAIL, RCPT and DATA, and similarly for other protocols. There are different protocols depending on the nature of the communication e.g. text message transfer, file transfer or video streaming. Any commands that are sent by either partner that do not obey those protocol rules is either: not delivered to, ignored or rejected by the remote application. Nothing prohibits application partners using their “own” communication protocol – for example Skype developed an uses its own protocol. However, there are standard protocols that handle most conceivable communication needs, so developing a new protocol may be unnecessary. If an application uses a “private” protocol its communication will be limited to remote applications that support the same private protocol. In addition to protocols, in a practical, real-world, system there are programming libraries (Programming interfaces) that enable the end-user application to request and participate in this communication. It is the end-user application’s responsibility to interact (send and receive data and parameters) with the (local programming) interface and the comms system’s responsibility to process those requests and deliver the data. In simplistic terms, one data transmission starts at one application and ends at another. However, IP provides other applications, such as DNS, that are used to set up connections, but the operation of these is not visible at the end-user application level. In summary, the collection of protocols and the programming interfaces available and used by an end-user application is the application layer on a system.”
I have two applications that need to communicate with each other running on the same system.
I've been using the very strange practice of opening a TCP COM channel between the two applications for communication.
Is this practice frowned-upon in anyway? Is there any alternative (apart from using stdio, not possible due to other reasons).
Is there a restriction on maximum transfer rate and/or any latency involved (compared to piped stdio)?
I'm using the local (127.0.0.1) address for both server and client, will the connection be guaranteed to stay within the local machine itself or could it relay off the nearest router before coming back to itself and does the network card influence the properties of the connection at all?
I worked on a system a while ago with Java. and I was looking for the same question. I don't have much experience with it. But I ended up using tcp connections for communications for the following advantages:
1) The ability to put the different application in different servers in the future if needed to.
2) The applications are totaly independent. one application could crash without effecting the other one. If the working application then tries to connect it gets an error and you can handle that.
I saw this used in many other type of applications. So I went with it and it is working fine. But you have to be carefull and handle networks errors and IO errors and closing all open sockets after finishing with the connection. I was only closing the socket from the client end so I ended up with many CLOSE_WAIT ports in the server.
Regards,
It's pretty common to use TCP for inter-application communication.
Performance should not be an issue.
Sockets On Same Machine For Windows and Linux
You should consider security. What will happen if another user on the machine connects to the port, how will the application authenticate etc.
I'm developing a multi-player game and I know nothing about how to connect from one client to another via a server. Where do I start? Are there any whizzy open source projects which provide the communication framework into which I can drop my message data or do I have to write a load of complicated multi-threaded sockety code? Does the picture change at all if teh clients are running on phones?
I am language agnostic, although ideally I would have a Flash or Qt front end and a Java server, but that may be being a bit greedy.
I have spent a few hours googling, but the whole topic is new to me and I'm a bit lost. I'd appreciate help of any kind - including how to tag this question.
If latency isn't a huge issue, you could just implement a few web services to do message passing. This would not be a slow as you might think, and is easy to implement across languages. The downside is the client has to poll the server to get updates. so you could be looking at a few hundred ms to get from one client to another.
You can also use the built in flex messaging interface. There are provisions there to allow client to client interactions.
Typically game engines send UDP packets because of latency. The fact is that TCP is just not fast enough and reliability is less of a concern than speed is.
Web services would compound the latency issues inherent in TCP due to additional overhead. Further, they would eat up memory depending on number of expected players. Finally, they have a large amount of payload overhead that you just don't need (xml anyone?).
There are several ways to go about this. One way is centralized messaging (client/server). This means that you would have a java server listening for udp packets from the clients. It would then rebroadcast them to any of the relevant users.
A second way is decentralized (peer to peer). A client registers with the server to state what game / world it's in. From that it gets a list of other clients in that world. The server maintains that list and notifies the other clients of people who join / drop out.
From that point forward clients broadcast udp packets directly to the other users.
If you look for communication framework with high performance try look at ACE C++ framework (it has Java bindings).
Official web-site is: http://www.cs.wustl.edu/~schmidt/ACE-overview.html
You could also look into Flash Media Interactive Server, or if you want a Java implementation, Wowsa or Red5. Those use AMF and provide native functionality for ShareObjects including synching of the ShareObjects among connected clients.
Those aren't peer to peer though (yet, it's coming soon I hear). They use centralized messaging managed by the server.
Good luck
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
If I have a separate system with its own concept of users and presence, what is the most appropriate architecture for creating a bridge to an XMPP server network? As far as I can tell there are three primary ways:
Act as a server. This creates one touchpoint, but I fear it has implications for compatibility, and potentially creates complexity in my system for emulating a server.
Act as a clients. This seems to imply that I need one connection per user in my system, which just isn't going to scale well.
I've heard of an XMPP gateway protocol, but it's unclear if this is any better than the client solution. I also can't tell if this is standard or not.
Any suggestions or tradeoffs would be appreciated. For example, would any of these solutions require running code inside the target XMPP server (not likely something I can do).
The XMPP gateway protocol you've heard of is most likely to do with transports. A transport is a server that connects to both a XMPP server and a non-XMPP server. By running a transport, I can use my Jabber client to talk to someone using, say, MSN Messenger.
A transport typically connects once to the remote network for each JID that it sees as online. That is, it's your option 2 in reverse. This is because there is no special relationship between the transport and the non-XMPP network; the transport is simply acting as a bunch of regular clients. For this to work, XMPP clients must first register with the transport, giving login credentials for the remote network, and allowing the transport to view their presence.
The only reason this has a chance of scaling better is that there can be many transports for the same remote network. For example, my Jabber server could run a transport to MSN, another Jabber server could run another one, and so on, each one providing connections for a different subset of XMPP users. While this spreads out the load on the Jabber side, and load balancing on your system may spread out the load as well, it still requires many connections between the two systems.
In your case, because (I assume) the non-XMPP side of things is cooperating, putting a XMPP server interface on the non-XMPP server is likely your best bet. That server interface is best suited for managing the mapping between XMPP JIDs and how that JID will appear on its own network, rather than forcing XMPP users to register and so on.
In case you haven't seen these, you might find them useful:
http://www.jabber.org/jabber-for-geeks/technology-overview
http://www.xmpp.org/protocols/
http://www.xmpp.org/extensions/
Hope that helps.
I too am working on a similar system.
I am going with the gateway/component route. I have looked at several options and settled with this one.
The gateway is basically a component with the specific purpose of bridging Jabber/XMPP with another network. You will have to build most of the things you take for granted when using XMPP as a client. Stuff like roster control.
There is very little help online on the actual design and building of a component. Like the above answer I found that the xmpp protocols/extensions to be of help. The main ones being:
Basic Client 2008
Basic Server 2008
Intermediate Client 2008
Intermediate Server 2008
Reading through these will show you what XEPs you will be expected to be able to handle. Ignore the stuff that will be handled by the server that your component will be attched to.
It's a shame that Djabberd has such poor documentation as their system of "everything is a module" gave the possibility of backend of the server could interface directly to the other network. I made no headway on this.
There are basically two types of server to server (s2s) connections. The first is either called a gateway or a transport, but they're the same thing. This is probably the kind you're looking for. I couldn't find specific documentation for the non-XMPP side, but how XMPP thinks about doing translations to legacy servers is at http://xmpp.org/extensions/xep-0100.html. The second kind really isn't explained in any additional XEPs -- it's regular XMPP s2s connections. Look for "Server-to-Server Communication" in RFC 3920 or RFC 3920bis for the latest draft update.
Since you have your own users and presence on your server, and it's not XMPP, the concepts aren't going to map completely to the XMPP model. This is where the work of the transport comes in. You have to do the translation from your model to the XMPP model. While this is some work, you do get to make all the decisions.
Which brings us right to one of the key design choices -- you need to really decide which things you are going to map to XMPP from your service and what you aren't. These feature and use case descriptions will drive the overall structure. For example, is this like a transport to talk to AOL or MSN chat services? Then you'll need a way to map their equivalent of rosters, presence, and keep session information along with logins and passwords from your local users to the remote server. This is because your transport will need to pretend to be those users and will need to login for them.
Or, maybe you're just an s2s bridge to someone else's XMPP based chess game, so you don't need a login on the remote server, and can just act similarly to an email server and pass the information back and forth. (With normal s2s connections the only session that would be stored would be SASL authentication used with the remote server, but at the user level s2s just maintains the connection, and not the login session.)
Other factors are scalability and modularity on your end. You nailed some of the scalability concerns. Take a look at putting in multiple transports to balance the load. For modularity, see where you want to make decisions about what to do with each packet or action. For example, how do you handle and keep track of subscription data? You can put it on your transport, but then that makes using multiple transports harder. Or if you make that decision closer to your core server you can have simpler transports and use some common code if you need to talk to services other than XMPP. The trade off is a more complex core server with more vulnerability potential.
What architecture you should use depends on the non-XMPP system.
Do you operate the non-XMPP system? If yes, you should find a way to add an XMPP-S2S interface to that system, in other words, make it act as an XMPP server. AOL is using this approach for AIM. Unfortunately, they have restricted their gateway to GoogleTalk.
You don't operate the non-XMPP system but it has a federation interface that you can use - i. e. your gateway can talk to the other system as a server and has a namespace of its own. In this case, you can build a gateway that acts as a federated server on both sides. For I don't know of any example of a gateway that uses this approach but you could use it if you want to build a public XMPP-to-SIP bridge.
If the non-XMPP system doesn't give you a federation interface, then you have no other option but acting as a bunch of clients. In the XMPP world, this is called a "transport". The differences between a transport and a normal server are basically:
the JIDs of the transport are mapped from another system (e.g. john.doe\40example.net#msngateway.example.org - really ugly!)
XMPP users who want to use the transport need to create an account on the non-XMPP system and give the login credentials of that account to the transport service. The XMPP protocol even has a protocol extension that allows XMPP users to do transport registrations in-band.
One other approach is to work with your XMPP server vendor. Most have internal APIs that make injecting presence possible from third party applications. For example, Jabber XCP provides an API for this that's really easy to use.
(Disclosure: I work for Jabber, Inc, the company behind Jabber XCP)