I want to create a program for my desktop and an app for my android. Both of them will do the same, just on those different devices. They will be something like personal assistants, so I want to put a lot of data into them ( for example contacts, notes and a huge lot of other stuff). All of this data should be saved on a server (at least for the beginning I will use my own Ubuntu server at home).
For the android app I will obviously use java and the database on the server will be a MySQL database, because that's the database I have used for everything. The Windows program will most likely be written in of these languages: Java, C#c C++, as these are the languages I am able to use quite well.
Now to the problem/question: The server should have a good backend which will be communicating with the apps/programs and read/write data in the database, manage the users and all that stuff. But I am not sure how I should approach programming the backend and the "network communication" itself. I would really like to have some relatively easy way to send secured messages between server and clients, but I have no experience in that matter. I do have programming experience in general, but not with backend and network programming.
side notes:
I would like to "scale big". At first this system will only be used by me, but it may be opened to more people or even sold.
Also I would really like to a (partly) self programmed backend on the server, because I could very well use this for a lot of other stuff, like some automation features in my house, which will be implemented.
EDIT: I would like to be able to scale big. I don't need support for hundreds of people at the beginning ;)
You need to research Socket programming. They provide relatively easy, secured network communication. Essentially, you will create some sort of connection or socket listener on your server. The clients will create Sockets, initialize them to connect to a certain IP address and port number, and then connect. Once the server receives these connections, the server creates a Socket for that specific connection, and the two sockets can communicate back and forth.
If you want your server to be able to handle multiple clients, I suggest creating a new Thread every time the server receives a connection, and that Thread will be dedicated to that specific client connection. Having a multi-threaded server where each client has its own dedicated Thread is a good starting point for an efficient server.
Here are some good C# examples of Socket clients and servers: https://msdn.microsoft.com/en-us/library/w89fhyex(v=vs.110).aspx
As a side note, you can also write Android apps in C# with Xamarin. If you did your desktop program and Android app both in C#, you'd be able to write most of the code once and share it between the two apps easily.
I suggest you start learning socket programming by creating very simple client and server applications in order to grasp how they will be communicating in your larger project. Once you can grasp the communication procedures well enough, start designing your larger project.
But I am not sure how I should approach programming the backend and
the "network communication" itself.
Traditionally, a server for your case would be a web server exposing REST API (JSON). All clients need to do http requests and render/parse JSON. REST API is mapped to database calls and exposes some data model. If it was in Java, it would be Jetty web server, Jackson Json parser.
I would really like to have some relatively easy way to send secured
messages between server and clients,
Sending HTTP requests probably the easiest way to communicate with a service. Having it secured is a matter of enabling HTTPS on the server side and implementing some user access authentication and action authorization. Enabling HTTPS with Jetty for Java will require few lines of code. Authentication is usually done via OAuth2 technique, and authorization could be based on ACL. You may go beyond of this and enable encryption of data at rest and employ other practices.
I would like to "scale big". At first this system will only be used by
me, but it may be opened to more people or even sold.
I would like to be able to scale big. I don't need support for
hundreds of people at the beginning
I anticipate scalability can become the main challenge. Depending on how far you want to scale, you may need to go to distributed (Big Data) databases and distributed serving and messaging layers.
Also I would really like to a (partly) self programmed backend on the
server, because I could very well use this for a lot of other stuff,
like some automation features in my house, which will be implemented.
I am not sure what you mean self-programmed. Usually a backend encapsulates some application specific business logic.
It could be a piece of logic between your database and http transport layer.
In more complicated scenario your logic can be put into asynchronous service behind the backend, so the service can do it's job without blocking clients' requests.
And in the most (probably) complicated scenario your backend may do machine learning (for example, if you would like you software stack to learn your home-being habits and automate house accordingly to your expectations without actually coding this automation)
but I have no experience in that matter. I do have programming
experience in general, but not with backend and network programming.
If you can code, writing a backend is not very hard problem. There are a lot of resources. However, you would need time (or money) to learn and to do it, what may distract you from the development of your applications or you may enjoy it.
The alternative to in-house developed of a backend could be a Backend-as-a-Service (BaaS) in cloud or on premises. There are number of product in this market. BaaS will allow you to eliminate the development of the backend entirely (or close to this). At minimum it should do:
REST API to data storage with configurable data model,
security,
scalability,
custom business-logic
Disclaimer: I am a member of webintrinsics.io team, which is a Backend-as-a-Service. Check our website and contact if you need to, we will be able to work with you and help you either with BaaS or with guiding you towards some useful resources.
Good luck with your work!
Related
(Sorry for long looking post, It is actually really short in context)
Hello, I am new in network programming and concepts. However I have worked with PUN before, several times. I am familiar with PUN way of synchronizing things. RPC and Serialization etc.
I want to achieve a full-authoritative and future scalable server architecture that works in dedicated servers and manages room/lobby services for clients. The server architecture that I am planing is, similar to the games like Rust/ARK/Hurtworld etc. However the game will consist 7v7 matches, (not 100ish like them, 15 max), but will also have mechanics that alters the world. (like building something to world or altering terrain by mining etc). Gameplay would resemble Rust in visuals, but the mechanics would differ. However server architecture should be very similar to those games. Meaning it won't be Peer to Peer.
To start: "I know" that I have to use Photon Server for these type of things. So I can code server side logic right? The authoritative architecture, the persistent world, user&world databases, server management all these should be done in the server side if I am not mistaken. However as i said in the beginning I have only worked with PUN before, and only client-server architecture I know is that: one trusted client (user) hosts the game and others joins to him.
Now. I can learn Photon Server and server side programming. (hell, even started learning it now. It is kinda similar how PUN does it's work. Operations and Events instead of RPC's etc.) However, to create a sustained server architecture, I have to learn and practice it in depth. Meaning: I need profound knowledge to create a commercial product server.
So for the initial prototyping phase of the game, I don't want to focus my workload to learning and creating the Photon Server. Instead I have come up with an idea.
What if I create Unity instances in dedicated servers, that creates(hosts) rooms with PUN (cloud). And other players around the world join to these hosted matches? Dedicated servers would simply be "non-player" master clients that hosts the games and let other players join to games. And also does server stuff by itself.
So, I would create a standalone "server-only" Unity project file, that can manage servers, manage databases, create and sustain persistent worlds. All the features that won't be in user clients, can be in the server(host/master) client's project. And I can manage all these in the dedicated server inself.
So to clarify: A "special" unity project that runs the game simulation, manage world, manage database, manage players, manage everything and run everything, will be the master client. Will host the game. And "client" unity projects will join to this game, and will "send" inputs for authoritative architecture. Server will "accept" these inputs, simulate it and send it back to clients.
To more clarify: Everything that Photon Server should be handling will be handled by PUN and Cloud relay servers, RPC's and Serialize funtions. So I wont hustle my workload with the server programming. Win-Win. Right?
I know, It is probably not the best solution out there. I haven't found any single document/topic about it. So It may be a really stupid idea to begin with. But I wonder. If I start with this type of architecture, when I get pass the prototype phase, would it be easy to implement Photon Server to server side? Would it be easy and work efficient to follow this approach? Or should I stop whatever I am doing, and start learning Photon server?
if you need to prototype you may do it in whatever way is convinient for you. but if you start to develop real product this is not a solution at all.
in order to make live of developers easier we introduced plugins. You may try to start with them. if at some point you see that this is not enough for you, you may extend PhotonServer.
And last thing, for sure we (PhotonTeam) and community will help you either here, or on our forum.
Best,
Photonians
I am searching for a good method to transfer data over internet, and I work in C++/windows environment. The data is binary, a compressed blob of an extracted image. Input and requirements are as follows:
6kB/packet # 10 packets/sec (60kBytes per second)
Reliable data transfer
I am new to network programming and so far I could figure out that one of the following methods will be suitable.
Sockets
MSMQ (MS Message Queuing)
The Client runs on a browser (Shows realtime images on browser). While server runs native C++ code. Please let me know if there are any other methods for achieving the same? Which one should I go for and why?
If the server determines the pace at which images are sent, which is what it looks like, a server push style solution would make sense. What most browsers (and even non-browsers) are settling for these days are WebSockets.
The main advantage WebSockets have over most proprietary protocols, apart from becoming a widely adopted standard, is that they run on top of HTTP and can thus permeate (most) proxies and firewalls etc.
On the server side, you could potentially integrate node.js, which allows you to easily implement WebSockets, and comes with a lot of other libraries. It's written in C++, and extensible via C++ and JavaScript, which node.js hosts a VM for. node.js's main feature is being asynchronous at every level, making that style of programming the default.
But of course there are other ways to implement WebSockets on the server side, maybe node.js is more than you need. I have implemented a C++ extension for node.js on Windows and use socket.io to do WebSockets and non-WebSocket transports for older browsers, and that has worked out fine for me.
But that was textual data. In your binary data case, socket.io wouldn't do it, so you could check out other libraries that do binary over WebSockets.
Is there any specific reason why you cannot run a server on your windows machine? 60kb/seconds, looks like some kind of an embedded device?
Based on our description, you ned to show image information, in realtime on a browser. You can possibly use HTTP. but its stateless, meaning once the information is transferred, you lose the connection. You client needs to poll the C++/Windows machine. If you are prety confident the information generated is periodic, you can use this approach. This requires a server, so only if a yes to my first question
A chat protocol. Something like a Jabber client running on your client, and a Jabber server on your C++/Windows machine. Chat protocols allow almost realtime
While it may seem to make sense, I wouldn't use MSMQ in this scenario. You may not run into a problem now, but MSMQ messages are limited in size and you may eventually hit a wall because of this.
I would use TCP for this application, TCP is built with reliability in mind and you can simply feed data through a socket. You may have to figure out a very simple protocol yourself but it should be the best choice.
Unless you are using an embedded device that understands MSMQ out of the box, your best bet to use MSMQ would be to use a proxy and you are then still forced to play with TCP and possibly HTTP.
I do home automation that includes security cameras on my personal time and I use the .net micro framework and even if it did have MSMQ capabilities I still wouldn't use it.
I recommend that you look into MJPEG (Motion JPEG) which sounds exactly like what you would like to do.
http://www.codeproject.com/Articles/371955/Motion-JPEG-Streaming-Server
I would like to write an application to manage files, directories and processes on hundreds of remote PCs. There are measurement programs running on these machines, which are currently managed manually using TightVNC / RealVNC. Since the number of machines is large (and increasing) there is a need for automatic management. The plan is that our operators would get a scriptable client application, from which they could send queries and commands to server applications running on each remote PC.
For the communication, I would like to use a TCP-based custom protocol, but it is administratively complicated and would take very long to open pinholes in every firewall in the way. Fortunately, there is a program with a built-in TinyWeb-based custom web server running on every remote PC, and port 80 is opened in every firewall. These web servers serve requests coming from a central server, by starting a CGI program, which loads and sends back parts of the log files of measurement programs.
So the plan is to write a CGI program, and communicate with it from the clients through HTTP (using GET and POST). Although (most of) the remote PCs are inside the corporate intranet, they are scattered all over the country, I would like to secure the communication. It would not be wise to send commands, which manipulate files and processes, in plain text. Unfortunately the program which contains the web server cannot be touched, so I cannot simply prepare it for HTTPS. I can only implement the security layer in the client and in the CGI program. What should I do?
I have read all similar questions in SO, but I am still not sure what to do in this specific situation. Thank you for your help.
There are several webshells but as far as I can see ( http://www-personal.umich.edu/~mressl/webshell/features.html ) they run on the top of an existing SSL/TLS layer.
There is also S-HTTP.
There are several ways of authenticating to an server (username/passwort) in a protected way, without SSL. http://www.switchonthecode.com/tutorials/secure-authentication-without-ssl-using-javascript . But these solutions are focused only on sending a username/password to the server.
Would it be possible to implement something like message-level security in SOAP/WS-Security? I realise this might be a bit heavy duty and complicated to implement, but at least it is
standardised
definitely secure
possibly supported by some libraries or frameworks you could use
suitable for HTTP
I'm developing a multi-player game and I know nothing about how to connect from one client to another via a server. Where do I start? Are there any whizzy open source projects which provide the communication framework into which I can drop my message data or do I have to write a load of complicated multi-threaded sockety code? Does the picture change at all if teh clients are running on phones?
I am language agnostic, although ideally I would have a Flash or Qt front end and a Java server, but that may be being a bit greedy.
I have spent a few hours googling, but the whole topic is new to me and I'm a bit lost. I'd appreciate help of any kind - including how to tag this question.
If latency isn't a huge issue, you could just implement a few web services to do message passing. This would not be a slow as you might think, and is easy to implement across languages. The downside is the client has to poll the server to get updates. so you could be looking at a few hundred ms to get from one client to another.
You can also use the built in flex messaging interface. There are provisions there to allow client to client interactions.
Typically game engines send UDP packets because of latency. The fact is that TCP is just not fast enough and reliability is less of a concern than speed is.
Web services would compound the latency issues inherent in TCP due to additional overhead. Further, they would eat up memory depending on number of expected players. Finally, they have a large amount of payload overhead that you just don't need (xml anyone?).
There are several ways to go about this. One way is centralized messaging (client/server). This means that you would have a java server listening for udp packets from the clients. It would then rebroadcast them to any of the relevant users.
A second way is decentralized (peer to peer). A client registers with the server to state what game / world it's in. From that it gets a list of other clients in that world. The server maintains that list and notifies the other clients of people who join / drop out.
From that point forward clients broadcast udp packets directly to the other users.
If you look for communication framework with high performance try look at ACE C++ framework (it has Java bindings).
Official web-site is: http://www.cs.wustl.edu/~schmidt/ACE-overview.html
You could also look into Flash Media Interactive Server, or if you want a Java implementation, Wowsa or Red5. Those use AMF and provide native functionality for ShareObjects including synching of the ShareObjects among connected clients.
Those aren't peer to peer though (yet, it's coming soon I hear). They use centralized messaging managed by the server.
Good luck
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
If I have a separate system with its own concept of users and presence, what is the most appropriate architecture for creating a bridge to an XMPP server network? As far as I can tell there are three primary ways:
Act as a server. This creates one touchpoint, but I fear it has implications for compatibility, and potentially creates complexity in my system for emulating a server.
Act as a clients. This seems to imply that I need one connection per user in my system, which just isn't going to scale well.
I've heard of an XMPP gateway protocol, but it's unclear if this is any better than the client solution. I also can't tell if this is standard or not.
Any suggestions or tradeoffs would be appreciated. For example, would any of these solutions require running code inside the target XMPP server (not likely something I can do).
The XMPP gateway protocol you've heard of is most likely to do with transports. A transport is a server that connects to both a XMPP server and a non-XMPP server. By running a transport, I can use my Jabber client to talk to someone using, say, MSN Messenger.
A transport typically connects once to the remote network for each JID that it sees as online. That is, it's your option 2 in reverse. This is because there is no special relationship between the transport and the non-XMPP network; the transport is simply acting as a bunch of regular clients. For this to work, XMPP clients must first register with the transport, giving login credentials for the remote network, and allowing the transport to view their presence.
The only reason this has a chance of scaling better is that there can be many transports for the same remote network. For example, my Jabber server could run a transport to MSN, another Jabber server could run another one, and so on, each one providing connections for a different subset of XMPP users. While this spreads out the load on the Jabber side, and load balancing on your system may spread out the load as well, it still requires many connections between the two systems.
In your case, because (I assume) the non-XMPP side of things is cooperating, putting a XMPP server interface on the non-XMPP server is likely your best bet. That server interface is best suited for managing the mapping between XMPP JIDs and how that JID will appear on its own network, rather than forcing XMPP users to register and so on.
In case you haven't seen these, you might find them useful:
http://www.jabber.org/jabber-for-geeks/technology-overview
http://www.xmpp.org/protocols/
http://www.xmpp.org/extensions/
Hope that helps.
I too am working on a similar system.
I am going with the gateway/component route. I have looked at several options and settled with this one.
The gateway is basically a component with the specific purpose of bridging Jabber/XMPP with another network. You will have to build most of the things you take for granted when using XMPP as a client. Stuff like roster control.
There is very little help online on the actual design and building of a component. Like the above answer I found that the xmpp protocols/extensions to be of help. The main ones being:
Basic Client 2008
Basic Server 2008
Intermediate Client 2008
Intermediate Server 2008
Reading through these will show you what XEPs you will be expected to be able to handle. Ignore the stuff that will be handled by the server that your component will be attched to.
It's a shame that Djabberd has such poor documentation as their system of "everything is a module" gave the possibility of backend of the server could interface directly to the other network. I made no headway on this.
There are basically two types of server to server (s2s) connections. The first is either called a gateway or a transport, but they're the same thing. This is probably the kind you're looking for. I couldn't find specific documentation for the non-XMPP side, but how XMPP thinks about doing translations to legacy servers is at http://xmpp.org/extensions/xep-0100.html. The second kind really isn't explained in any additional XEPs -- it's regular XMPP s2s connections. Look for "Server-to-Server Communication" in RFC 3920 or RFC 3920bis for the latest draft update.
Since you have your own users and presence on your server, and it's not XMPP, the concepts aren't going to map completely to the XMPP model. This is where the work of the transport comes in. You have to do the translation from your model to the XMPP model. While this is some work, you do get to make all the decisions.
Which brings us right to one of the key design choices -- you need to really decide which things you are going to map to XMPP from your service and what you aren't. These feature and use case descriptions will drive the overall structure. For example, is this like a transport to talk to AOL or MSN chat services? Then you'll need a way to map their equivalent of rosters, presence, and keep session information along with logins and passwords from your local users to the remote server. This is because your transport will need to pretend to be those users and will need to login for them.
Or, maybe you're just an s2s bridge to someone else's XMPP based chess game, so you don't need a login on the remote server, and can just act similarly to an email server and pass the information back and forth. (With normal s2s connections the only session that would be stored would be SASL authentication used with the remote server, but at the user level s2s just maintains the connection, and not the login session.)
Other factors are scalability and modularity on your end. You nailed some of the scalability concerns. Take a look at putting in multiple transports to balance the load. For modularity, see where you want to make decisions about what to do with each packet or action. For example, how do you handle and keep track of subscription data? You can put it on your transport, but then that makes using multiple transports harder. Or if you make that decision closer to your core server you can have simpler transports and use some common code if you need to talk to services other than XMPP. The trade off is a more complex core server with more vulnerability potential.
What architecture you should use depends on the non-XMPP system.
Do you operate the non-XMPP system? If yes, you should find a way to add an XMPP-S2S interface to that system, in other words, make it act as an XMPP server. AOL is using this approach for AIM. Unfortunately, they have restricted their gateway to GoogleTalk.
You don't operate the non-XMPP system but it has a federation interface that you can use - i. e. your gateway can talk to the other system as a server and has a namespace of its own. In this case, you can build a gateway that acts as a federated server on both sides. For I don't know of any example of a gateway that uses this approach but you could use it if you want to build a public XMPP-to-SIP bridge.
If the non-XMPP system doesn't give you a federation interface, then you have no other option but acting as a bunch of clients. In the XMPP world, this is called a "transport". The differences between a transport and a normal server are basically:
the JIDs of the transport are mapped from another system (e.g. john.doe\40example.net#msngateway.example.org - really ugly!)
XMPP users who want to use the transport need to create an account on the non-XMPP system and give the login credentials of that account to the transport service. The XMPP protocol even has a protocol extension that allows XMPP users to do transport registrations in-band.
One other approach is to work with your XMPP server vendor. Most have internal APIs that make injecting presence possible from third party applications. For example, Jabber XCP provides an API for this that's really easy to use.
(Disclosure: I work for Jabber, Inc, the company behind Jabber XCP)