I know how to build server api with java. Is it possible to send real time data from the arduino to my server api to save real time data.
Yes it's certainly possible however your server API must also be taken into consideration for best practices.
Typically you would use a protocol called MQTT to achieve this. It enables you to transmit data from resource constrained devices to a server, however HTTP has also been used. It really depends on the kinds of performance & reliability needs of your system.
If you're using a popular cloud platform they offer managed IOT services that provide well defined API's to do this.
Related
I want to create a web app using React as the front end technology. A requirement for the app is that the server will be able to update all the clients with information about changes (not have to be an exact real time, but should update after no more than 10 seconds).
Solutions like clients requesting updates from the server every several seconds are out of the question.
Requirements:
1) The server's should be implemented with either .NET or with Node.js.
2) The connection MUST be secured via port 443 of the IIS.
I read a bit about Micorsoft's SignalR and about Pusher Channels which seems to provide exactly the kind of service I require.
Could you please elaborate about what exactly are the differences between them? When should I choose each? Which of them got more community support? Which is easier to implement? Stuff like that...
Both SignalR and Pusher Channels ultimately both use websockets to deliver messages to clients, so both should meet your requirements to deliver messages to clients in realtime.
1) Both offerings also meet your requirements for both library support:
SignalR supports .NET:
https://dotnet.microsoft.com/apps/aspnet/signalr
Pusher Channels has server support for both nodejs and .NET:
https://github.com/pusher/pusher-http-node
https://github.com/pusher/pusher-http-dotnet
2) Both offerings also meet your requirements for sending messages over TLS/WSS:
SignalR:
https://kimsereyblog.blogspot.com/2018/07/signalr-with-asp-net-core.html
Pusher Channels:
Securing Pusher's messages
In terms of the differences between them this depends on your implementation, if you just run SignalR on your own ISS server then it will be down to you to manage all of the websocket connections and all of the scaling challenges that come with this.
However similar to how Channels works, SignalR also has a managed websocket service, so you do not need to manage the connections or scaling. You just make an API request with the message you want to send to either Channels or SignalR and this message is then broadcast to the interested clients connected by websockets. In this scenario you do not manage the websocket connections yourself.
However in terms of pricing Channels appears to be far more competitive (especially the free offering), so if you are looking at the managed offering Channels looks to be a better value proposition:
https://azure.microsoft.com/en-gb/pricing/details/signalr-service/
https://pusher.com/channels/pricing
Both offerings look fairly similar in terms of implementation (assuming you are using the managed service). The complexity would increase if you implement SignalR on ISS:
https://learn.microsoft.com/en-us/aspnet/core/signalr/scale?view=aspnetcore-2.2
In terms of support Pusher has a free application support offering:
https://support.pusher.com/hc/en-us
Hope this helps!
This presentation has some answers A 10 Minute Guide to Choosing a Realtime Framework
As per the KAA references, I understand that once should only use the Notification feature, When it required to send data from server (External apps) to endpoints and Events are only used when there is a need for endpoint to endpoint communication (kind of device binding requirement)
So, To achieve request/response functionality using KAA. I need to implement any hybrid solutions like as below.
1) In my server, I can run one KAA SDK instance and use the event feature for request to the endpoint and response from the endpoint.
OR
2) From my server, I use the notification REST API for request and get the response back through the data logger feature using any in-build appender by configuring "LogUploadStrategy" as to uploads every log record as soon as it is created.
Notes For Point 1
As per Andrew, Solutions Architect of Kaa IoT platform
"You can always embed an SDK to a standalone application and host in
on the same server where kaa-node is present. This application may
receive REST API calls and forward them to particular endpoints via
Kaa events feature. However, this is useful for test purposes. I
would not recommend this solution in production because it is hard to
scale and has potential security issues"
Notes For Point 2
It satisfies the KAA reference document as well as Andrew's suggestion for request only but how can i achieve the response.
Questions For Point 1
1) What causes to scale the application and what type of security issues it faces even through it uses RSA 2048 encryption for communication?
2) Can we embed more then one SDK in standalone application and host in on the same server where kaa-node is present.
Questions For Point 2
3) if device sends the notification response along with the telemetry data, can it increase the latency and any other performance issue.
Common Questions
4) Which one is the better approach to achieve request/response functionality?
Any help or suggestion is really appreciated.
1) What causes to scale the application and what type of security issues it faces even though it uses RSA 2048 encryption for communication?
It makes the EP on the server side as a single point of failure and does not allow load balancing.
About security issues, Andrew meant: This application may receive REST API calls and this forces one to provide additional security for this REST API calls and better use your first hybrid solutions using solely event feature.
2) Can we embed more then one SDK in standalone application and host in on the same server where kaa-node is present.
No, you can't use more than one SDK in one application, but you can run a couple of instance on one machine in different directories
in order to prevent collisions of autogenerated security keys and other files.
3) if device sends the notification response along with the telemetry data, can it increase the latency and any other performance issue.
Of course, you will face some delays if start sending very frequently and big portions of data on both sides. If you have a lot of devices that sends in total a big amount of telemetry data, you can increase performance on the server side by start-up KAA in the cluster mode or add new nodes for processing requests.
4) Which one is the better approach to achieve request/response functionality?
The second hybrid solution – data collection and notification features. This doesn't cause any problem with scale and you can easily launch Kaa server in cluster mode.
I want to create a program for my desktop and an app for my android. Both of them will do the same, just on those different devices. They will be something like personal assistants, so I want to put a lot of data into them ( for example contacts, notes and a huge lot of other stuff). All of this data should be saved on a server (at least for the beginning I will use my own Ubuntu server at home).
For the android app I will obviously use java and the database on the server will be a MySQL database, because that's the database I have used for everything. The Windows program will most likely be written in of these languages: Java, C#c C++, as these are the languages I am able to use quite well.
Now to the problem/question: The server should have a good backend which will be communicating with the apps/programs and read/write data in the database, manage the users and all that stuff. But I am not sure how I should approach programming the backend and the "network communication" itself. I would really like to have some relatively easy way to send secured messages between server and clients, but I have no experience in that matter. I do have programming experience in general, but not with backend and network programming.
side notes:
I would like to "scale big". At first this system will only be used by me, but it may be opened to more people or even sold.
Also I would really like to a (partly) self programmed backend on the server, because I could very well use this for a lot of other stuff, like some automation features in my house, which will be implemented.
EDIT: I would like to be able to scale big. I don't need support for hundreds of people at the beginning ;)
You need to research Socket programming. They provide relatively easy, secured network communication. Essentially, you will create some sort of connection or socket listener on your server. The clients will create Sockets, initialize them to connect to a certain IP address and port number, and then connect. Once the server receives these connections, the server creates a Socket for that specific connection, and the two sockets can communicate back and forth.
If you want your server to be able to handle multiple clients, I suggest creating a new Thread every time the server receives a connection, and that Thread will be dedicated to that specific client connection. Having a multi-threaded server where each client has its own dedicated Thread is a good starting point for an efficient server.
Here are some good C# examples of Socket clients and servers: https://msdn.microsoft.com/en-us/library/w89fhyex(v=vs.110).aspx
As a side note, you can also write Android apps in C# with Xamarin. If you did your desktop program and Android app both in C#, you'd be able to write most of the code once and share it between the two apps easily.
I suggest you start learning socket programming by creating very simple client and server applications in order to grasp how they will be communicating in your larger project. Once you can grasp the communication procedures well enough, start designing your larger project.
But I am not sure how I should approach programming the backend and
the "network communication" itself.
Traditionally, a server for your case would be a web server exposing REST API (JSON). All clients need to do http requests and render/parse JSON. REST API is mapped to database calls and exposes some data model. If it was in Java, it would be Jetty web server, Jackson Json parser.
I would really like to have some relatively easy way to send secured
messages between server and clients,
Sending HTTP requests probably the easiest way to communicate with a service. Having it secured is a matter of enabling HTTPS on the server side and implementing some user access authentication and action authorization. Enabling HTTPS with Jetty for Java will require few lines of code. Authentication is usually done via OAuth2 technique, and authorization could be based on ACL. You may go beyond of this and enable encryption of data at rest and employ other practices.
I would like to "scale big". At first this system will only be used by
me, but it may be opened to more people or even sold.
I would like to be able to scale big. I don't need support for
hundreds of people at the beginning
I anticipate scalability can become the main challenge. Depending on how far you want to scale, you may need to go to distributed (Big Data) databases and distributed serving and messaging layers.
Also I would really like to a (partly) self programmed backend on the
server, because I could very well use this for a lot of other stuff,
like some automation features in my house, which will be implemented.
I am not sure what you mean self-programmed. Usually a backend encapsulates some application specific business logic.
It could be a piece of logic between your database and http transport layer.
In more complicated scenario your logic can be put into asynchronous service behind the backend, so the service can do it's job without blocking clients' requests.
And in the most (probably) complicated scenario your backend may do machine learning (for example, if you would like you software stack to learn your home-being habits and automate house accordingly to your expectations without actually coding this automation)
but I have no experience in that matter. I do have programming
experience in general, but not with backend and network programming.
If you can code, writing a backend is not very hard problem. There are a lot of resources. However, you would need time (or money) to learn and to do it, what may distract you from the development of your applications or you may enjoy it.
The alternative to in-house developed of a backend could be a Backend-as-a-Service (BaaS) in cloud or on premises. There are number of product in this market. BaaS will allow you to eliminate the development of the backend entirely (or close to this). At minimum it should do:
REST API to data storage with configurable data model,
security,
scalability,
custom business-logic
Disclaimer: I am a member of webintrinsics.io team, which is a Backend-as-a-Service. Check our website and contact if you need to, we will be able to work with you and help you either with BaaS or with guiding you towards some useful resources.
Good luck with your work!
I am searching for a good method to transfer data over internet, and I work in C++/windows environment. The data is binary, a compressed blob of an extracted image. Input and requirements are as follows:
6kB/packet # 10 packets/sec (60kBytes per second)
Reliable data transfer
I am new to network programming and so far I could figure out that one of the following methods will be suitable.
Sockets
MSMQ (MS Message Queuing)
The Client runs on a browser (Shows realtime images on browser). While server runs native C++ code. Please let me know if there are any other methods for achieving the same? Which one should I go for and why?
If the server determines the pace at which images are sent, which is what it looks like, a server push style solution would make sense. What most browsers (and even non-browsers) are settling for these days are WebSockets.
The main advantage WebSockets have over most proprietary protocols, apart from becoming a widely adopted standard, is that they run on top of HTTP and can thus permeate (most) proxies and firewalls etc.
On the server side, you could potentially integrate node.js, which allows you to easily implement WebSockets, and comes with a lot of other libraries. It's written in C++, and extensible via C++ and JavaScript, which node.js hosts a VM for. node.js's main feature is being asynchronous at every level, making that style of programming the default.
But of course there are other ways to implement WebSockets on the server side, maybe node.js is more than you need. I have implemented a C++ extension for node.js on Windows and use socket.io to do WebSockets and non-WebSocket transports for older browsers, and that has worked out fine for me.
But that was textual data. In your binary data case, socket.io wouldn't do it, so you could check out other libraries that do binary over WebSockets.
Is there any specific reason why you cannot run a server on your windows machine? 60kb/seconds, looks like some kind of an embedded device?
Based on our description, you ned to show image information, in realtime on a browser. You can possibly use HTTP. but its stateless, meaning once the information is transferred, you lose the connection. You client needs to poll the C++/Windows machine. If you are prety confident the information generated is periodic, you can use this approach. This requires a server, so only if a yes to my first question
A chat protocol. Something like a Jabber client running on your client, and a Jabber server on your C++/Windows machine. Chat protocols allow almost realtime
While it may seem to make sense, I wouldn't use MSMQ in this scenario. You may not run into a problem now, but MSMQ messages are limited in size and you may eventually hit a wall because of this.
I would use TCP for this application, TCP is built with reliability in mind and you can simply feed data through a socket. You may have to figure out a very simple protocol yourself but it should be the best choice.
Unless you are using an embedded device that understands MSMQ out of the box, your best bet to use MSMQ would be to use a proxy and you are then still forced to play with TCP and possibly HTTP.
I do home automation that includes security cameras on my personal time and I use the .net micro framework and even if it did have MSMQ capabilities I still wouldn't use it.
I recommend that you look into MJPEG (Motion JPEG) which sounds exactly like what you would like to do.
http://www.codeproject.com/Articles/371955/Motion-JPEG-Streaming-Server
I'm developing a multi-player game and I know nothing about how to connect from one client to another via a server. Where do I start? Are there any whizzy open source projects which provide the communication framework into which I can drop my message data or do I have to write a load of complicated multi-threaded sockety code? Does the picture change at all if teh clients are running on phones?
I am language agnostic, although ideally I would have a Flash or Qt front end and a Java server, but that may be being a bit greedy.
I have spent a few hours googling, but the whole topic is new to me and I'm a bit lost. I'd appreciate help of any kind - including how to tag this question.
If latency isn't a huge issue, you could just implement a few web services to do message passing. This would not be a slow as you might think, and is easy to implement across languages. The downside is the client has to poll the server to get updates. so you could be looking at a few hundred ms to get from one client to another.
You can also use the built in flex messaging interface. There are provisions there to allow client to client interactions.
Typically game engines send UDP packets because of latency. The fact is that TCP is just not fast enough and reliability is less of a concern than speed is.
Web services would compound the latency issues inherent in TCP due to additional overhead. Further, they would eat up memory depending on number of expected players. Finally, they have a large amount of payload overhead that you just don't need (xml anyone?).
There are several ways to go about this. One way is centralized messaging (client/server). This means that you would have a java server listening for udp packets from the clients. It would then rebroadcast them to any of the relevant users.
A second way is decentralized (peer to peer). A client registers with the server to state what game / world it's in. From that it gets a list of other clients in that world. The server maintains that list and notifies the other clients of people who join / drop out.
From that point forward clients broadcast udp packets directly to the other users.
If you look for communication framework with high performance try look at ACE C++ framework (it has Java bindings).
Official web-site is: http://www.cs.wustl.edu/~schmidt/ACE-overview.html
You could also look into Flash Media Interactive Server, or if you want a Java implementation, Wowsa or Red5. Those use AMF and provide native functionality for ShareObjects including synching of the ShareObjects among connected clients.
Those aren't peer to peer though (yet, it's coming soon I hear). They use centralized messaging managed by the server.
Good luck