I am exploring Apache Kafka for building an application and it really suits our needs. However, as part of our application, we also need to push data streams and notifications to user’s apps and web dashboards. AFAIK Kafka clients do not support JavaScript so integration with web dashboards seems to be a challenge.
I want to know
What would be the best way to extend messages received in Kafka topic to web dashboards, assuming thousands of logged-in users. ( web socket is no option here)
What are the best available Kafka clients for iOS and Android.
Also, I am wondering that generally speaking, what would be the best way to fan-out messages coming in from a Kafka topic to thousands of end devices ( like websites, phones, IoT terminals, etc.). Any general or tool-specific suggestion would be awesome.
-Shyam
kafka-connect is a good use-case.
Please check the details here : https://www.npmjs.com/package/kafka-connect.
The other option is to have a micro-service to produce and consume. It is a good option if you later want to do additional operations before publish or after publishing.
Related
I am hoping this is not too off topic for a post here.
I have an asp.net webAPI service, which provides a number of routes to get near realtime data (ie within say 10 seconds), which required the client application to poll for changes.
I am investigating on which technology would be best to add an "opt in" push notification service, which just pushes "thin" payloads to tell the client application it is now time to call the existing REST route for an update. This way, the push payload is small, and does not contain any security sensitive data (it still gets this using the existing REST security infrastructure)
Cloud based messaging
Previously, I have been told that, for a Mobile application, I should use something like Firebase cloud messaging, or some other messaging service, however this does not seem like the right solution for "subscription based notifications" I am talking about here. I can certainly see this would be useful, if the client is on either iOS or Android device, and wanted messages/notifications/alarms (etc), which could also work when the application is not running, but this does not seem like the right thing to use of these notifications of changed data (which may be happening all the time, sometimes every 5 seconds). Also, I do not want to only target these mobile devices, but also, for example either a web or desktop application, which may also use the same REST service
Other technologies
I have seen mention of Web sockets, or, in the case of asp.net, the option to use SignalR (which will wrap the web sockets, with fallback). SignalR looks good, but my worry is the availability of client libraries for non web / Windows applications (eg iOS, Android). I am also looking at Rest Hooks. These look interesting, but I can't quite see what the actual "push mechanism" is; it almost looks like they need to POST to the subscriber using HTTP, which means the subscriber has to also act as a "server endpoint".
Just after any thoughts / best practices on this, or what others have used?
In particular, (the verification or otherwise), that for this use case, using cloud based messaging is not the right thing to use due to the frequency of these push notifications (ie something where my server gets to the application via another 3rd party service which pushes to the device/application)
Thanks in advance for any suggestions!
Signalr is an option
There are some libraries which you can use in iOS and android. I suggest you to read once https://visualstudiomagazine.com/articles/2013/11/01/how-to-use-signalr-in-ios-and-android-apps.aspx (its a bit older, but on the point)
Android Client: See How to use signalr in Android
Some alternatives :
Pusher (https://pusher.com/)
Android Client: https://pusher.com/docs/android_quick_start
iOS Client: https://pusher.com/docs/ios_quick_start
Socket.IO (https://socket.io/)
Details iOS Client: https://socket.io/blog/socket-io-on-ios/
Details Andriod Client: https://github.com/socketio/socket.io-client-java
To discuss:
Why you will only send a thin payload whith signalr? I see no benefit for that.
Why "using cloud based messaging is not the right thing"? I do not understand your arguments but I do not know how your application looks like.
I'm designing a database monitoring application. Basically, the database will be hosted in the cloud and record-level access to it will be provided via custom written clients for Windows, iOS, Android etc. The basic scenario can be implemented via web services (ASP.NET WebAPI). For example, the client will make a GET request to the web service to fetch an entry. However, one of the requirements is that the client should automatically refresh UI, in case another user (using a different instance of the client) updates the same record AND the auto-refresh needs to happen under a second of record being updated - so that info is always up-to-date.
Polling could be an option but the active clients could number in hundreds of thousands, so I'm looking for a more robust and lightweight (on server) solution. I'm versed in .NET and C++/Windows and I could roll-out a complete solution in C++/Windows using IO Completion Ports but feel like that would be an overkill and require too much development time. Looked into ASP.NET WebAPI but not being able to send out notifications is its limitation. Are there any frameworks/technologies in Windows ecosystem that can address this scenario and scale easily as well? Any good options outside windows ecosystem e.g. node.js?
You did not specify a database that can be used so if you are able to use MSSQL Server, you may want to lookup SQL Dependency feature. IF configured and used correctly, you will be notified if there are any changes in the database.
Pair this with SignalR or any real-time front-end framework of your choice and you'll have real-time updates as you described.
One catch though is that SQL Dependency only tells you that something changed. Whatever it was, you are responsible to track which record it is. That adds an extra layer of difficulty but is much better than polling.
You may want to search through the sqldependency tag here at SO to go from here to where you want your app to be.
My first thought was to have webservice call that "stays alive" or the html5 protocol called WebSockets. You can maintain lots of connections but hundreds of thousands seems too large. Therefore the webservice needs to have a way to contact the clients with stateless connections. So build a webservice in the client that the webservices server can communicate with. This may be an issue due to firewall issues.
If firewalls are not an issue then you may not need a webservice in the client. You can instead implement a server socket on the client.
For mobile clients, if implementing a server socket is not a possibility then use push notifications. Perhaps look at https://stackoverflow.com/a/6676586/4350148 for a similar issue.
Finally you may want to consider a content delivery network.
One last point is that hopefully you don't need to contact all 100000 users within 1 second. I am assuming that with so many users you have quite a few servers.
Take a look at Maximum concurrent Socket.IO connections regarding the max number of open websocket connections;
Also consider whether your estimate of on the order of 100000 of simultaneous users is accurate.
I am pretty new to stuff related to server and backend services and I want to develop a mobile app with a backend part. I want this backend to serve an ios app, an android app as well as a website.
My concerns today are how does the frontend part communicate with the backend part :
does it work the same way a website works ? (Http request to the server ?)
how does happen the exchange of datas between the frontend and the backend ?
which are the common solutions to my problem ?
is there an efficient way to desing this backend to serve mobile apps as well as a website ?
is parse (https://parse.com/) a good starting point ?
Thanks
Looking at your questions in turn:
does it work the same way a website works ? (Http request to the server ?)
There are many options, but probably the most common, or fashionable, at the moment is to use a RESTFUL interface:
http://en.wikipedia.org/wiki/Representational_state_transfer
Previously, a SOAP based web service might have been the most common choice:
http://en.wikipedia.org/wiki/SOAP
See here for some discussion on why you might use REST rather than the SOAP now:
Why would one use REST instead of SOAP based services?
how does happen the exchange of datas between the frontend and the backend ?
Assuming REST, HTTP is used to transport messages and application data is typically included in XML or JSON forms
which are the common solutions to my problem ?
I think this is covered by the other parts of the question/answer.
is there an efficient way to desing this backend to serve mobile apps as well as a website ?
Thats very dependent on your particular server application, especially its size and architecture. If the server application is broken down into components or parts, and the parts that generate the 'views' or the 'HTML' pages for the web app are distinct and well separated from the 'backend' parts of your server application, AND your application is of a type that the functionality is largely the same whether the end user is using a web site or a mobile and it is just the way the view are generated for the different devices that differs, then an efficient design would be one that keeps as much of the backend common as possible. If the use of the application is very different when used by a mobile client this may not make sense. More generally, an efficient design would keep as much functionality as possible common between the Mobile and Web applications.
It would definitely be worth becoming familiar with the 'Model View Controller' architectural pattern as most of the server side frameworks, as well as many of the Javascript Web client frameworks and even the iOS and (to a lesser extent) Android frameworks use these concepts:
http://en.wikipedia.org/wiki/Model–view–controller
One important considerations whether you need 'push' or notification like functionality on your mobile app. If so you may want to look at some of the common solutions to understand if they meet your needs - probably easiest to start with Apple and Google's offerings to get an understanding, but there are lots of other solutions available also:
https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/ApplePushService.html
http://developer.android.com/google/gcm/index.html
is parse (https://parse.com/) a good starting point ?
I am not familiar with this service but you might be better looking at a simple REST based approach first and see if it meets your needs.
To answer your question
is parse (https://parse.com/) a good starting point ?
Yes it is.
But I would recommend you to read well on topics such as
REST services
RESTful services vs SOAP - a good article
REST/JSON vs REST/JSON
Services such as parse are called Mobile Backend as a Service (MBaaS).They are ideal to quickly create web services for mobile developers who have little experience with backend development.
A quick search on google on 'MBaaS' will return many services similar to parse and most offer free developer accounts. (With a certain Number of free API calls per second/app)
I have used Apigee similarly & the open source equivalent is Usergrid.
These services will provide a GUI for the developer to create & deploy services and the services are immediately available.
Separate test & production end points will be available.
In addition to basic CRUD operations, these services will also enable easy social network integration, caching & analytics (Depends on service provider)
Features such as security, scalability are built in by the MBaaS provider(Like Parse).
We have a very basic application (iOS/Android) done in Appcelerator that will receive a single update every week. This update will be sent to all the users subscribed to the push notifications service.
By this moment, we have around 35k installs but 7,000 active users on this application on last month. We've been evaluating two services for all the push notifications:
StackMob
Parse
Appcelerator Cloud services is fine, but we're not willing to pay that much. Parse and StackMob prices are lower than Appcelerator Cloud services and by our analysis, we could even use the free service on both services (StackMob = 60k push notifications + 60k api calls, and Parse 1M api calls + 1M pushes).
If we're going to use Parse, we'll need to buy the Android and iOS module from the Marketplace ($30/year each). Which is fine. On the counterpart, I think we could use the REST API on StackMob for subscribing to the push service.
Questions:
What are your thoughts on both services? Which one do you prefer and why?
Have you used StackMob REST API for subscribing to push notifications?
How do you retrieve Android's token?
Is there any (cost effective) alternative to these services? I also reviewed PubNub, which seems to be great but costs are higher than StackMob and Parse.
Thanks in advance.
Update
I asked the same question on Appcelerator forums. After a while, users came back with several answers and users using Parse.com for this.
I ended implementing Parse.com, which was really simple by using the Android and iOS plugins that are on the Appcelerator Market.
I wanted to chime in and point you to some StackMob references around Appcelerator.
Aaron Saunders has several projects on github showing how to use StackMob with Appcelerator.
https://github.com/aaronksaunders
He also wrote a series of blog posts about it.
http://developer.appcelerator.com/blog/2011/11/titanium-appcelerator-quickie-stackmob-api-module-part-one.html
Our REST API reference is available at https://developer.stackmob.com/tutorials/dashboard/REST-API-Reference
One of the big differentiators between StackMob and others is our custom code option. You can write your own logic in Java, Scala or Clojure and host it on StackMob. The custom code can interact with your user data and other 3rd party APIs.
https://developer.stackmob.com/tutorials/custom%20code
I haven't used those services myself, so I cant comment. However, Another alternative we use (and have been pretty happy with) is Urban Airship. It's relatively cost effective, supports Android, iOS and BB and it has server side libraries for a bunch different languages. There is also a neat blog post outlining how to easily do device registration (at least for iOS) via simple web requests in Appcelerator.
The blog post on its Appcelerator integration is here.
I have been given access to a real time data feed which provides location information, and I would like to build a website around this, but I am a little unsure on what architecture to use to achieve my needs.
Unfortunately the feed I have access to will only allow a single connection per IP address, therefore building a website that talks directly to the feed is out - as each user would generate a new request, which would be rejected. It would also be desirable to perform some pre-processing on the data, so I guess I will need some kind of back end which retrieves the data, processes it, then makes it available to a website.
From a front end connection perspective, web services sounds like it may work, but would this also create multiple connections to the feed for each user? I would also like the back end connection to be persistent, so that data is retrieved and processed even when the site is not being visited, I believe IIS will recycle web services and websites when they are idle?
I would like to keep the design fairly flexible - in future I will be adding some mobile clients, so the API needs to support remote connections.
The simple solution would have been to log all the processed data to a database, which could then be picked up by the website, but this loses the real-time aspect of the data. Ideally I would be looking to push the data to the website every time the data changes or now data is received.
What is the best way of achieving this, and what technologies are there out there that may assist here? Comet architecture sounds close to what I need, but that would require building a back end that can handle multiple web based queries at once, which seems like quite a task.
Ideally I would be looking for a C# / ASP.NET based solution with Javascript client side, although I guess this question is more based on architecture and concepts than technological implementations of these.
Thanks in advance for all advice!
Realtime Data Consumer
The simplest solution would seem to be having one component that is dedicated to reading the realtime feed. It could then publish the received data on to a queue (or multiple queues) for consumption by other components within your architecture.
This component (A) would be a standalone process, maybe a service.
Queue consumers
The queue(s) can be read by:
a component (B) dedicated to persisting data for future retrieval or querying. If the amount of data is large you could add more components that read from the persistence queue.
a component (C) that publishes the data directly to any connected subscribers. It could also do some processing, but if you are looking at doing large amounts of processing you may need multiple components that perform this task.
Realtime web technology components (D)
If you are using a .NET stack then it seems like SignalR is getting the most traction. You could also look at XSockets (there are more options in my realtime web tech guide. Just search for '.NET'.
You'll want to use signalR to manage subscriptions and then to publish messages to registered client (PubSub - this SO post seems relevant, maybe you can ask for a bit more info).
You could also look at offloading the PubSub component to a hosted service such as Pusher, who I work for. This will handle managing subscriptions and component C would just need to publish data to an appropriate channel. There are other options all listed in the realtime web tech guide.
All these components come with a JavaScript library.
Summary
Components:
A - .NET service - that publishes info to queue(s)
Queues - MSMQ, NServiceBus etc.
B - Could also be a simple .NET service that reads a queue.
C - this really depends on D since some realtime web technologies will be able to directly integrate. But it could also just be a simple .NET service that reads a queue.
D - Realtime web technology that offers a simple way of routing information to subscribers (PubSub).
If you provide any more info I'll update my answer.
A good solution to this would be something like http://rubyeventmachine.com/ or http://nodejs.org/ . It's not asp.net, but it can easily solve the issue of distributing real time data to other users. Since user connections, subscriptions and broadcasting to channels are built in to each, that will make coding the rest super simple. Your clients would just connect over standard tcp.
If you needed clients to poll for updates then you would need a que system to store info for the next request. That could be a simple array, or a more complicated que system depending on your requirements and number of users.
There may be solutions for .net that I am not aware of that do the same thing, but those are the 2 I know of.