I have an ASP.NET website with a a number of long-running (5 mins to 2 hours) user-initiated tasks. I want each user to be able to see the progress of there own jobs, and be able to close their browser and return at a later time.
Current plan is to store each job in the database when it's started and publish a message to a RabbitMQ queue, which a windows service will receive and start processing the job.
However, I'm not sure of the best way to pass the progress information back to the webserver from the service? I see two options:
Store the progress information in the database, and have the web-app poll for it
Have a RabbitMQ consumer in the webserver and have the windows service post progress messages to that queue
I'm leaning towards the second option, as I don't really want to add more overhead to the database by regular polling / writing progress info. However, there are lots of warnings about using RabbitMQ (as a consumer) - as I am not sending vital messages (it doesn't matter if progress messages aren't processed), I'm wondering if this matters? It's not that (famous last words) difficult to restart the RabbitMQ consumer whenever the web app is restarted.
Does that option sound reasonable? Any better choices out there?
Store the progress information in the database, and have the web-app poll for it
Have a RabbitMQ consumer in the webserver and have the windows service post progress messages to that queue
the correct answer is C) All Of The Above!
A database is not an integration layer for applications.
RabbitMQ is not meant for end-user consumption of messages.
But when you combine RabbitMQ with a database, you get beautiful things...
have your background service send progress updates through RabbitMQ. the web server will listen for these updates and write the new status to the database. use websockets (signalr) to push the progress update to the user immediately, but you still have the current status in the database in case the user does a full refresh of the page or comes back later.
i wrote about this basic setup in a blog post on using rabbitmq to do user notifications
Related
I am trying to communicate between four services in Microservice. Every service has serperate Database. I am using saga pattern.
My question is, if RabbitMQ goes down or crash or no way to come back, how would I rollback or revert database without help of message Queue?
I'm assuming that you're using SAGA pattern to maintain consistency in your microservice application. Main component for this pattern is the Orchestrator which coordinates your microservice and I'm assuming that in your application you are using rabitmq for message passing.
One way is to store your message in the local database of your microservice if you're not able to send a message to the queue you can add a scheduler that will try to send the message to the queue after some interval.
The second way can be to use rabitmq in cluster mode this will prevent a Single Point of failure for your application.
In an interesting blogpost about 'Firebase Authentication with the Firebase 3.0 SDK and Auth0 Integration', it is stated that:
You can even have Firebase communicate with Webtask!
Now I can imagine the (web)client triggering a Firebase operation and subsequently a Webtask, but not the other way around. Or am I missing something?
Firebase can run as a serverless app, but it can also run on the server. You can even have Firebase communicate with Webtask! (sic!)
I think that paragraph is misleadingly phrased, perhaps it was just added at the last minute to spark interest. You can have a webtask communicate with Firebase, not the other way around. You don't "run Firebase" on your server either.
TL;DR: A client application may call a webtask with an HTTP request, and that task can read/write the database, but not in any other order.
Here's a quick and dirty reality check as of Nov. 2016:
The Realtime Database by itself does not provide you with a way of executing code. This includes responding to database changes and user requests, handling fan-in and fan-out operations, etc. There is no support for webhooks either.
Which means you have to provide your own execution environment for such logic on a custom server, or you can try to cram as much as possible into the client code. This is a pretty exhaustive topic by itself.
Webtasks are short-lived functions that respond to HTTP requests. Their lifecycle always starts with a request, so they are not fit for continuously watching the database for changes. But they are perfectly valid for handling requests coming in from your client application.
As you can store "secrets" for the webtasks, you can authenticate the task on an admin access level. This gives you the possibility to verify client tokens – which should be sent along with the request –; perform complex authorization and validation, and perform RTDB write operations you wouldn't trust the clients with.
Or trigger external services securely. The possibilities are close to endless.
I am writing a connected home device (alarm system) which can receive events/messages from a mobile device (e.g. a message to dismiss an alarm).
Example of a scenario (mobile device is an iphone for the sake of the example):
Connected device at home sounds alarm and notifies the iphone (using iOS push notifications, not firebase)
iphone user brings up the app, and clicks "dismiss"
connected device gets "dismiss" message and stops the alarm
I was thinking about using firebase's live database for this interaction, so the iphone app would set a db field, and the home device (which runs python) would subscribe to this field and see that it has been set.
The problem is that this is not a very clean implementation, as I would need the home device to turn off the dismiss field after it has received it, so that subsequent dismiss event can be recognized.
Essentially I am implementing messaging on top of a live database.
Is there a cleaner way to do this in firebase?
If not, is this a reasonable implementation?
Are there alternatives to firebase that take care of such a scenario?
What I really need is a web-based event-broker as-a-service...
I would suggest looking at https://github.com/firebase/firebase-queue
There are a few examples of usage on SO such as My Firebase Queue doesn't do anything after I changed to Firebase 3
Many of our developers are using Firebase with a server to perform tasks like background processing, integrating with third party APIs, or handling advanced authentication requirements. Today, we're introducing Firebase Queue, a fault-tolerant multi-worker job pipeline built on Firebase.
If you're writing your server code in Node, Firebase Queue makes it easy to handle background jobs with a worker queue. We're already using it in our private backups feature, where we reliably handle hundreds of jobs per day across multiple machines. ( https://firebase.googleblog.com/2015/05/introducing-firebase-queue_97.html )
I'm trying to understand what I will need to build on my server for Push notifications to work successfully.
My thoughts were:
The phone sends the notify URL to my server
The server stores the information in a Database
A separate process or PHP script will query the database and open continuous looping process for each device. (Each socket will be querying a 3rd party API)
When there is a change detected in the API for that device a push notification will be sent to the device's notify url.
Is this the right method on what needs to be done. Isn't this going to eat up server resources or is it the expected outcome of Push a push notifications server?
I've produced a simple diagram on all this below:
First of all, let's separate the process in the main stages needed for PUSH.
Device subscription.
Send the PUSH
Process the notification on device.
Subscription
For the subscription, your device (more specifically, your App) must call the PUSH api,for enabling PUSH notifications. This call to the push API will give you a URL that uniquely identify the device where your application is installed and running. You should store this URL on your database, the same way you store a user's email, or a user's phone number. No special black magic here. You only use it when you need to send a communication to a user.
Send the PUSH
For the push stuff, the same approach as for email, or SMS messaging here: "One does not simply make an infinite loop and send a message if any change is detected". What you have to do is, just send the PUSH message when your application needs to. So you have the user to which you want to send a message, instead of opening a SMTP connection to send ane mail, just build the PUSH XML Message and call the URL associated with that user. Some things to consider here are:
Network reliability (you need to retry if you can't connect to the server).
Response error code-handling (you don't need to retry if the server tells you that the phone has uninstalled your application, for example).
Scalability. You don't want to send a PUSH message from your PHP code, because you don't know how long it will take for the task to be completed. You have to make this thing asynchronously. So just queue up all the push messages, you can create a separate process (windows service, nodeJS service, cron job, daemon, etc.) to send the PUSH, handle retries and errors and clean the queue.
Process the notification on Device
So now that you are this far, you need to handle the notification on the phone. It depends on the type of PUSH notification that you are sending:
Tile. You will update the image, text and counter of the application tile, if the user has put your application to the start screen. On client side you need nothing to so, as all these parameters are part of your PUSH request.
Toast. This one requires a title, text (limited to some 35 characters more or less) and a relative URL inside of your APP. Your application will be launched (like when you click on a Toast notification from Twitter, for example) using the URI that you specify in the payload. So a bit of data can be already injected here. You may/or may not make a request to your server for new data. It is up to you.
Raw. This one is pretty much silent. Is not seen by the user if your APP is not running. As you might guess, this kind of PUSH is useful to live update your running APP, instead of continuously polling your server, wasting user battery and bandwidth and wasting your server resources. You can send anything (raw bytes or strings) up to the max size of the payload allowed my Microsoft.
If yo have any more questions, don't hesitate to ask.
Bottom line: separate the PUSH sending, make it async, don't you ever forget that...
Your PHP script that continually pings the database for changes...THAT is what will eat up your system resources. Push notifications go hand in hand with Event Driven Programming. This means that ideally, your code shouldn't continuously ping your DB. Rather, when something happens (ie, an "event"), THEN your code does something...like contact your phone via push notification.
Your steps for push notifications are more or less correct, but are incomplete. Step 4: the server contacts the client via the notify url (which you have). Step 5 is that the client then contacts the server to actually pull down the information it needs. That is: The new information is not provided to the client via the notify url. Once the client has its new information, then the program continues as normal (populates a list, downloads skynet, etc.)
Your third step is very wasteful and not practical if your app is installed on more than a few devices.
Instead, each device should be subscribed to types of server updates it cares about. Your server's DB will have a mapping from each type of update you support to the list of notification channel URLs of devices that care about this update type.
When your server detects an update of type X, it would send a notification to all devices subscribed to that type of update.
I have 2 roles in my Azure cloud service application: a web role (signalr connections here) and a worker role.
The web role uses Azure service bus as its scaleout provider.
At certain points in time the worker role will emit certain events. I'd like to send this data directly to clients connected to a Hub.
My current implementation involves the worker role placing a message on a service bus queue which the web role subscribes to, the web role then forwards this message to clients via a HubContext call.
My question is: how can I send this message directly to connected clients from the worker role? So far I have considered 3 methods:
Configure signalr as in the web role so that they use the same servicebus topic. - this done not work as intended as worker role instances "steal messages" from topic subscriptions intended for the web role. This would seem to be the cleanest way of doing it but configuration is a problem.
Use the .Net client to send a hub message - this is not ideal as it places unnecessary load on the web role, as well as double the amount of service bus messages when compared to the above method.
Manually write a signalr compatible message to the topic - very hacky and succeptable to breaking changes.
I know that the team are currently rewriting scaleout for the next release but will this be possible at some point?
Edit:
I have noticed that this is supported in the RabbitMq implementation.
It seems an issue with my configuration was responsible for the first method not working.
However, it seems like that method is slower end to end (by about 150 ms) even with one less message in the loop.
I will wait and see if the scaleout work brings any improvements to this method before making any changes.