I want to build a task scheduling service on the Google Cloud Platform. The tasks can be as simple as triggering a URL. Tasks can be recurring (once an hour, twice a day, every thursday, ...) and can be created and removed dynamically.
Which services/APIs on the Google Cloud Platform can I use for this?
I have looked into Google App Engine cron jobs but there seems to be no way to programmatically modify them. If possible I would like to avoid running a cron job every minute just to check if there is some task to run.
My framework of choice is ASP.NET Core but if there is a better solution available, e.g. in Java, I'm willing to try it out.
As you have found out, App Engine Cron Service does not have an API for programmatically managing cron tasks. Cron tasks are configured using a file called cron.yaml and this file can be programmatically modified and uploaded to google cron service(details). I'm not sure about exact requirements for your task scheduling service, bu this could be a good enough solution for your problem.
Another option would be to run a Google compute engine instance. As this is basically a virtual server maintained by you, you will have full control over it; allowing you to choose OS, backend/frontend technologies etc. For example you can run a Linux server, use an asp.net core backend to manage crontab tasks.
I would also recommend to take a look on Google Cloud Tasks. You can create queues and push/pull tasks to/from them:
Push queues run tasks by delivering HTTP requests to App Engine worker services. They dispatch these requests at a reliable, steady
rate and guarantee reliable task execution. Because you can control
the rate at which tasks are sent from the queue, you can control the
workers' scaling behavior and hence your costs.
Pull queues do not dispatch tasks at all. They depend on other worker services to "lease" tasks from the queue on their own
initiative. Pull queues give you more power and flexibility over when
and where tasks are processed, but they also require you to do more
process management. When a task is leased the leasing worker declares
a deadline. By the time the deadline arrives the worker must either
complete the task and delete it or the Task Queue service will allow
another worker to lease it.
Source: https://cloud.google.com/appengine/docs/standard/java/taskqueue/
Related
Sorry if that has been answered - I couldn't find a complete answer - as there seems to be conflicting resources.
What I'm trying to achieve is the architecture where my backend 'communicates' in real time with millions of clients through Firebase DB (RTD or FS). Kind of like what's drawn here:
The architecture in a nutshell: millions of clients write 'events' to the Firebase DB, my java server listens to these 'events', processes them and writes 'responses' to the Firebase DB that are synced back to the clients.
The question: Is App Engine the best solution? Is it even suitable for the job?
On the one hand, in App Engine's documentation there's an example of it used that way:
https://cloud.google.com/solutions/mobile/mobile-firebase-app-engine-flexible
On the other hand, there are (seemingly) known issues with that approach:
1) App Engine instances awake on http requests, not on firebase events. https://stackoverflow.com/a/38357458/1806956
Jobs have a timeout, so even if we do a cron wakeup every minute, it doesn't ensure (or does it?) that the listener will keep living forever.
2) App Engine does not support the Firebase Admin SDK due to background threads? https://stackoverflow.com/a/45046605/1806956
3) App Engine limits the number of background threads. In a real app, we're talking about potential thousands of concurrent users, all throwing events...
Are the above issues not updated? Thank you...
I have so far been very impressed with the firebase platform for hosting a client-side single page app and for data storage. However, I have one component that I don't know where to host...
I want to have a background process that aperiodically updates the database. The nature of when an update is needed is based on an external source and, although the general timeframe of when updates are available is known, the exact timing is not. My thinking was to have a background task running that has some smarts to determine when an update is needed, and then trigger an update at that time.
I don't know where I would host something like this. I considered running it in a loop in a firebase function, but due to pricing model being based on time, that would get very expensive, and functions are not suited for daemon-type processes. The actual "database update" would be suitable for a function, but not the triggering logic. Also, I have seen functions-cron which does offload the triggering logic, but since my updates are not truly periodic, it doesn't seem exactly appropriate. I haven't looked too much into AppEngine and how that relates to the firebase platform...so basically my question:
What are the options for "reasonably-priced" hosting an always-running background task?
Google App Engine - Standard is something you want to look at more. It is reasonably priced since what you are doing will likely fit into GAE-Std's free daily quota. In GAE-Std, you create a scheduled cron job: GAE will call you task as if it was an incoming web request.
See Firebase doc for integrating with GAE
See GAE doc for cron jobs
I'm currently fighting my way through Event Hubs and EventProcessorHost. All guidance I found so far suggests running an EventProcessor in an Azure Cloud Service worker role. Since those are very slow to deploy and update I was wondering if there is any Azure service that lets me run an EventProcessor in a more agile environment?
So far my rough architecture looks like this
Device > IoT Hub > Stream Analytics Job > Event Hub > [MyEventProcessor] > SignalR > Clients...
Or maybe there is another way of getting from Steam Analytics to fire SignalR messages?
Any recommendations are highly appreciated.
Thanks, Philipp
You may use Azure Web App service with the SignalR enabled and merge your pipeline "steps" [MyEventProcessor] and SignalR into one step.
I have done that a few times, started from the simple SignalR chat demo and added the Event Hub receiver functionality to the SignalR processing. That article is close to what i mean in terms of approach.
You may take a look at Azure WebJobs as well. Basically, it can work as a background service doing your logic. WebJobs SDK has the support of Event Hub.
You can run an EventProcessorHost in any Azure thing that will run arbitrary C# code and will keep running. The options for where you should run it end up depending on how much you want to spend and what you need. So Azure Container Service may be the new fancy deployment system, but it's minimum cost may not be suitable for you. I'm running my binaries that read data from EventHubs on normal Azure Virtual Machines with our deployment system in charge of managing them.
If your front end processes using SignalR to talk to clients have a process that stays around for a while, you could just make each one of those their own logical consumer (consumer group) and have them consume the entire stream. Or even if they don't stay around (ie you're using an Azure hosting option that turns off the process when idle) you could write your receiver to just start at the end of stream (as opposed to reprocessing older data), if that's what your scenario requires.
I am working with an e-commerce platform, and I have a task to synchronize with some remote accounting software. The task requires syncing orders, products, inventory...etc. With large amounts of data being synced,the process can take awhile. So, I don't think asp.net application would be the best place to handle this. So, the requirements are:
To be able to schedule this process to run overnight
To be able to manually fire off this process and pass into it some variables like order numbers to export.
Possibly get back status info when fired off manually.
Has to work on .net 3.5
Issues: Can't use a windows service because the site is hosted remotely on a shared service, and the host won't allow a service.
Ideas: I'm having a really hard time finding the best way to handle this outside asp.net that fits all requirements, but I do have access to their FTP and thought possibly a console app that hosts a web-service may work, and I can put Quartz scheduler in global file to fire off service from the site.
Anyway, please offer some thoughts and experiences if you have them on which methods have worked for you.
Can't use a windows service because the site is hosted remotely on a shared service, and the host won't allow a service.
That might be a problem. Does this hosting service provide any other kind of scheduling functionality? If not then you may need to consider changing your hosting services.
You're correct in that ASP.NET is not the tool you'd use for scheduling tasks. A web application is a request/response system (and is very much at the mercy of the hosting process, IIS usually for ASP.NET). So you need some way to schedule the task to execute at regular intervals. Windows Services, Windows Task Scheduler, or some other task scheduling tool.
As for the requirement to be able to invoke the process manually, that's a simple matter of separating the invocation of the logic from the logic itself. Picture the following components:
A module which performs the logic, not bound to any UI or any way of invoking it. Basically a Class Library project (or part of one).
A Windows Service or Console Application which references the Class Library and invokes the logic.
A Web Application which references the Class Library and invokes the logic.
Once you've sorted out how to schedule the Console Application, just schedule it and it's all set. If the process returns some information then the Console Application can also perform any notifications necessary to inform people of that information.
The Web Application can then also have an interface somewhere to invoke the process manually. Since the process "can take a while" then of course you won't want the interface to wait for it to complete. This can result in timeouts and leave the system in an unknown state. Instead you'd want to return the UI to the user indicating that the process has started (or been queued) and that they will be notified with the results when it completes. There are a couple of options for this...
You can use a BackgroundWorker to actually invoke the process. When the process completes, send a notification to the user who invoked it.
You can write a record to a database table to "queue" the process and have something like a Windows Service or scheduled Console Application (same scenario as above) which regularly polls that table for queued tasks, performs the task, and sends the notification. (Of course updating the status in the table along the way so it doesn't perform it twice.)
There are pros and cons either way, it's really up to you how you'd like to proceed. Ultimately you're looking at two main things here:
Separate the logic itself from the scheduling/invocation of the logic.
Utilize a scheduling system to schedule tasks. (If your hosting provider doesn't have one, find one that does.)
I'm building a project to send messages to users. The client wants a way to schedule these messages to be sent out at a certain time, for example, he creates the message at 2am but wants it to be sent out at 10am without his intervention, where do I begin with this sort of thing? I'm using ASP.NET MVC3, any help is appreciated.
Update
Darin has suggested Quartz.net, I've finally gotten around to attempting to set it up. But I'm not really understanding how to implement it with my web app.
I'm assuming I should be able to make an httprequest from my service to an action on my webapp, triggered by quartz. But I'm not sure how to communicate between the webapp and this service, such as sending instructions to the quartz server.
So far, I've created a windows service, set up the installers, and added the Quartz.net server 2010 solution to my service project, am I on the right track?
Using a managed Windows Service with Quartz.NET or a console application which you would schedule with the Windows task scheduler seems like a good approaches to achieve that.
Welp, there are scheduled tasks... either make a localhost request at a specific time, or write an executable/service to be called.
A possible alternative if you can't use scheduled tasks (but may be dependent upon the website being available to the Internet) is to make a remote request-maker program or use a website monitoring service. Have either of those make a request to a particular page/querystring on your site periodically. Then, make the web application perform a particular task (send an email) whenever that resource is requested.
A few free monitoring services are limited to one request every hour or half-hour, or you can pay to have it checked more often. The resource's code could be made to record the message-sending event, (thus making them only get sent once, regardless of how often the request is made)