I'm digging into creating a simple web interface for scheduling iOS push notifications to occur either at a specific time or periodically in the future.
For example, someone could use this notification data:
"This is a periodic push notification!" - Every Monday - Expires Oct 31
"This will only happen once!" - Sept 20
and have the first one be executed every monday until october 31st and the second occur on september 20th.
I've done some research for some server software I could install to do this sort of thing, but I'm at a bit of a loss as to the recommended software to achieve this sort of thing. Is there ready-made software that could be recommended for scheduling push notifications?
If not, I'm also curious about software that would allow me to schedule my own tasks from input in a web form. Could I add/remove cron tasks through PHP? Or is it more appropriate to use something like Celery for this sort of thing? I guess since I haven't got much web development experience, I'm unsure of what the most appropriate approach and tools would be fore this.
You don't have to make cron task for specific events, you have to call at a define (but high) frequency a program through cron. TMHO having a program updating CRON for the purpose you describe is a bad idea.
Then this specific program has to send notifications based on the business logic you want. I implement such programs in PHP usually, then I can call them through cli or wget (this case require a bit more of work to avoid security problems since an attacker can call it too).
Related
I need to schedule actions (HTTP requests is enough) at a certain point in time. Every programmed request will only run once.
Sure I could implement this myself; saving the event to a database, then have an event-loop check if an action should be launched.
However this is such a generic need, there must be an existing service for this general type of need, feels like something I shouldn't implement myself. Any ideas where this can be found? I'm thinking one could just specify the http request to be saved (uri, body, headers)
AWS sure has a way of doing this using Cloudwatch events with a cron configured at the specific point in time. But this is way to clunky IMO. Is there an existing service/solution for this?
Agenda-Rest is a solution that does exactly what I asked for. It has to be self hosted though, as there seems to be no commercial hosting of it. It's also not actively developed, which could very well be that it's pretty much feature complete. After all it's a small wrapper on top of the library Agenda.
There's an alternative, suggested in a GitHub issue of Agenda-Rest, called AgenDash build on top of the same library. It's actively developed, as of autumn 2022. It's primarily a UI on top of Agenda, but it has rest routes that can be called.
There are also several libraries in various languages that exposes this functionality provided a persistence mechanism
agenda (nodejs + mongodb)
redbeat (python + redis)
db-scheduler (java + any rdbms)
I'm quite surprised that I can't find this functionality as a first class citizen in the public cloud providers.
Edit:
AWS introduced the feature EventBridge Scheduler in nov 2022. It does not allow for a http request per see, but things like invoke a lambda or post a message to a queue is possible. They support one-time schedules so no need for cron and no need for removing it later as mentioned in my question above.
I have a very basic question.
I have built with R a script than can send me reports by email.
I then used task scheduler to send those emails on specific times.
But now I would like to automate more frequently these scripts (let's say every 2 hours) and to only receive an email when the alert I created in my script (a flag) is on, and this even when my pc is off.
As task scheduler won't work, does that mean I need a server or something ?
Does it have a cost ? I am complete newbie in that area. Thank you.
Unfortunately, you would need a separate server for when your computer is off. I might suggest the Free Tier services from Amazon Web Services.
I am testing a feature which requires a Firebase database write to happen at midnight everyday. Now it is possible that at this particular time, the client app might not be connected to the internet.
I have been using Firebase with persistence off as that can potentially cause issues of stale data in another feature of mine.
From my observation, if I disconnect the app before the write and keep it this way for a minute or so, Firebase eventually reconnects when I turn on the connectivity again and performs the write.
My main questions are:
Will this behaviour be consistent even if the connectivity is lost for quite a few hours?
Will Firebase timeout?
Since it is inside a forever running service, does it still need persistence to ensure that writes are not lost? (assume that the service does not restart).
If the service does restart, will the writes get lost?
I have some experience with this exact case, and I actually do NOT recommend the use of a background service for managing your Firebase requests. In fact, I wouldn't recommend managing Firebase requests at all (explained later).
Services, even though we can make them run forever, tend to get killed by the system quite a lot actually (unless you set their CPU priority to a higher level, but even then the system still might kill them).
If you call a Firebase Write call (of any kind), and your service gets killed, the write will get lost as you said. Unless, you create a sophisticated manager in which you store requests that haven't been committed into your internal storage, and load them up each time the service is restarted - but that is a very dirty work to do, considering the fact that Firebase Developers took care of us and made .setPersistenceEnabled(true) :)
I know, you mentioned you don't want to use it, but I STRONGLY advise you to do so. It works like charm, no services required, and you don't have to worry at all about managing your write requests. Perhaps it would be better to solve the other issue you have in order to make this possible.
To sum up, here's what I would do in your case:
I would call the .setPersistenceEnabled(true) someplace at the beginning (extending the Application class and calling it from onCreate() is recommended)
I would use Android's AlarmManager and register a BroadcastReceiver to receive an alarm at midnight (repetitive or not - you decide)
Inside the BroadcastReceiver, I'd simply call a write function of Firebase and worry about nothing :)
To make sure I covered all of your questions:
will this behaviour be consistent....
No. Case-scenario: Midnight time, your service has successfully received the call and is now trying to write into Firebase. If, for example, the user has no connection until 6 AM (just a case scenario), there is a very high chance that the system will kill it during those 6 hours, and your write will get lost. Flight Time, or staying in an area with no internet coverage - both are examples of risky scenarios that could break your app's consistency
Will Firebase Timeout?
It definitely could, as mentioned. I wouldn't take the risk and make a 80-90% working app. Use persistence and have a 100% working app :)
I believe I covered the rest of the questions..
Good luck!
I know there's a bunch of APIs out there that do this, but I also know that the hosting environment (being ASP.NET) puts restrictions on what you can reliably do in a separate thread.
I could be completely wrong, so please correct me if I am, this is however what I think I know.
A request typically timeouts after 120 seconds (this is configurable) but eventually the ASP.NET runtime will kill a request that's taking too long to complete.
The hosting environment, typically IIS, employs process recycling and can at any point decide to recycle your app. When this happens all threads are aborted and the app restarts. I'm however not sure how aggressive it is, it would be kind of stupid to assume that it would abort a normal ongoing HTTP request but I would expect it to abort a thread because it doesn't know anything about the unit of work of a thread.
If you had to create a programming model that easily and reliably and theoretically put a long running task, that would have to run for days, how would you accomplish this from within an ASP.NET application?
The following are my thoughts on the issue:
I've been thinking a long the line of hosting a WCF service in a win32 service. And talk to the service through WCF. This is however not very practical, because the only reason I would choose to do so, is to send tasks (units of work) from several different web apps. I'd then eventually ask the service for status updates and act accordingly. My biggest concern with this is that it would NOT be a particular great experience if I had to deploy every task to the service for it to be able to execute some instructions. There's also this issue of input, how would I feed this service with data if I had a large data set and needed to chew through it?
What I typically do right now is this
SELECT TOP 10 *
FROM WorkItem WITH (ROWLOCK, UPDLOCK, READPAST)
WHERE WorkCompleted IS NULL
It allows me to use a SQL Server database as a work queue and periodically poll the database with this query for work. If the work item completed with success, I mark it as done and proceed until there's nothing more to do. What I don't like is that I could theoretically be interrupted at any point and if I'm in-between success and marking it as done, I could end up processing the same work item twice. I might be a bit paranoid and this might be all fine but as I understand it there's no guarantee that that won't happen...
I know there's been similar questions on SO before but non really answers with a definitive answer. This is a really common thing, yet the ASP.NET hosting environment is ill equipped to handle long-running work.
Please share your thoughts.
Have a look at NServiceBus
NServiceBus is an open source
communications framework for .NET with
build in support for publish/subscribe
and long-running processes.
It is a technology build upon MSMQ, which means that your messages don't get lost since they are persisted to disk. Nevertheless the Framework has an impressive performance and an intuitive API.
John,
I agree that ASP.NET is not suitable for Async tasks as you have described them, nor should it be. It is designed as a web hosting platform, not a back of house processor.
We have had similar situations in the past and we have used a solution similar to what you have described. In summary, keep your WCF service under ASP.NET, use a "Queue" table with a Windows Service as the "QueueProcessor". The client should poll to see if work is done (or use messaging to notify the client).
We used a table that contained the process and it's information (eg InvoicingRun). On that table was a status (Pending, Running, Completed, Failed). The client would submit a new InvoicingRun with a status of Pending. A Windows service (the processor) would poll the database to get any runs that in the pending stage (you could also use SQL Notification so you don't need to poll. If a pending run was found, it would move it to running, do the processing and then move it to completed/failed.
In the case where the process failed fatally (eg DB down, process killed), the run would be left in a running state, and human intervention was required. If the process failed in an non-fatal state (exception, error), the process would be moved to failed, and you can choose to retry or have human intervantion.
If there were multiple processors, the first one to move it to a running state got that job. You can use this method to prevent the job being run twice. Alternate is to do the select then update to running under a transaction. Make sure either of these outside a transaction larger transaction. Sample (rough) SQL:
UPDATE InvoicingRun
SET Status = 2 -- Running
WHERE ID = 1
AND Status = 1 -- Pending
IF ##RowCount = 0
SELECT Cast(0 as bit)
ELSE
SELECT Cast(1 as bit)
Rob
Use a simple background tasks / jobs framework like Hangfire and apply these best practice principals to the design of the rest of your solution:
Keep all actions as small as possible; to achieve this, you should-
Divide long running jobs into batches and queue them (in a Hangfire queue or on a bus of another sort)
Make sure your small jobs (batched parts of long jobs) are idempotent (have all the context they need to run in any order). This way you don't have to use a quete which maintains a sequence; because then you can
Parallelise the execution of jobs in your queue depending on how many nodes you have in your web server farm. You can even control how much load this subjects your farm to (as a trade off to servicing web requests). This ensures that you complete the whole job (all batches) as fast and as efficiently as possible, while not compromising your cluster from servicing web clients.
Have thought about the use the Workflow Foundation instead of your custom implementation? It also allows you to persist states. Tasks could be defined as workflows in this case.
Just some thoughts...
Michael
I'm writing an application where the user will create an appointment, and instantly get an email confirming their appointment. I'd also like to send an email the day of their appointment, to remind them to actually show up.
I'm in ASP.NET (2.0) on MS SQL . The immediate email is no problem, but I'm not sure about the best way to address the reminder email. Basically, I can think of three approaches:
Set up a SQL job that runs every night, kicking off SQL emails to people that have appointments that day.
Somehow send the email with a "do not deliver before" flag, although this seems like something I might be inventing.
Write another application that runs at a certain time every night.
Am I missing something obvious? How can I accomplish this?
Choice #1 would be the best option, create a table of emails to send, and update the table as you send each email. It's also best not to delete the entry but mark it as sent, you never know when you'll have a problem oneday and want to resend out emails, I've seen this happen many times in similar setups.
One caution - tightly coupling the transmission of the initial email in the web application can result in a brittle architecture (e.g. SMTP server not available) - and lost messages.
You can introduce an abstraction layer via an MSMQ for both the initial and the reminder email - and have a service sweeping the queue on a scheduled basis. The initial message can be flagged with an attribute that means "SEND NOW" - the reminder message can be flagged as "SCHEDULED" - and the sweeper simply needs to send any messages that it finds that are of the "SEND NOW" or that are "SCHEDULED" and have a toBeSentDate >= the current date. Once the message is successfully sent - the unit of work can be concluded by deleting the message from the queue.
This approach ensures messages are not lost - and enables the distribution of load to off-peak hours by adjusting the service polling interval.
As Rob Williams points out - my suggestion of MSMQ is a bit of overkill for this specific question...but it is a viable approach to keep in mind when you start looking at problems of scale - and you want (or need) to minimize/reduce database read/write activity (esepcially during peak processing periods).
Hat tip to Rob.
For every larger project I usually also create a service which performs regular or periodical tasks.
The service updates its status and time of last execution somewhere in the database, so that the information is available for applications.
For example, the application posts commands to a command queue, and the service processes them at the schedule time.
I find this solution easier to handle than SQL Server Tasks or Jobs, since it's only a single service that you need to install, rather than ensuring all required Jobs are set up correctly.
Also, as the service is written in C#, I have a more powerful programming language (plus libraries) at hand than T-SQL.
If it's really pure T-SQL stuff that needs to be handled, there will be a Execute_Daily stored procedure that the service is going to call on date change.
Create a separate batch service, as others have suggested, but use it to send ALL of the emails.
The web app should record the need to send notifications in a database table, both for the immediate notice and for the reminder notice, with both records annotated with the desired send date/time.
Using MSMQ is overkill--you already have a database and a simple application. As the complexity grows, MSMQ or something similar might help with that complexity and scalability.
The service should periodically (every few minutes to a few hours) scan the database table for notifications (emails) to send in the near future, send them, and mark them as sent if successful. You could eventually leverage this to also send text messages (SMS) or instant messages (IMs), etc.
While you are at it, you should consider using the Command design pattern, and implement this service as a reusable Command executor. I have done this recently with a web application that needs to keep real estate listing (MLS) data synchronized with a third-party provider.
Your option 2 certainly seems like something you are inventing. I know that my mail system won't hold messages for future delivery if you were to send me something like that.
I don't think you're missing anything obvious. You will need something that runs the day of the appointment to send emails. Whether that might be better as a SQL job or as a separate application would be up to your application architecture.
I would recommend the first option, using either an SQL or other application to run automatically every day to send the e-mails. It's simple, and it works.
Microsoft Office has a delivery delay feature, but I think that is an Outlook thing rather than an Exchange/Mail Server thing, so you're going to have to go with option 1 or 3. Or option 4 would be to write a service. That way you won't have to worry about scheduled tasks to get the option 3 application to run.
If you are planning on having this app hosted at a cheap hosting service (like GoDaddy), then what I'd recommend is to spin off a worker thread in Global.asax at Application_Start and having it sleep, wake-up, send emails, sleep...
Because you won't be able to run something on the SQL Server machine, and you won't be able to install your own service.
I do this, and it works fine.