How to time-delay email deliveries? - drupal

I'm currently learning about the Drupal email functions, such as drupal_mail and hook_mail, and hook_mail_alter, and I have a problem before me where I'll need to be able to queue emails for delayed delivery. For example, an event signup notification that needs to wait for an hour after the event was registered by a user. And that email has user specific data, so can't be just a generic template email...
I'm using the MailQ module, mainly for it's logging capabilities, but I wonder if it (or something else) could be modified to add a configurable delay function?
Any ideas?
======================
Some additional thoughts:
There are 2 modules in D6 that are useful for queuing emails, MailQ and Job queue (in Nikit's list below). They both have the mail queue functionality, which can be very useful, and both have different approaches, strengths and weaknesses- from a cursory investigation. EG: MailQ has a great logging function, with a very useful admin interface for dealing with categories of mail such as queued, sending, sent, failed etc. And you can set how many emails to go out with each cron run.
But while Job Queue doesn't have those categories and logging-archiving as far as I can tell, it does allow for prioritizing different jobs, which can be very useful if you have different email types going out simultaneously, such as contact confirmations, event signup confirmations, and "invite-a-friend" emails for example.
So my idea of a perfect queue would have both of those modules integrated (they don't get along very well) with an additional delay function for each job type.
But writing such a thing is way beyond my meager talents, so I'll just have to write a little custom module that is activated by cron, and that will look for flagged contacts with a creation date of at least an hour in the past, and find a way to write the relevant info directly into the MailQ DB table.
There seems to be some collision going on when I have both my new function being triggered by cron, as well as the MailQ function being triggered by the same cron run, so I'm trying to sort out how to cron-prioritize the MailQ over my function- so that my function will add the table data AFTER MailQ has run.


It's not the most elegant solution, but I did something in D6 before I knew of transactional email services with hook_mail_alter where I loaded the email parameters into a database table, set the message value to null so it didn't send after the hook, and then later a cron job running every 5 mins would check the table and send a couple messages at a time if any where waiting. Once it sent, I flipped the status to sent. You will get a watchdog error saying the email failed to send.
But with this method, you could either call something like Mailgun/Mandrill and send a scheduled email within the hook, or put it in a database table with a timestamp field like strtotime('+2 weeks') and then have a cron job check on the table every x minutes and send when timestamp conditions are met. hook_mail_alter gives you a field to identify the message, so you could have a switch that would only catch specific emails if you want.
If you're comfortable with custom modules, take a look at the Mandrill module. It likely has a more elegant solution for taking over Drupal's email than what I've described above.

Related

Google PubSub queue for distributed state management

I have n sources that a job depends on.
Each source has a separate topic in Google PubSub; when a source is updated it sends a message in the corresponding topic subscription. When all sources are updated (i.e. when there is at least one new message in each subscription) the job can start.
The job is scheduled with airflow. The DAG starts with a series of parallel tasks one for each subscription that check if a new message has been published, but without aknowledging it. The next task waits for all the previous ones and uses XCOM to see if all contains a message. In that case it proceeds with the job (it first aknowledge the messages), otherwise it stops.
In this way I acknowledge the messages only when they are all available, using PubSub as a coordinator. The messages frequency is once or twice a day at most.
Basically I'm using PubSub as way to keep "state". Suppose I have different jobs that depend on the same source. I can create a subscription for the same topic on each job and it all works fine.
Is there a better way/tool/framework to do this?

According with the volume of message that you have, and from my previous implementations, I can recommend you to persist states in Firestore: serverless, affordable, fast...
When a message is published, trigger a function that persist state in Firestore
Then, trigger the number of processes that you want, query Firestone to check if all the states are OK, and continue or stop.
It's my pattern for synchronization. Not that the best!
Anyway, if you create a subscription per process, it also works. The message are duplicated in each subscription and thus you can process them independently.

Delay Queue for amount of time per session

I'd like to create a system that 'appends' mails to each other.
Situation: Everytime an entity is changed I'd like to send a mail to subscribers of that entity.
But when the entity is changed 10 times on a small time (like 5 / 10 minutes) the subscribers don't need to be spammed with emails.
So I was thinking of creating a 'Queue'. And to be more precise I was thinking about using the Azure Servicebus.
After searching some of the documentation. I found two interesting properties.
SessionId => Would be the entity of the Id
BatchFlushInterval (Client-side batching)
'If the client sends additional messages during this time period, it transmits the messages in a single batch'
This sounded perfect.
In this way I recieve all the 'changes of the entity' in a single batch. And could construct a single e-mail to send.
But I don't seem to find this option anymore in the new "Azure Service Bus NuGet".
Now that I searched for alternatives, I have a feeling this is not a 'normal' practice.
Does someone have some experience in this field?
I would like to avoid having to use a cron job. But if this is the best solution please let me know.
I know this a really broad question and more a 'need for information'. So commenting with links can already make me real happy.
Thanks in advance
Brecht

Don't think Message Sessions or BatchFlushInterval is the approach to take here. What you're looking for is to buffer messages to create a single notification rather than multiple ones. I'd personally go with receiving a batch from the Azure Service Bus and process the batch to "append" notifications.

Using SQS or DynamoDB to control order status

I am building a system that processes orders. Each order will follow a workflow. So this order can be, e.g., booked,accepted,payment approved,cancelled and so on.
Every time a status of a order changes I will post this change to SNS. To know if a status order has changed I will need to make a request to a external API, and compare to the last known status.
The question is: What is the best place to store the last known order status?
1. A SQS queue. So every time I read a message from queue, check status using the external API, delete the message and insert another one with the new status.
2. Use a database (like Dynamo DB) to control the order status.

You should not use the word "store" to describe something happening with stateful facts and a queue. Stateful, factual information should be stored -- persisted -- to a database.
The queue messages should be treated as "hints" on what work needs to be done -- a request to consider the reasonableness of a proposed action, and if reasonable, perform the action.
What I mean by this, is that when a queue consumer sees a message to create an order, it should check the database and create the order if not already present. Update an order? Check the database to see whether the order is in a correct status for the update to occur. (Canceling an order that has already shipped would be an example of a mismatched state).
Queues, by design, can't be as precise and atomic in their operation as a database should. The Two Generals Problem is one of several scenarios that becomes an issue in dealing with queues (and indeed with designing a queue system) -- messages can be lost or delivered more than once.
What happens in a "queue is authoritative" scenario when a message is delivered (received from the queue) more than once? What happens if a message is lost? There's nothing wrong with using a queue, but I respectfully suggest that in this scenario the queue should not be treated as authoritative.

I will go with the database option instead of SQS:
1) option SQS:
You will have one application which will change the status
Add the status value into SQS
Now another application will check your messages and send notification, delete the message
2) Option DynamoDB:
Insert you updated status in DynamoDB
Configure a Lambda function on update of that field
Lambda function will send notifcation
The database option looks clear additionally, you don't have to worry about maintaining any queue plus you can read one message from the queue at a time unless you implement parallel reader to read from the queue. In a database, you can update multiple rows and it will trigger the lambda and you don't have to worry about it.
Hope that helps

Emails via rules are sent multiple times

I created a rule to send an email to the author of the node after saving the node. In case of new nodes and whend changed existing ones.
But the mails are sent multiple times. Sometimes at the same time and sometimes over about 3 hours. Simetimes 10 emails, sometimes 20.
I don't know where I can search for a reason.

you might want to turn debugging on for your workflow (admin/config/workflow/rules/settings) and check the logs after the emails are sent (admin/reports/dblog)
I found out after hours of trying to solve a similar problem, that some triggers were being defined automatically in (admin/structure/trigger/workflow) for my workflow and they were saving and publishing my node again, thus creating a recursion. Drupal stops after a few iterations of this, so a random number of messages is being sent. My server was sending over 40 every time I changed workflow states
Also please look at your rules page and make sure you don't have any contradicting rules in your workflow that makes it run the same stuff all over again.

sending an email, but not now

I'm writing an application where the user will create an appointment, and instantly get an email confirming their appointment. I'd also like to send an email the day of their appointment, to remind them to actually show up.
I'm in ASP.NET (2.0) on MS SQL . The immediate email is no problem, but I'm not sure about the best way to address the reminder email. Basically, I can think of three approaches:
Set up a SQL job that runs every night, kicking off SQL emails to people that have appointments that day.
Somehow send the email with a "do not deliver before" flag, although this seems like something I might be inventing.
Write another application that runs at a certain time every night.
Am I missing something obvious? How can I accomplish this?

Choice #1 would be the best option, create a table of emails to send, and update the table as you send each email. It's also best not to delete the entry but mark it as sent, you never know when you'll have a problem oneday and want to resend out emails, I've seen this happen many times in similar setups.

One caution - tightly coupling the transmission of the initial email in the web application can result in a brittle architecture (e.g. SMTP server not available) - and lost messages.
You can introduce an abstraction layer via an MSMQ for both the initial and the reminder email - and have a service sweeping the queue on a scheduled basis. The initial message can be flagged with an attribute that means "SEND NOW" - the reminder message can be flagged as "SCHEDULED" - and the sweeper simply needs to send any messages that it finds that are of the "SEND NOW" or that are "SCHEDULED" and have a toBeSentDate >= the current date. Once the message is successfully sent - the unit of work can be concluded by deleting the message from the queue.
This approach ensures messages are not lost - and enables the distribution of load to off-peak hours by adjusting the service polling interval.
As Rob Williams points out - my suggestion of MSMQ is a bit of overkill for this specific question...but it is a viable approach to keep in mind when you start looking at problems of scale - and you want (or need) to minimize/reduce database read/write activity (esepcially during peak processing periods).
Hat tip to Rob.

For every larger project I usually also create a service which performs regular or periodical tasks.
The service updates its status and time of last execution somewhere in the database, so that the information is available for applications.
For example, the application posts commands to a command queue, and the service processes them at the schedule time.
I find this solution easier to handle than SQL Server Tasks or Jobs, since it's only a single service that you need to install, rather than ensuring all required Jobs are set up correctly.
Also, as the service is written in C#, I have a more powerful programming language (plus libraries) at hand than T-SQL.
If it's really pure T-SQL stuff that needs to be handled, there will be a Execute_Daily stored procedure that the service is going to call on date change.

Create a separate batch service, as others have suggested, but use it to send ALL of the emails.
The web app should record the need to send notifications in a database table, both for the immediate notice and for the reminder notice, with both records annotated with the desired send date/time.
Using MSMQ is overkill--you already have a database and a simple application. As the complexity grows, MSMQ or something similar might help with that complexity and scalability.
The service should periodically (every few minutes to a few hours) scan the database table for notifications (emails) to send in the near future, send them, and mark them as sent if successful. You could eventually leverage this to also send text messages (SMS) or instant messages (IMs), etc.
While you are at it, you should consider using the Command design pattern, and implement this service as a reusable Command executor. I have done this recently with a web application that needs to keep real estate listing (MLS) data synchronized with a third-party provider.

Your option 2 certainly seems like something you are inventing. I know that my mail system won't hold messages for future delivery if you were to send me something like that.
I don't think you're missing anything obvious. You will need something that runs the day of the appointment to send emails. Whether that might be better as a SQL job or as a separate application would be up to your application architecture.

I would recommend the first option, using either an SQL or other application to run automatically every day to send the e-mails. It's simple, and it works.

Microsoft Office has a delivery delay feature, but I think that is an Outlook thing rather than an Exchange/Mail Server thing, so you're going to have to go with option 1 or 3. Or option 4 would be to write a service. That way you won't have to worry about scheduled tasks to get the option 3 application to run.

If you are planning on having this app hosted at a cheap hosting service (like GoDaddy), then what I'd recommend is to spin off a worker thread in Global.asax at Application_Start and having it sleep, wake-up, send emails, sleep...
Because you won't be able to run something on the SQL Server machine, and you won't be able to install your own service.
I do this, and it works fine.

Resources