I've a requirement in which i need to send 10000+ mails on a quarterly basis. For this purpose i used a windows service that triggers every day and executes the mailing functionality only after the third month. I've to fetch last three months records from database and need to send one mail for each record.
The problem i faced was the mail server i used do not allow bulk mailing.
How can i do this effectively by providing a delay between each sent (20 mails per minute)?
There are many way to archieve this. We once had a similar requirement and solved it via a home grown service, which would fetch items from a special database table (mail queue) and sent each mail individially. The queue is filled over time by regular business logic. The necessary locking can also be done via db: a SCHEDULE column stores the expected scheduled time of sending the mail. That way the service collects only those mails wich are 'ready' for sending. After successfull send, another column (SENT_TIMESTAMP) is used to mark the success.
We implemented the whole service in ASP and triggered it via regular Windows Task Planner jobs. In your case, the service would start every minute and the queue would provide the next 20 mails. An even easier way could be to utilize SQL Server Jobs. SQL Server is capable of delivering mails to a local SMTP server as well.
If not done yet, please note that SO question as well: What is the best way to send large batches of emails in ASP.NET?
Related
I'm trying to get started implementing Web Push in one of my apps. In the examples I have found, the client's endpoint URL is generally stored in memory with a comment saying something like:
In production you would store this in your database...
Since only registered users of my app can/will get push notifications, my plan was to store the endpoint URL in the user's meta data in my database. So far, so good.
The problem comes when I want to allow the same user to receive notifications on multiple devices. In theory, I will just add a new endpoint to the database for each device the user subscribes with. However, in testing I have noticed that endpoints change with each subscription/unsubscription on the same device. So, if a user subscribes/unsubscribes several times in a row on the same device, I wind up with several endpoints saved for that user (all but one of which are bad).
From what I have read, there is no reliable way to be notified when a user unsubscribes or an endpoint is otherwise invalidated. So, how can I tell if I should remove an old endpoint before adding a new one?
What's to stop a user from effectively mounting a denial of service attack by filling my db with endpoints through repeated subscription/unsubscription?
That's more meant as a joke (I can obvioulsy limit the total endpoints for a given user), but the problem I see is that when it comes time to send a notification, I will blast notification services with hundreds of notifications for invalid endpoints.
I want the subscribe logic on my server to be:
Check if we already have an endpoint saved for this user/device combo
If not add it, if yes, update it
The problem is that I can't figure out how to reliably do #1.
I will just add a new endpoint to the database for each device the user subscribes with
The best approach is to have a table like this:
endpoint | user_id
add an unique constraint (or a primary key) on the endpoint: you don't want to associate the same browser to multiple users, because it's a mess (if an endpoint is already present but it has a different user_id, just update the user_id associated to it)
user_id is a foreign key that points to your users table
if a user subscribes/unsubscribes several times in a row on the same device, I wind up with several endpoints saved for that user (all but one of which are bad).
Yes, unfortunately the push API has a wild unsubscription mechanism and you have to deal with it.
The endpoints can expire or can be invalid (or even malicious, like android.chromlum.info). You need to detect failures (using the HTTP status code, timeouts, etc.) when you try to send the push message from your application server. Then, for some kind of failures (permanent failures, like expiration) you need to delete the endpoint.
What's to stop a user from effectively mounting a denial of service attack by filling my db with endpoints through repeated subscription/unsubscription?
As I described above, you need to properly delete the invalid endpoints, once you realize that they are expired or invalid. Basically they will produce at most one invalid request. Moreover, if you have high throughput, it takes only a few seconds for your server to make requests for thousands of endpoints.
My suggestions are based on a lot of experiments and thinking done when I was developing Pushpad.
Another way is to have a keep alive field on you server and have your service worker update it whenever it receives a push notification. Then regularly purge endpoints which haven't been responded to recently.
Im creating a chat app in ASP.NET MVC3.
im using long polling and AsyncController to do so
when a user posts a chat its saved in database , to retrieve should i constantly check database for change in record or after definite interval
or is there an better/ efficient way of doing it
i came across this question but could not get a usable answer.
You may take a look at SignalR for an efficient way. Contrary to the standard polling mechanism (in which you are sending requests at regular intervals to check for changes), SignalR uses a push mechanism in which the server sends notifications to connected clients to notify them about changes.
Since you're already using long polling and an asynccontrolller, why not create a message pool? Take a look at this solution.
In a nutshell, instead of just writing the updated chat to the database, you should also stick it in some sort of queue. Then each user's async thread is listening to that pool waiting for a message to appear. When one appears return the data to the user through your normal operation. When all listening threads have picked up the message it can be removed from the queue. This will prevent you from having several threads hammering your database looking for a new message.
You can give PServiceBus(http://pservicebus.codeplex.com/) a try and here is a sample web chat app(http://74.208.226.12/ChatApp/chat.html) running and does not need database in between to pass message between two web clients. If you want to persist data in the database for logging sake, you can always subscribe to the chat message and log it to database.
The web site I am developing will be sending tens of thousands of emails daily (and that number will be growing) - registration, notifications, alerts, etc. I will have a dedicated server box that will be actually generating and sending emails by request from the asp.net application (asp.net app calls a WCF method on the email box and provides various parameters for an email).
Now, I am trying to figure out what's the best way of queueing those email jobs on the email server. The call from asp.net app has to be async so that asp.net app doesn't wait for email server to create and send actual email.
Originally I was just creating a worker thread for each email job request but number of emails is going to be really high and I'm not sure if creating hundreds of simultaneous threads is a good idea performance wise. My next thought is to use MSMQ but I'm not sure about its performance and scalability.
Any ideas/production examples?
Thanks!
At a previous job, we had to queue messages for delivery, much like you are explaining. We decided to create a database record that represented each message. At message creation time, we created the mail message in .NET and then saved it into the database. A separate process (Windows service built in .NET) would periodically check to see if there were messages to be sent (delivery date was in the past and status was unsent). It would then re-create the mail message from the information it received from the stored procedure and sent the message along its merry way.
The procedure that returned the messages ready for sending also performed throttling logic based on the day and time of the call (we allowed more of our bandwidth to be used at night and the weekends than during the day).
We also had need for tracking bouncebacks, message opens, and click-throughs which meant having a database record that represented the email was necessary so we could relate events (bounce, open, click) with individual emails and recipients.
I have a data entry and editing form and in every data entry or update event, I have to send an email to a dynamic list of recipients. I have been sending the email as soon as the user clicks the save or edit buttons but am thinking of first saving the data to the database, and then sending the email later. I want to do this partly to improve the response time of the application as the email sending tends to take a long time than desired.
Has any one done some thing some how related to this, is there a better way of implementing something similar or does one know a good tutorial on such.
The email body is html formatted.
You could write a Windows Service that handles sending your emails, then use a Message Queue as the method of passing data from your application to the service. I.e. your applicaiton saves the data, then adds a message to the Queue. The service continually polls the queue for messages, sending each one as an email.
I agree with ck about using a service and a message queue, but there are some alternatives.
One is to use a service that polls the database at a regular interval. This lets you avoid the message queue at the cost of a higher cpu load (the service will do many unnecessary database calls).
You could also do this directly in the database using either a database trigger or a scheduled job in the database. The latest versions of SQL Server supports running stored procedures written in C# or Vb.Net so you could probably reuse much of your existing code here.
Finally you could go for a simple solution where you do the email sending on a separate thread in your asp.net application. This way you avoid the need of a service application and you can reuse your code more or less as it is today.
One way to do this is write to the database, and then put a message on a queue that tells an email service (written as a Windows service) that there are emails to send. The email service then talks to the database to find what it actually needs to do. This decouples the email service from the web application and also avoids polling.
This is slightly different to ck's solution in that the queue message is used as a trigger rather than containing the email information. This decouples the web app and the email service to some extent, and means the email service can be reused by multiple clients without each client having to observe (and keep in step with) the same email message format.
Is there any mail queue concept in ASP.NET?
I want to send thousands of different mail to thousands of users (i.e. each user will have a different mail). I want to send the mail at a particular time, so each user receives it at a constant time.
There really is not mail queue in the Core framework. You can send individual messages synchronously or asynchronously, but you can't really send a bunch at once.
You can queue your messages by storing them to a database or file server and then kicking off a job to loop through your saved messages and send them off.
Also, not all of your users will receive the messages at the same time, even if you could send them at the same time. There are too many external variables and dependencies (network traffic, mail server loads, spam filters) to accurately predicate when or even if your users receive their messages.
There's no native MailQueue concept within .NET framework. The queue will have to be implemented yourself. In your case, you would like the mails for each recipients to be sent at about the same time for all batches. Am I right?
Well, this is a bit tricky. You can use any SMTP server, localhost or external ones. But that also mean although you can dispatch to the SMTP server at a specific time, there's no guarantee it will reach the recipients immediately.
There are a whole bunch of stuff on mail delivery which are not exactly programming related (grey listing, spam filtering etc etc).
The alternative is to have full control on the sending and have your app directly sending the mails to the recipients' mail servers. Well that is workable and I suggest you use a commercial or a good open source component for that. Anyhow, there's still a whole bunch of issues you need to deal with, (e.g. some receiving mail server like Yahoo might block the sending a few times and let it through after a few retries).
I've posted a related question, take a look at the replies here.