I am writing an application that will need to send a massive amount of emails to our students who will be selected from our database (each email will be personalized to the extent that will include their name, course of study etc...so needs to be sent one at a time).
I could do this looping over an SmtpClient, but I'm afraid that with the numbers I'm trying to send, I'll ultimately run into timeout issues or my thread being killed because of lack of machine resources.
At this point I'm just looking for suggestions of a better way to handle this, or if looping over SmtpClient is an ok solution, how I should go about handling it to prevent what I posted above.
Would a web service be a better alternative?
Please advise,
TIA
My suggestion would be to batch this out. Do not try to run ASP.NET code to create 20K emails and send them out as you are bound to runinto timeout and performance issues. Instead populate a table with recipient, subject, body and begin a batch process from a windows service. This way your code is executed in a way where lag can be managed, instead of having a web page wait for the request to return.
Ok, first - this is hardly massive, I have been handling 50.000 emails+
Let the emails be written to a FOLDER - directory. Then use the local SMTP service you can install, pointing it at the folder as pickup directory. This will at least make sure you have a decent buffer in between (i.e. you can finish the asp.net side, while the emails are not sent at that point).
What about using Database Mail tool provided by SQL Server itself. You can still use asp.net but Email will be sent by SQL Server and all you have to do call the stored procedure and leave things to SQL Server.
I wouldn't recommend batch option since there won't be any optimization or queue thing unless you have to put a lot of effort to do so.
Database Mail tool provides queue and priority options as well. If a problem occurs, the e-mail will be queued again and sent later but the other ones will be still being sent.
It is called Database Mail in SQL Server 2008 and SQL Mail in previous versions.
For more information, please check the links below :
http://blog.sqlauthority.com/2008/08/23/sql-server-2008-configure-database-mail-send-email-from-sql-database/
http://msdn.microsoft.com/en-us/library/ms175887.aspx
You could use javascript on a timer to request the script which sends mail in small chunks that wont time out. Then you'd call the script from the browser. Add a progress bar, authentication, etc.
You don't need a web service, but rather a way to batch the emails into a queue (perhaps an "Email_Queue" table in the DB). Then a Windows service running on a server could read through the queue in a first-in/first-out basis at reasonable chunks at a time being sent to the SMTPClient, removing items from the queue as they are processed. You would probably need to run some measurements to determine what the chunk size and delay would be for your mail server.
Related
After many years of programming, I need to do something asynchronously for the very first time (because it takes several minutes and the web page times out -- don't want the user waiting that long anyway). This action is done by only a few people but could be done a few times per day (for each of them).
From a "Save" click on an ASP.NET web page using LINQ, I'm inserting a record into a SQL Server table. That then triggers an SSIS package to push that record out to several other databases around the country.
So..
How can I (hopefully simply) make this asynchronous so that the user can get on with other things?
Should this be set up on the .NET side or on the SQL side?
Is there a way (minutes later) that the user can know that the process has completed and successfully? Maybe an email? Not sure how else the user can know it finished fine.
I read some threads on this site about it but they were from 2009 so not sure if much different now with Visual Studio 2012/.NET Framework 4.5 (we're still using SQL Server 2008 R2).
It is generally a bad idea to perform long-running tasks in ASP.Net. For one thing, if the application pool is recycled before the task completes, it would be lost.
I would suggest writing the request to a database table, and using a separate Windows Service to do the long-running work. It could update a status column in the database table that could be checked at a later time to see if the task completed or not, and if there was an error.
You could use Service Broker on the SQL side; it'sa SQL Server implementation of Message Queueing.
Good examples here and here
What you do is create a Service Broker service and define some scaffolding (queues, message types, etc).
Then you create a service "Activation" procedure which is basically a stored procedure that consumes messages from queue. This SP would receive for example a message with an ID of a record in a table, and would then go on and do whatever needs to be done to it, perhaps sending an email when it's done, etc.
So from your code-behind, you'd call a simple stored procedure which would insert the user's data into a table, and send a message to the queue with for e.g the ID of the new record, and then immediately return. I suppose you should tell the user upfront that this could take a few minutes and they'll receive an email, etc.
The great thing about Service Broker is message delivery is pretty much guaranteed - even if your SQL Server falls over right after the message is queued, when you bring it back up the activation SP will just kick off again, so it's very robust.
i have to write application for sending newsletter.
what is the best way to send newsletter thoundands of users?
My requirement is
Each mail is seprately as To :
Every mail has unique Unsubscribe link
Is is good to use SMTP mail class of .net?
I look aound may questions in so but can't decide which approcah i should go?
There are many suggestions
Multi threaded Windows service
Use Mail Server
Add thread.sleep(2000) between each send.
can anyone suggest good way to imepement this?
I would not recommend asp.net webpage to send, even if you do start it in a separate background thread. I would think you run the risk of the server recycling your process in the middle of the send, which would mess it up. You really need to write some kind of separate service or application to send your emails.
The simplest option would be to just create a quick and dirty console or windows form application.
Also logging is critical just like the other poster said. If it fails you want to know exactly what got sent out and where it stopped so that when you restart it you don't mail all the people who it did work for again. You want to be able to input the starting point for the send, so if you need to restart at number email #5000 you can.
The classes in System.Net.Mail namespace will work just fine for sending your mail.
One of the biggest problems will be finding a email host that will let you send so many emails. Most email hosts have throttling and sometime it changes depending upon server conditions so if the server is being heavily used then the email limits will be more restrictive, and you may only get to set 500 emails per hour.
We have a newsletter that goes out to around 20000 people as separate emails and we had to play around with the delay between emails until we found one that would work for our email host. We ended up with 1.2 sec between emails, so that might be a good starting point.
I think there are email hosts specialize in bulk mailings though so if you get one of those it might not be a problem.
Also if you host your own email this may not be a problem. And if you do host your own mail you will have the option of dropping the mail in the pickup directory and you could just dump it all in there as fast as you want, and let the email service pick it up at it's own pace.
EDIT: Here is the settings to add to the config file for setting the pickup directory
<system.net>
<mailSettings>
<smtp from="support#test.com" deliveryMethod="SpecifiedPickupDirectory" >
<specifiedPickupDirectory pickupDirectoryLocation="Z:\Path\To\Pickup"/>
</smtp>
</mailSettings>
</system.net>
Definitely do not do this in ASP.NET. This is one of the biggest mistakes that new web developers make.
This needs to be a windows app or service that can handle this much volume.
I've written pages that send emails, but not nearly the volume yours will. Nonetheless, I would recommend the following based on code I have implemented in the past:
Use the web application to write out the email and all the recipient addresses to database table(s).
Have a process that is outside of ASP.NET actually send the emails. This could be a vbs file that is set up as a scheduled task, or (preferably) a windows service. The process would take the text of the email, append the unsubscribe link, and once sent successfully flag the database record as sent. That way, if the send fails, it can try again later (the send process loops over all the records flagged as unsent).
If you need a log of what was sent and when, you just need to keep the sent records in the database tables. Otherwise, just delete the records once sent successfully.
IMHO sending emails within the ASP.NET worker process is a bad idea because you don't know how long it will take and if the send fails there's little opportunity to retry before the page times out.
Create a webpage to "Design" the newsletter in. When they hit Send, queue the newsletter up somewhere (database) and use another program (windows service, etc) to send the queued letter. This will be many times more effecient and potentially fault tolerant if designed properly.
I have written a Newsletter module (as part of a bigger system) in ASPNET MVC 2, Entity Framework and using the System.Net.Mail namespace. It is kicked off in view and actually just runs in a controller with a supporting method to do the send. As each email is sent I track whether there is a hard bouce (an exception is thrown) and I update that database record stating a fail with the exception, otherwise I update the record stating success. We also do personalisation so we have 'tags' that get replaced by an extra field in the database (stored as XML for flexibility). This helps handle an unsubscribe function.
My code is quite simple (please don't flame me for using exception handling as business logic ;) and it works like a charm.
This is all done on a VPS at http://maximumasp.com which also hosts 4 sites with pretty decent traffic. We use their SMTP servers. We notified them that we needed this service and have had no problems relationship-wise.
We had 2GB of RAM on the machine running Windows 2008 and it was doing 6 emails/sec. We bumped it up to 3GB as the web sites needed it and now the mailout is doing about 20emails/sec. Our mailouts range from 2,000 to 100,000 email addresses.
In short, ASP.NET can be used to handle a mailout, and if you add in some logic to handle record updating the worry of losing your way mid-send is mitigated. Yes there are probably slicker ways to do this. We are looking in to MQMS and threading, and separating that out to windows service to make it more stable and scalable as we put more clients and larger lists on, but for now it works just fine with reasonable reporting and error handling.
I have developed a chat web application which uses a SqlServer database for exchanging messages.
All clients poll every x seconds to check for new messages.
It is obvious that this approach consumes many resources, and I was wondering if there is a "cheaper" way of doing that.
I use the same approach for "presence": checking who is on.
Without using a browser plugin/extension like flash or java applet, browser is essentially a one way communication tool. The request has to be initiated by the browser to fetch data. You cannot 'push' data to the browser.
Many web app using Ajax polling method to simulate a server 'push'. The trick is to balance the frequency/data size with the bandwidth and server resources.
I just did a simple observation for gmail. It does a HttpPost polling every 5 seconds. If there's no 'state' change, the response data size is only a few bytes (not including the http headers). Of course google have huge server resources and bandwidth, that's why I mention: finding a good balance.
That is "Improving user experience vs Server resource". You might need to come out with a creative way of polling strategy, instead of a straightforward polling every x seconds.
E.g. If no activity from party A, poll every 3 seconds. While party A is typing, poll every 5 seconds. This is just a illustraton, you can play around with the numbers, or come out with a more efficient one.
Lastly, the data exchange. The challenge is to find a way to pass minimum data sizes to convey the same info.
my 2 cents :)
For something like a real-time chat app, I'd recommend a distributed cache with a SQL backing. I happen to like memcached with the Enyim .NET provider, so I'd do something like the following:
User posts message
System writes message to database
System writes message to cache
All users poll cache periodically for new messages
The database backing allows you to preload the cache in the event the cache is cleared or the application restarts, but the functional bits rely on in-memory cache, rather than polling the database.
If you are using SQL Server 2005 you can look at Notification Services. Granted this would lock you into SQL 2005 as Notification Services was removed in SQL 2008 it was designed to allow the SQL Server to notify client applications of changes to the database.
If you want something a little more scalable, you can put a couple of bit flags on the Users record. When a message for the user comes in change the bit for new messages to true. When you read the messages change it to 0. Same for when people sign on and off. That way you are reading a very small field that has a damn good chance of already being in cache.
Do the workflow would be ready the bit. If it's 1 then go get the messages from the message table. If it's 0 do nothing.
In ASP.NET 4.0 you can use the Observer Pattern with JavaScript Objects and Arrays ie: AJAX JSON calls with jQuery and or PageMethods.
You are going to always have to hit the database to do analysis on whether there is any data to return or not. The trick will be on making those calls small and only return data when needed.
There are two related solutions built-in to SQL Server 2005 and still available in SQL Server 2008:
1) Service Broker, which allows subscribers to post reads on queues (the RECEIVE command with WAIT..). In your case you would want to send your message through the database by using Service Broker Services fronting these Queues, which could then be picked up by the waiting clients. There's no polling, the waiting clients just get activated when a message is received.
2) Query Notifications, which allow a subscriber to define a Query, and the receive notifications when the dataset that would result from executing that query would change. Built on Service Broker, Query Notifications are somewhat easier to use, but may also be somewhat less efficient. (Not that Query Notifications and their siblings, Event Notifications are frequently mistaken for Notification Services (NS), which causes concern because NS is decommitted in 2008, however, Query & Event Notifications are still fully available and even enhanced in SQL Server 2008).
Hai Guys,
My application deals scheduled mail concept (i.e) every morning 6.00 am my users gets a remainder mail about their activities for the day... I dont know how to do this.... Many told use windows service but i will host my website on a shared server i may not get rights to do windows service... Is there any dll for sending mails at a schduled time through asp.net application ..please help me out guys......
You cant do much in a shared hosting. Try upgrading your hosting or else write a windows service, to run on your machine, which will call an asp.net which can send out emails. Of course your machine has to be switched on all the time or at least during 6:00 AM :). You will have to take proper steps to avoid unauthorized request for that aspx page.
you can check this article too: http://www.codeproject.com/KB/aspnet/ASPNETService.aspx
You can't really do this with ASP.Net. ASP.Net is for web pages - which are reactive to HTTP requests.
You need a scheduled task or a service. All a website can do is respond to requests. I guess you could program the functionality into a web page and have a remote process request the page every morning - but what happens if someone else requests the page?
You can either have a program that runs constantly and has a timer or a loop that checks the time of day and then sleeps for a really long time and when the timer goes off or it's the right time of day it sends an email, or you can launch a program as a scheduled task. The first method can also be implemented as a service if you would like. Keep in mind you dont need ASP.Net to send emails, all you need is a console application that uses System.Net.Mail. Check out the mailer sample on MSDN for a very simple idea.
One other thing you can consider: IIS has an smtp service that you can install and it uses a pickup directory to send mail. You write an email to the pickup directory as an .eml file and IIS grabs it and sends it almost immediately. If you do that, you'll still have to write the emails (System.net.Mail will write the .eml files from a MailMessage, just set SmtpClient.DeliveryMethod to SpecifiedPickupDirectory or PickupDirectoryFromIIS and call SmtpClient.Send) but it will then send them for you. You'll still need to schedule something somehow so this might not be all that more useful but I thought I'd at least let you know that it exists.
One thing to be aware of: when the IIS SMTP service reads the send envelope of the .eml file, the order of the Sender and From headers is significant; if the From header appears before the Sender header then the MAIL FROM command will use the From header, which is incorrect (and MS won't be fixing this one). This appears to be an issue ONLY with the IIS SMTP service as it hasn't been reported anywhere else that I'm aware of. Reversing the order of the headers is the work-around. By default SmtpClient always writes the From header first. I'm aware of the issue and IIS isn't fixing it but I may be able to get a fix into SmtpClient for the .NET 4.0 RC build that re-orders the headers for you but no promises.
If you happen to have it handy (and I assume you do), you can use a SQL Server Agent job to make a request to an ASP.NET page that sends the email.
Here's some example code:
http://nicholasclarke.co.uk/blog/2008/01/16/web-request-from-sql-server-via-c/
Of course, since you're using SQL Server to call CLR code anyway, you could just have that code send out the emails (via System.Net.Mail) rather than requesting a page on IIS to do so. To do this, SQL Server would need:
Access to all of the data needed to send the emails
Outbound firewall access to send an email
CLR code that encapsulates all of the logic needed to know where/what to send.
Okay this is interesting, and what I did fits silky's definite of 'cheating', but no it was pretty cool for me.
What I did was spawn a new thread from ASP.Net code (it was possible on that host), and that thread did the scheduled job.
I checked whether the thread was alive (which is pretty easy) on every visit to the website (not so reliable I know, but it worked cause that website has plenty of visitor).
If at all you do this
Treat this as a stop-gap while you arrange to get a dedicated host or VPS.
Rest assured that the hosting company will kill your thread and withdraw permissions when they discover you're doing this.
I'm writing an application where the user will create an appointment, and instantly get an email confirming their appointment. I'd also like to send an email the day of their appointment, to remind them to actually show up.
I'm in ASP.NET (2.0) on MS SQL . The immediate email is no problem, but I'm not sure about the best way to address the reminder email. Basically, I can think of three approaches:
Set up a SQL job that runs every night, kicking off SQL emails to people that have appointments that day.
Somehow send the email with a "do not deliver before" flag, although this seems like something I might be inventing.
Write another application that runs at a certain time every night.
Am I missing something obvious? How can I accomplish this?
Choice #1 would be the best option, create a table of emails to send, and update the table as you send each email. It's also best not to delete the entry but mark it as sent, you never know when you'll have a problem oneday and want to resend out emails, I've seen this happen many times in similar setups.
One caution - tightly coupling the transmission of the initial email in the web application can result in a brittle architecture (e.g. SMTP server not available) - and lost messages.
You can introduce an abstraction layer via an MSMQ for both the initial and the reminder email - and have a service sweeping the queue on a scheduled basis. The initial message can be flagged with an attribute that means "SEND NOW" - the reminder message can be flagged as "SCHEDULED" - and the sweeper simply needs to send any messages that it finds that are of the "SEND NOW" or that are "SCHEDULED" and have a toBeSentDate >= the current date. Once the message is successfully sent - the unit of work can be concluded by deleting the message from the queue.
This approach ensures messages are not lost - and enables the distribution of load to off-peak hours by adjusting the service polling interval.
As Rob Williams points out - my suggestion of MSMQ is a bit of overkill for this specific question...but it is a viable approach to keep in mind when you start looking at problems of scale - and you want (or need) to minimize/reduce database read/write activity (esepcially during peak processing periods).
Hat tip to Rob.
For every larger project I usually also create a service which performs regular or periodical tasks.
The service updates its status and time of last execution somewhere in the database, so that the information is available for applications.
For example, the application posts commands to a command queue, and the service processes them at the schedule time.
I find this solution easier to handle than SQL Server Tasks or Jobs, since it's only a single service that you need to install, rather than ensuring all required Jobs are set up correctly.
Also, as the service is written in C#, I have a more powerful programming language (plus libraries) at hand than T-SQL.
If it's really pure T-SQL stuff that needs to be handled, there will be a Execute_Daily stored procedure that the service is going to call on date change.
Create a separate batch service, as others have suggested, but use it to send ALL of the emails.
The web app should record the need to send notifications in a database table, both for the immediate notice and for the reminder notice, with both records annotated with the desired send date/time.
Using MSMQ is overkill--you already have a database and a simple application. As the complexity grows, MSMQ or something similar might help with that complexity and scalability.
The service should periodically (every few minutes to a few hours) scan the database table for notifications (emails) to send in the near future, send them, and mark them as sent if successful. You could eventually leverage this to also send text messages (SMS) or instant messages (IMs), etc.
While you are at it, you should consider using the Command design pattern, and implement this service as a reusable Command executor. I have done this recently with a web application that needs to keep real estate listing (MLS) data synchronized with a third-party provider.
Your option 2 certainly seems like something you are inventing. I know that my mail system won't hold messages for future delivery if you were to send me something like that.
I don't think you're missing anything obvious. You will need something that runs the day of the appointment to send emails. Whether that might be better as a SQL job or as a separate application would be up to your application architecture.
I would recommend the first option, using either an SQL or other application to run automatically every day to send the e-mails. It's simple, and it works.
Microsoft Office has a delivery delay feature, but I think that is an Outlook thing rather than an Exchange/Mail Server thing, so you're going to have to go with option 1 or 3. Or option 4 would be to write a service. That way you won't have to worry about scheduled tasks to get the option 3 application to run.
If you are planning on having this app hosted at a cheap hosting service (like GoDaddy), then what I'd recommend is to spin off a worker thread in Global.asax at Application_Start and having it sleep, wake-up, send emails, sleep...
Because you won't be able to run something on the SQL Server machine, and you won't be able to install your own service.
I do this, and it works fine.