The web site I am developing will be sending tens of thousands of emails daily (and that number will be growing) - registration, notifications, alerts, etc. I will have a dedicated server box that will be actually generating and sending emails by request from the asp.net application (asp.net app calls a WCF method on the email box and provides various parameters for an email).
Now, I am trying to figure out what's the best way of queueing those email jobs on the email server. The call from asp.net app has to be async so that asp.net app doesn't wait for email server to create and send actual email.
Originally I was just creating a worker thread for each email job request but number of emails is going to be really high and I'm not sure if creating hundreds of simultaneous threads is a good idea performance wise. My next thought is to use MSMQ but I'm not sure about its performance and scalability.
Any ideas/production examples?
Thanks!
At a previous job, we had to queue messages for delivery, much like you are explaining. We decided to create a database record that represented each message. At message creation time, we created the mail message in .NET and then saved it into the database. A separate process (Windows service built in .NET) would periodically check to see if there were messages to be sent (delivery date was in the past and status was unsent). It would then re-create the mail message from the information it received from the stored procedure and sent the message along its merry way.
The procedure that returned the messages ready for sending also performed throttling logic based on the day and time of the call (we allowed more of our bandwidth to be used at night and the weekends than during the day).
We also had need for tracking bouncebacks, message opens, and click-throughs which meant having a database record that represented the email was necessary so we could relate events (bounce, open, click) with individual emails and recipients.
Related
Scenario
I am building courier service system using Microservices. I am not sure of few things and here is my Scenario
Booking API - This is where customer Place order
Payment API - This is where we process the payment against booking
Notification API - There service is responsible for sending the notification after everything is completed.
The system is using event-driven Architecture. When customer places booking order , i commit local transaction in booking API and publish event. Payment API and notification API are subscribed to their respective event . Once Done Payment and notification API need to acknowledge back to Booking API.
My Questions is
After publishing the event my booking service can't block the call and goes back to the client (front end). How does my client app will have to check the status of transaction or it would know that transaction is completed? Does it poll every couple of seconds ? Since this is distributed transaction and any service can go down and won't be able to acknowledge back . In that case how do my client (front end) would know since it will keep on waiting. I am considering saga for distributed transactions.
What's the best way to achieve all of this ?
Event Sourcing
I want to implement Event sourcing to track the complete track of the booking order. Does i have to implement this in my booking API with event store ? Or event store are shared between services since i am supposed to catch all the events from different services . What's the best way to implement this ?
Many Thanks,
The way I visualize this is as follows (influenced by Martin Kleppmann's talk here and here).
The end user places an order. The order is written to a Kafka topic. Since Kafka has a log structured storage, the order details will be saved in the least possible time. It's an atomic operation ('A' in 'ACID') - all or nothing
Now as soon as the user places the order, the user would like to read it back (read-your-write). To acheive this we can write the order data in a distributed cache as well. Although dual write is not usually a good idea as it may cause partial failure (e.g. writing to Kafka is successful, but writing to cache fails), we can mitigate this risk by ensuring that one of the Kafka consumer writes the data in a database. So, even in a rare scenario of cache failure, the user can read the data back from DB eventually.
The status of the order in the cache as written at the time of order creation is "in progress"
One or more kafka consumer groups are then used to handle the events as follows: the payment and notification are handled properly and the final status will be written back to the cache and database
A separate Kafka consumer will then receive the response from the payment and notification apis and write the updates to cache, DB and a web socket
The websocket will then update the UI model and the changes would be then reflected in the UI through event sourcing.
Further clarifications based on comment
The basic idea here is that we build a cache using streaming for every service with data they need. For e.g. the account service needs feedback from the payment and notification services. Therefore, we have these services write their response to some Kafka topic which has some consumers that write the response back to order service's cache
Based on the ACID properties of Kafka (or any similar technology), the message will never be lost. Eventually we will get all or nothing. That's atomicity. If the order service fails to write the order, an error response is sent back to the client in a synchronous way and the user probably retries after some time. If the order service is successful, the response to the other services must flow back to its cache eventually. If one of the services is down for some time, the response will be delayed, but it will be sent eventually when the service resumes
The clients need not poll. The result will be propagated to it through streaming using websocket. The UI page will listen to the websocket As the consumer writes the feedback in the cache, it can also write to the websocket. This will notify the UI. Then if you use something like Angular or ReactJS, the appropriate section of the UI can be refreshed with the value received at the websocket. Until that happens user keeps seeing the status "in progress" as was written to the cache at the time of order creation Even if the user refreshes the page, the same status is retrieved from the cache. If the cache value expires and follows a LRU mechanism, the same value will be fetched from the DB and wriitten back to the cache to serve future requests. Once the feedback from the other services are available, the new result will be streamed using websocket. On page refresh, new status would be available from the cache or DB
You can pass an Identifier back to client once the booking is completed and client can use this identifier to query the status of the subsequent actions if you can connect them on the back end. You can also send a notification back to the Client when other events are completed. You can do long polling or you can do notification.
thanks skjagini. part of my question is to handle a case where other
microservices don't get back in time or never. lets say payment api is
done working and charged the client but didn't notify my order service
in time or after very long time. how my client waits ? if we timeout
the client the backend may have processed it after timeout
In CQRS, you would separate the Commands and Querying. i.e, considering your scenario you can implement all interactions with Queues for interaction. (There are multiple implementations for CQRS with event sourcing, but in simplest form):
Client Sends a request --> Payment API receives the request --> Validates the request (if validation fails throws error back to the user) --> On successful validation --> generates a GUID and writes the message request to Queue --> passes the GUID to the user
Payment API subscribes the payment queue --> After processing the request --> writes to Order queue or any other queues
Order APi subscribes to Order Queue and processes the request.
User has a GUID which can get him data for all the interactions.
If use a pub/sub as in Kafka instead of Kafka (all other subsequent systems can read from the same topic, you don't need to write for each queue)
If any of the services fail to process, once the services are restarted they should be able to pick where they left off, if the services are down in the middle of a transaction as long as they roll back their resp changes you system should be stable condition
I'm not 100% sure what you are asking. But it sounds like you should be using a messaging service. As #Saptarshi Basu mentioned kafka is good. I would really recommend NATS - although I'm biased because that's the one I work with
With NATS you can create request-reply messages to interface between client and booking service. That's a 1-1 communication
If you have multiple instances of each of your services running, you can use the Queuing service to automatically load balance. NATS will just randomly select a server for you
And then you can use pub-sub feeds for communication between all of your services.
This will give you a very resilient and scalable architecture, and NATS makes it all incredibly easy
Im creating a chat app in ASP.NET MVC3.
im using long polling and AsyncController to do so
when a user posts a chat its saved in database , to retrieve should i constantly check database for change in record or after definite interval
or is there an better/ efficient way of doing it
i came across this question but could not get a usable answer.
You may take a look at SignalR for an efficient way. Contrary to the standard polling mechanism (in which you are sending requests at regular intervals to check for changes), SignalR uses a push mechanism in which the server sends notifications to connected clients to notify them about changes.
Since you're already using long polling and an asynccontrolller, why not create a message pool? Take a look at this solution.
In a nutshell, instead of just writing the updated chat to the database, you should also stick it in some sort of queue. Then each user's async thread is listening to that pool waiting for a message to appear. When one appears return the data to the user through your normal operation. When all listening threads have picked up the message it can be removed from the queue. This will prevent you from having several threads hammering your database looking for a new message.
You can give PServiceBus(http://pservicebus.codeplex.com/) a try and here is a sample web chat app(http://74.208.226.12/ChatApp/chat.html) running and does not need database in between to pass message between two web clients. If you want to persist data in the database for logging sake, you can always subscribe to the chat message and log it to database.
I've a requirement in which i need to send 10000+ mails on a quarterly basis. For this purpose i used a windows service that triggers every day and executes the mailing functionality only after the third month. I've to fetch last three months records from database and need to send one mail for each record.
The problem i faced was the mail server i used do not allow bulk mailing.
How can i do this effectively by providing a delay between each sent (20 mails per minute)?
There are many way to archieve this. We once had a similar requirement and solved it via a home grown service, which would fetch items from a special database table (mail queue) and sent each mail individially. The queue is filled over time by regular business logic. The necessary locking can also be done via db: a SCHEDULE column stores the expected scheduled time of sending the mail. That way the service collects only those mails wich are 'ready' for sending. After successfull send, another column (SENT_TIMESTAMP) is used to mark the success.
We implemented the whole service in ASP and triggered it via regular Windows Task Planner jobs. In your case, the service would start every minute and the queue would provide the next 20 mails. An even easier way could be to utilize SQL Server Jobs. SQL Server is capable of delivering mails to a local SMTP server as well.
If not done yet, please note that SO question as well: What is the best way to send large batches of emails in ASP.NET?
I have a data entry and editing form and in every data entry or update event, I have to send an email to a dynamic list of recipients. I have been sending the email as soon as the user clicks the save or edit buttons but am thinking of first saving the data to the database, and then sending the email later. I want to do this partly to improve the response time of the application as the email sending tends to take a long time than desired.
Has any one done some thing some how related to this, is there a better way of implementing something similar or does one know a good tutorial on such.
The email body is html formatted.
You could write a Windows Service that handles sending your emails, then use a Message Queue as the method of passing data from your application to the service. I.e. your applicaiton saves the data, then adds a message to the Queue. The service continually polls the queue for messages, sending each one as an email.
I agree with ck about using a service and a message queue, but there are some alternatives.
One is to use a service that polls the database at a regular interval. This lets you avoid the message queue at the cost of a higher cpu load (the service will do many unnecessary database calls).
You could also do this directly in the database using either a database trigger or a scheduled job in the database. The latest versions of SQL Server supports running stored procedures written in C# or Vb.Net so you could probably reuse much of your existing code here.
Finally you could go for a simple solution where you do the email sending on a separate thread in your asp.net application. This way you avoid the need of a service application and you can reuse your code more or less as it is today.
One way to do this is write to the database, and then put a message on a queue that tells an email service (written as a Windows service) that there are emails to send. The email service then talks to the database to find what it actually needs to do. This decouples the email service from the web application and also avoids polling.
This is slightly different to ck's solution in that the queue message is used as a trigger rather than containing the email information. This decouples the web app and the email service to some extent, and means the email service can be reused by multiple clients without each client having to observe (and keep in step with) the same email message format.
Is there any mail queue concept in ASP.NET?
I want to send thousands of different mail to thousands of users (i.e. each user will have a different mail). I want to send the mail at a particular time, so each user receives it at a constant time.
There really is not mail queue in the Core framework. You can send individual messages synchronously or asynchronously, but you can't really send a bunch at once.
You can queue your messages by storing them to a database or file server and then kicking off a job to loop through your saved messages and send them off.
Also, not all of your users will receive the messages at the same time, even if you could send them at the same time. There are too many external variables and dependencies (network traffic, mail server loads, spam filters) to accurately predicate when or even if your users receive their messages.
There's no native MailQueue concept within .NET framework. The queue will have to be implemented yourself. In your case, you would like the mails for each recipients to be sent at about the same time for all batches. Am I right?
Well, this is a bit tricky. You can use any SMTP server, localhost or external ones. But that also mean although you can dispatch to the SMTP server at a specific time, there's no guarantee it will reach the recipients immediately.
There are a whole bunch of stuff on mail delivery which are not exactly programming related (grey listing, spam filtering etc etc).
The alternative is to have full control on the sending and have your app directly sending the mails to the recipients' mail servers. Well that is workable and I suggest you use a commercial or a good open source component for that. Anyhow, there's still a whole bunch of issues you need to deal with, (e.g. some receiving mail server like Yahoo might block the sending a few times and let it through after a few retries).
I've posted a related question, take a look at the replies here.