Problem
Consider a social networking website supporting the following actions:
MediaUploadService: You can upload media files (images and videos) either a single file or multiple files.
TaggingService: Once the files are uploaded, all the persons identified in the files are tagged automatically.
NotificationService: Once the files are tagged, all the persons get notified.
The following requirements must be satisfied:
The user can cancel the upload at any time which means the uploading should also be stopped. It also means that the tagging and notification services should not even be triggered for such requests.
All the services should be able to retry the failed jobs.
All the services communicate through messaging infrastructure.
The services must be scalable and available.
My Take
We can have a global task queue and the upload service can listen for new jobs. The request can be represented as:
{
"request_id":"abcd-defg-pqrs",
"total_files": 2,
"files":[
{
"id":"bcde-efgh-qrst",
"name":"cat.jpg",
"type":"image"
},
{
"id":"cdef-fghi-rstu",
"name":"kitty.mp4",
"type":"video"
}
]
}
The request is broken into the single file upload request and is pushed to the upload-request message queues:
{
"request_id":"abcd-defg-pqrs",
"total_files": 2,
"file":{
"id":"bcde-efgh-qrst",
"name":"cat.jpg",
"type":"image"
}
}
Each request is picked and processed as a background job and the response is sent to the upload-response aggregator which keeps count of the total files uploaded:
{
"request_id":"abcd-defg-pqrs",
"total_files": 2,
"uploaded_files": 1,
"file":[
"bcde-efgh-qrst"
]
}
Once all the files are uploaded the final response is sent to the tagging-request message queues:
{
"request_id":"abcd-defg-pqrs",
"total_files": 2,
"files":[
"bcde-efgh-qrst",
"cdef-fghi-rstu"
]
}
When the tagging service is done with the job, it sends the request to the notification-request message queues. Finally, once we have all the tasks completed, the user can be notified about it using global-response message queues.
Concerns
For retry failed jobs, we can have other low-priority queues for each of the services. What if we want to give the same priority and process with retries in a real quick time as well?
Processing the jobs respecting the dependencies on the services, i.e., upload → tag → notify is taken care of using the messaging queues. Is there any better way to achieve the same?
How can we immediately stop files uploading (assuming file upload is still in progress by the time we are making a cancellation request)? For the uploaded files, we can simply go ahead and delete the files.
Look at temporal.io which provides a much better way to model such use cases. It is essentially a workflow engine that uses code without any intermediate representation. Cancellation and compensations are supported out of the box.
Related
We have a Multitrack web conference implementation using AMS 2.4.1 version. Its working great for our use case, except in one scenario. When there are N (< 3) number of users and they on there camera simultaneously, then few remote users are not rendered as we don't receive the video tracks for those users in newStreamAvailable. We only receive the audio track for those users. We are able to reproduce this quite frequently.
As a backup, I am trying to poll AMS using getTrackList with the main track Id to get all available streams, but I am not getting any message trackList
var jsCmd =
{
command : "getTrackList",
streamId : streamId, // this is roomId or main track id
token : token
}
Any insight would be helpful.
Thanks,
We were able to resolve the issue, posting here to help anyone who might be facing a similar issue.
With push notifications from the server, we might encounter issues when for some reason push operation doesn't succeed. In that case, it's better to have a backup plan to pull from the server and sync.
The Ant Media Server suggests pulling the server periodically for the room info. The server will respond with active streams and the application should synchronize.
For reference, please refer to following link https://resources.antmedia.io/docs/webrtc-websocket-messaging-reference
I have been reading This Book on page 58 to understand how to do asynchronous event integration between microservices.
Using RabbitMQ and publish/subscribe patterns facilitates pushing events out to subscribers. However, given microservice architectures and docker usage I expect to have more than once instance of a microservice 'type' running. From what I understand all instances will subscribe to the event and therefore would all receive it.
The book doesn't clearly explain how to ensure only one of the instances handle the request.
I have looked into the duplication section, but that describes a pattern that explains how to deduplicate within a service instance but not necessarily against them...
Each microservice instance would subscribe using something similar to:
public void Subscribe<T, TH>()
where T : IntegrationEvent
where TH : IIntegrationEventHandler<T>
{
var eventName = _subsManager.GetEventKey<T>();
var containsKey = _subsManager.HasSubscriptionsForEvent(eventName);
if (!containsKey)
{
if (!_persistentConnection.IsConnected)
{
_persistentConnection.TryConnect();
}
using (var channel = _persistentConnection.CreateModel())
{
channel.QueueBind(queue: _queueName,
exchange: BROKER_NAME,
routingKey: eventName);
}
}
_subsManager.AddSubscription<T, TH>();
}
I need to understand how a multiple microservice instances of the same 'type' of microservice can deduplicate without loosing the message if the service goes down while processing.
From what I understand all instances will subscribe to the event and
therefore would all receive it.
Only one instance of subscriber will process the message/event. When you have multiple instances of a service running and subscribed to same subscription the first one to pick the message will set the message invisible from the subscription (called visibility timeout). If the service instance is able to process the message in given time it will tell the queue to delete the message and if it's not able to process the message in time , the message will re-appear in queue for any instance to pick it up again.
All standard service bus (rabbitMQ, SQS, Azure Serivce bus etc) provide this feature out of box.
By the way i have read this book and used the above code from eShotContainers and it works the way i described.
You should look into following pattern as well
Competing Consumers pattern
Hope that helps!
Is it possible to batch Rebus messages (using Azure Servicebus) ?
The reason is that we are going to send a lot of message to save log events and want to batch up.
While old versions of Rebus did have a batch API for wrapping up multiple logical messages inside one single transport message, that functionality turned out to bring very little advantage at the expense of increased complexity in many places.
If you want to send batches of messages, I suggest you simply code your own message batch message, something like
public class BatchOfLogEvents
{
public BatchOfLogEvents(IEnumerable<LogEvent> logEvents)
{
LogEvents = logEvents.ToArray();
}
public IReadOnlyCollection<LogEvent> LogEvents { get; }
}
and then you send that and create a handler for it in the other end.
Update regarding Azure Service Bus: Please remember that Azure Service Bus has a 256 kB maximum message size (or 1MB if you're on Premium).
Also: If you have not done so already, you can probably benefit from enabling GZip compression of messages by going
.Options(o => o.EnableCompression())
in your Rebus configurations.
I am trying to implement delayed queue with overriding of messages using Active MQ.
Each message is scheduled to be delivered with delay of x (say 60 seconds)
In between if same message is received again it should override previous message.
So even if I receive 10 messages say in x seconds. Only one message should be processed.
Is there clean way to accomplish this?
The question has two parts that need to be addressed separately:
Can a message be delayed in ActiveMQ?
Yes - see Delay and Schedule Message Delivery. You need to set <broker ... schedulerSupport="true"> in your ActiveMQ config, as well as setting the AMQ_SCHEDULED_DELAY property of the JMS message saying how long you want the message to be delayed (10000 in your case).
Is there any way to prevent the same message being consumed more than once?
Yes, but that's an application concern rather than an ActiveMQ one. It's often referred to as de-duplication or idempotent consumption. The simplest way if you only have one consumer is to keep track of messages received in a map, and check that map whether you receive a message. It it has been seen, discard.
For more complex use cases where you have multiple consumers on different machines, or you want that state to survive application restart, you will need to keep a table of messages seen in a database, and query it each time.
Please vote this answer up if it helps, as it encourages people to help you out.
Also according to method from ActiveMQ BrokerService class you should configure persistence to have ability to use scheduler functionality.
public boolean isSchedulerSupport() {
return this.schedulerSupport && (isPersistent() || jobSchedulerStore != null);
}
you can configure activemq broker to enable "schedulerSupport" with the following entry in your activemq.xml file located in conf directory of your activemq home directory.
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" schedulerSupport="true">
You can Override the BrokerService in your configuration
#Configuration
#EnableJms
public class JMSConfiguration {
#Bean
public BrokerService brokerService() throws Exception {
BrokerService brokerService = new BrokerService();
brokerService.setSchedulerSupport(true);
return brokerService;
}
}
I've set up a windows service to do the bulk mailing functionality which executes it batch by batch.
The service fetches one batch from DB during its schedule and after each batch i've given a delay of 20 seconds.
Do this prevent from considering the mails as spam or as bulk? If so do my code performs what i require. My code is as follows:
//get the batch and execute in a child thread and need to continue only after the thread get terminated.
for (int i = 0; i <= QueueCount/20;i++)
{
Thread newThread = new Thread(ProcessMailQueue);
newThread.Start();
while(!newThread.IsAlive);
Thread.Sleep(1);
newThread.Join();
}
//delay after each batch execution
private void ProcessMailQueue()
{
send the full mails in a batch
Thread.Sleep(20000);
}
Any one please give your suggestion....
AFAIK, most of the time, spam detection happens at recipient side i.e. at mail client or service/SMTP managing recipient mailboxes. Its unlikely to happen at originating SMTP (because these days, SMTP needs authentication and/or they does not support relaying so does not need to detect spam).
Regardless, spam detection happens based on many different parameters such as number of recipient (to/cc/bcc), mail contents, a list of know spammer SMTP getaways etc so you have to consider that. Typically, email messages with legitimate content, subject filled, having a valid form, and few recipients (with To specified) are unlikely considered as a spam.