Multiple input queues in one rebus process - rebus

I'm writing a "multi-workers" application using top-shelf and rebus.
My idea is to use the MyWorker1Namespace - MyWorker1Namespace.Messages, MyWorker2Namespace - MyWorker2Namespace.Messages pattern.
I would like to run the application without spanning multiple process, instead I would like to configure the application with moultiple input queues in order to be ready to split it up to multiple processes if necessary.
Is there any way to declare multiple input queues and multiple worker threads in one application using Rebus?
I guess configuration should be something like this:
<rebus inputQueue="MyWorker1Namespace.input" errorQueue="MyWorker1Namespace.error" workers="1" maxRetries="5"> </rebus>
<rebus inputQueue="MyWorker2Namespace.input" errorQueue="MyWorker2Namespace.error" workers="1" maxRetries="5"> </rebus>
...

Since the Rebus app.config XML is kind of optimized for one-bus-instance-per-process scenarios, you cannot configure multiple buses fully in XML, but there's nothing that keeps you from starting multiple bus instances inside the same process.
I've often done that e.g. in Azure worker roles where I want to host multiple logical endpoints without incurring the cost of physically separate deployments, and I've also sometimes done it with Topshelf-hosted Windows Services.
Usually, my app.config ends up with the following XML:
<rebus workers="1">
<add messages="SomeAssembly.Messages" endpoint="someEndpoint.input"/>
<add messages="AnotherAssembly.Messages" endpoint="anotherEndpoint.input"/>
</rebus>
thus allowing me to configure the default number of workers per bus and the endpoint mappings once and for all. Then, when my application starts up, it will keep an IoC container per bus for the duration of the application lifetime - with Windsor, I usually end up with a general bus installer that has parameters for the queue names, which allows me to configure Windsor like this:
var containers = new List<IWindsorContainer> {
new WindsorContainer()
// always handlers first
.Install(FromAssembly.Containing<SomeEndpoint.SomeHandler>())
// and then the bus, which gets started at this point
.Install(new RebusInstaller("someEndpoint.input", "error"))
// and then e.g. background timers and other "living things"
.Install(new PeriodicTimersInstannce()),
new WindsorContainer()
.Install(FromAssembly.Containing<AnotherEndpoint.AnotherHandler>())
.Install(new RebusInstaller("anotherEndpoint.input", "error"))
};
// and then remember to dispose each container when shutting down the process
where the RebusInstaller (which is a Windsor mechanism) basically just puts a bus with the right queue names into the container, e.g. like this:
Configure.With(new WindsorContainerAdapter(container))
.Transport(t => t.UseMsmq(_inputQueueName, _errorQueueName))
.(...) etc
.CreateBus().Start();
I like the idea that each IoC container instance functions as a logically independent application on its own. This way it would be easy to break things apart some time in the future if you want to e.g. be able to deploy your endpoints independently.
I hope this provides a bit of inspiration for you :) please don't hesitate to ask if you need more pointers.

Related

Raft.Next - persistent cluster configuration fails when running multiple processes

I'm currently investigating Raft in dotNext and would like to move from the fairly simplistic example which registers all the nodes in the cluster at startup to using an announcer to notify the leader when a new node has joined.
To my understanding this means that I should start the initial node in ColdStart but then subsequent nodes should use the ClusterMemberAnnouncer to add to the cluster as:
services.AddTransient<ClusterMemberAnnouncer<UriEndPoint>>(serviceProvider => async (memberId, address, cancellationToken) =>
{
// Register the node with the configuration storage
var configurationStorage = serviceProvider.GetService<IClusterConfigurationStorage<UriEndPoint>>();
if (configurationStorage == null)
throw new Exception("Unable to resolve the IClusterConfigurationStorage when adding the new node member");
await configurationStorage.AddMemberAsync(memberId, address, cancellationToken);
});
It makes sense to me that the nodes should use a shared/persisted configuration storage so that when the second node tries to start up and announce itself, it's able to see the first cold-started active node in the cluster. However if I use the documented services.UsePersistentConfigurationStorage("configurationStorage") approach and then run the nodes in separate console windows ie. separate processes, the second node understandably says:
The process cannot access the file 'C:\Projects\RaftTest\configurationStorage\active.list' because it is being used by another process.
Has anyone perhaps got an example of using an announcer in Raft dotnext?
And does anyone know the best way (hopefully with an example) to use persistent cluster configuration storage so that separate processes (potentially running in different docker containers) are able to access the active list?

Spring Kafka batch within time window

Spring Boot environment listening to kafka topics(#KafkaListener / #StreamListener)
Configured the listener factory to operate in batch mode:
ConcurrentKafkaListenerContainerFactory # setBatchListener
or via application.properties:
spring.kafka.listener.type=batch
How to configure the framework so that given two numbers: N and T, it will try to fetch N records for the listener but won't wait more than T seconds, like described here: https://doc.akka.io/docs/akka/2.5/stream/operators/Source-or-Flow/groupedWithin.html
Some properties I've looked at:
max-poll-records ensures you won't get more than N numbers in a batch
fetch-min-size get at least this amount of data in a fetch request
fetch-max-wait but don't wait more than necessary
idleBetweenPolls just sleep a bit between polls :)
It seems like fetch-min-size combined with fetch-max-wait should do it but they compare bytes, not messages/records.
It is obviously possible to implement that by hand, I'm looking whether it's possible to configure Spring to to that for me.
It seems like fetch-min-size combined with fetch-max-wait should do it but they compare bytes, not messages/records.
That is correct, unfortunately, Kafka provides no mechanism such as fetch.min.records.
I don't anticipate that Spring would layer this functionality on top of the kafka-clients; it would be better to ask for a new feature in Kafka itself.
Spring does not manipulate the records returned from the poll at all, except you can now specify subBatchPerPartition to get batches containing just one partition in order to properly support zombie fencing when using exactly once read/prcess/write.

How to deduplicate events when using RabbitMQ Publish/Subscribe Microservice Event Bus

I have been reading This Book on page 58 to understand how to do asynchronous event integration between microservices.
Using RabbitMQ and publish/subscribe patterns facilitates pushing events out to subscribers. However, given microservice architectures and docker usage I expect to have more than once instance of a microservice 'type' running. From what I understand all instances will subscribe to the event and therefore would all receive it.
The book doesn't clearly explain how to ensure only one of the instances handle the request.
I have looked into the duplication section, but that describes a pattern that explains how to deduplicate within a service instance but not necessarily against them...
Each microservice instance would subscribe using something similar to:
public void Subscribe<T, TH>()
where T : IntegrationEvent
where TH : IIntegrationEventHandler<T>
{
var eventName = _subsManager.GetEventKey<T>();
var containsKey = _subsManager.HasSubscriptionsForEvent(eventName);
if (!containsKey)
{
if (!_persistentConnection.IsConnected)
{
_persistentConnection.TryConnect();
}
using (var channel = _persistentConnection.CreateModel())
{
channel.QueueBind(queue: _queueName,
exchange: BROKER_NAME,
routingKey: eventName);
}
}
_subsManager.AddSubscription<T, TH>();
}
I need to understand how a multiple microservice instances of the same 'type' of microservice can deduplicate without loosing the message if the service goes down while processing.
From what I understand all instances will subscribe to the event and
therefore would all receive it.
Only one instance of subscriber will process the message/event. When you have multiple instances of a service running and subscribed to same subscription the first one to pick the message will set the message invisible from the subscription (called visibility timeout). If the service instance is able to process the message in given time it will tell the queue to delete the message and if it's not able to process the message in time , the message will re-appear in queue for any instance to pick it up again.
All standard service bus (rabbitMQ, SQS, Azure Serivce bus etc) provide this feature out of box.
By the way i have read this book and used the above code from eShotContainers and it works the way i described.
You should look into following pattern as well
Competing Consumers pattern
Hope that helps!

Defer Messages and Specify Queue

I have been playing around with Rebus and RabbitMQ, and came across a scenario I cannot seem to get working.
I have a couple of queues; queue1 & queue2 and they take the same class/message type. Now, Rebus seems to prefer different message types per queue, this is not an option for me right now, so i use the advanced routing bus.Advanced.Routing.Send("queue1", Message)
I would like to utilise the bus.defer functionality but am unsure how to combine them both. I know I might need to introduce a waiting queue as an external timeout manager (which I have yet to get working too, but thats for another day)
Has anyone done anything similar?
How to send the message
As you have probably discovered, when you bus.Defer, Rebus will use the endpoint mappings to look up the destination queue from the type of the message being deferred (which is analogous to bus.Send/bus.SendLocal, in that it has an accompanying bus.DeferLocal too, which always sends to the sender's own input queue).
What is missing, is something analogous to bus.Advanced.Routing.Send, but fortunately it is pretty easy to emulate a combination of bus.Defer and an explicitly routed message but setting the rbs2-deferred-recipient header on a message:
var headers = new Dictionary<string, string> {
{Headers.DeferredRecipient, "destination-queue"}
};
var delay = TimeSpan.FromMinutes(5);
await bus.DeferLocal(delay, yourMessage, headers);
How to configure the timeout manager
You can use Rebus' internal timeout manager by configuring some kind of timeout persistence – e.g. by pulling in Rebus.SqlServer and using SQL Server to store timeouts like so:
Configure.With(...)
.(...)
.Timeouts(t => t.StoreInSqlServer(...))
.Start();
Another option is to install a Rebus endpoint as a dedicated timeout manager, which simply uses the same configuration as can be seen above, and then all other endpoints do this:
Configure.With(...)
.(...)
.Timeouts(t => t.UseExternalTimeoutManager("timeouts"))
.Start();
assuming that your timeout manager uses the timeouts queue.
Update relevant from Rebus 5
Rebus 5 (which is currently available as a prerelease package on Nuget.org) has builtin support for deferring messages to an explicitly specified destination queue.
It can be done like this:
var delay = TimeSpan.FromMinutes(2);
await bus.Advanced.Routing.Defer("dest-queue", delay, message);
which will simply carry out the steps mentioned above underneath the covers.

a service which would be able to run jobs on a timed basis

I am working for my client using Asp.net webAPI2 and angularJS. Now my client have following requirement,but i am unable to understand what type of project i have to create like WebAPI project,window service or any other? Anyone please tell me what the client actually want and how can i do it?
QueueManager will need to be some kind of a service which would be able to run jobs on a timed basis. We envision it being a service that runs on a continuous loop, but has a Thread.Sleep at the end of each iteration with a duration of x-seconds (“x” being set in a config file.) You should create this QueueManager service as a new project within the Core.Jobs project; I would like to have the project name be “Core.Jobs.QueueManager”, along with the base namespace.
Here are the functions that the QueueManager will do for each iteration:
1) Do a worker healthcheck (JobsAPI: Queue/WorkerHealthCheck – already created)
a. This method will just return a 200 status code, and a count of workers. Not need to act on the return value.
Look at Hangfire, it is quite easy to set up and simple to use.
http://docs.hangfire.io/en/latest/background-methods/performing-recurrent-tasks.html

Resources