I'm using MassTransit with RabbitMQ on a .net core 6 web application. My goal is to keep in sync several instances of an application, running on different plants. The application needs to be able to publish / consume messages.
When a site publishes something, this is broadcasted to all the sites queues (also itself, it will simply discard the message).
In order to do it, I configured MassTransit queue names with the suffix of the plant: eg norm-queue-CV, norm-queue-MB. I configured also the Consumer to bind to a generic fanout exchange name (norm-exchange).
Here my configuration extract:
public void ConfigureServices(IServiceCollection services)
{
services.AddMassTransit(x =>
{
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(new Uri(_configuration["RabbitMQ:URI"] + _configuration["RabbitMQ:VirtualHost"]), $"ENG {_configuration["Application:PlantID"]} Producer", h =>
{
h.Username(_configuration["RabbitMQ:UserName"]);
h.Password(_configuration["RabbitMQ:Password"]);
});
cfg.Publish<NormCreated>(x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "fanout"; // default, allows any valid exchange type
});
}));
});
// consumer
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(new Uri(_configuration["RabbitMQ:URI"] + _configuration["RabbitMQ:VirtualHost"]), $"ENG {_configuration["Application:PlantID"]} Consumer", h =>
{
h.Username(_configuration["RabbitMQ:UserName"]);
h.Password(_configuration["RabbitMQ:Password"]);
});
cfg.ReceiveEndpoint($"norm-queue-{_configuration["Application:PlantID"]}", e =>
{
e.Consumer<NormConsumer>();
e.UseConcurrencyLimit(1);
e.UseMessageRetry(r => r.Intervals(100, 200, 500, 800, 1000));
e.Bind<NormCreated>();
e.Bind("norm-exchange");
});
});
busControl.Start();
And here how NormConsumer is defined
public class NormConsumer : IConsumer<NormCreated>
{
private readonly ILogger<NormConsumer>? logger;
public NormConsumer()
{
}
public NormConsumer(ILogger<NormConsumer> logger)
{
this.logger = logger;
}
public async Task Consume(ConsumeContext<NormCreated> context)
{
logger.LogInformation("Norm Submitted: {NormID}", context.Message.NormID);
//await context.Publish<NormCreated>(new
//{
// context.Message.OrderId
//});
}
}
Here the queues automatically created. To me they look fine
And here the exchange created. I was trying to get only one exchange (norm-exchange), but also the other 2 are created.
My problem is first of all understand if my layout makes sense (I'm quite new to Rabbit/Masstransit).
Moreover I'd like to override how exchanges are named, forcing to have for this queues only one exchange: "norm-exchange". I tried to override it in "producer" part, but not able to do it
RabbitMQ broker topology is covered extensively in RabbitMQ - The Details, and also in the documentation.
You do not need to call Bind in the receive endpoint, consumer message types are already bound for you. Remove both Bind statements, and any published messages will be routed by type to the receive endpoints.
Related
We configure MassTransit to use Azure Service Bus in this way:
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(new HostSettings
{
ServiceUri = new Uri(xxx),
TokenProvider = TokenProvider.CreateManagedIdentityTokenProvider()
});
busConfig.ConfigureJsonSerializer(ConfigureJsonSerialization);
busConfig.ConfigureJsonDeserializer(ConfigureJsonSerialization);
busConfig.ConfigureEndpoints(context);
});
How can we set e.g. subscription properties like EnableDeadLetteringOnMessageExpiration for all the subscriptions created automatically by MassTransit?
Thanks,
Peter
Update
I've tried this (EnableDeadLetteringOnMessageExpiration), but the dead letter option isn't enabled on the subscriptions in the Azure Service Bus (we've deleted all the topics and subscriptions first, so that they were newly created):
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(new HostSettings
{
ServiceUri = new Uri(xxx),
TokenProvider = TokenProvider.CreateManagedIdentityTokenProvider()
});
busConfig.EnableDeadLetteringOnMessageExpiration = true;
busConfig.ConfigureJsonSerializer(ConfigureJsonSerialization);
busConfig.ConfigureJsonDeserializer(ConfigureJsonSerialization);
busConfig.ConfigureEndpoints(context);
});
You can create a class that implements IConfigureReceiveEndpoint (see the docs) and in that function, pattern match the configurator to see if it is Azure Service Bus and set the properties. When registered in the container, MassTransit will run the class against each endpoint.
class ConfigureMyEndpoint :
IConfigureReceiveEndpoint
{
public void Configure(string name, IReceiveEndpointConfigurator configurator)
{
if(configurator is IServiceBusReceiveEndpointConfigurator sb)
{
sb.EnableDeadLetteringOnMessageExpiration = true;
}
}
}
I had a problem sending messages to clients via MassTransit and SignalR
Startup:
//SignalR
services.AddSignalR().AddMassTransitBackplane();
#region MassTransit RabbitMq
services.AddScoped<SendCosistListToScaleConsumer>();
services.AddScoped<CreateConsistListConsumer>();
services.AddMassTransit(x =>
{
x.AddSignalRHubConsumers<NotifyHub>();
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(conf =>
{
conf.Host(Configuration["Rabbit:Host"], host => {
host.Username(Configuration["Rabbit:Username"]);
host.Password(Configuration["Rabbit:Password"]);
});
conf.ReceiveEndpoint(Configuration["Rabbit:ReceiveEndpoint"], e => {
e.PrefetchCount = 16;
e.UseMessageRetry(n => n.Interval(3, 100));
#region Consumers
e.Consumer<SendCosistListToScaleConsumer>();
e.Consumer<CreateConsistListConsumer>();
#endregion
});
conf.AddSignalRHubEndpoints<NotifyHub>(provider);
}));
});
services.AddMassTransitHostedService();
#endregion
....
app.UseSignalR(endpoints =>
{
endpoints.MapHub<NotifyHub>("/notify");
});
Consumer:
public class CreateConsistListConsumer : IConsumer<ICreateConsistList>
{
IReadOnlyList<IHubProtocol> protocols = new IHubProtocol[] { new JsonHubProtocol() };
public Task Consume(ConsumeContext<ICreateConsistList> context)
{
context.Publish<All<NotifyHub>>(
new
{
Message = protocols.ToProtocolDictionary("SendMessageToAllUsers", new object[] { "CompanyId", context.Message.CompanyId })
});
return Task.CompletedTask;
}
}
Console App (SignalR Client):
hubConnection.On<Object>("SendMessageToAllUsers", param => {
Console.WriteLine(param);
});
If I understand correctly how MassTransii and SignalR work, then this code is enough to send messages to clients.
With the help of debugging, I looked that CreateConsistListConsumer is working, but clients do not receive reporting.
At the same time, the client connects to the hub and correctly receives messages from other sources, but not from MassTransit.
What am I doing wrong?
I have been facing the same issue last week.
It seem SignalR is doing some special work with handling hubs, and couldn't make Masstransit SignalR service to work.
I ended up using a static hub reference as described here.
Basically, I am just calling Core DI to get my hub context, then store it into a static property (as in the sample in the Github issue listed above).
When needed, I call the reference from within my MassTransit Consumer, and I am done.
The app below is tweaked to work with Azure Service Bus.
https://github.com/rebus-org/RebusSamples/tree/master/PubSub
However, below are created.
Queues
error
publisher
subscriber
Topics
messages_datetimemessage__messages: subscriber
messages_stringmessage__messages: subscriber
messages_timespanmessage__messages: subscriber
My question
is that is this correct?
And is it possible to reduce the number of artifacts that are created? For example, reduce to only one topic, because topic is used for pub sub.
Update
I need to use Pub sub pattern with one Topic and one or two subscriptions if possible.
However, I have got an error below:
System.AggregateException HResult=0x80131500 Message=One or more
errors occurred. (Could not publish to topic 'order')
Source=System.Private.CoreLib StackTrace: at
System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout,
CancellationToken cancellationToken) at
System.Threading.Tasks.Task.Wait() at Publisher.Program.Main() in
C:_MyLab\ReBus\PubSub\Publisher\Program.cs:line 43
Inner Exception 1: RebusApplicationException: Could not publish to
topic 'order'
Inner Exception 2: InvalidOperationException: Cannot open a Topic
client for entity type Queue.
TrackingId:5c380af2-ad8f-4788-85b8-5427dd7873e4_B4,
SystemTracker:myapp:Queue:order, Timestamp:2019-04-29T22:31:57
TrackingId:9c3e0c40-4410-4102-a705-86a6528cd030_B4,
SystemTracker:myapp:Queue:order, Timestamp:2019-04-29T22:31:57
TrackingId:401a15d284ad44989f5e451c963d81e5_G16,
SystemTracker:gateway7, Timestamp:2019-04-29T22:31:57
UseAzureServiceBus seems wrong, because it is using queue
class Publisher
{
static void Main()
{
using (var activator = new BuiltinHandlerActivator())
{
Configure.With(activator)
.Logging(l => l.ColoredConsole(minLevel: LogLevel.Warn))
.Transport(t => t.UseAzureServiceBus(Consts.ServiceBusConnectionString, Consts.Order))
.Start();
activator.Bus.Advanced.Topics.Publish(Consts.Order, new StringMessage("Hello there, I'm a publisher!")).Wait();
}
}
UseAzureServiceBus seems wrong, because it is using queue. Is the Handler able to handle message?
class Subscriber
{
static void Main()
{
using (var activator = new BuiltinHandlerActivator())
{
activator.Register(() => new Handler());
Configure.With(activator)
.Logging(l => l.ColoredConsole(minLevel: LogLevel.Warn))
.Transport(t => t.UseAzureServiceBus(Consts.ServiceBusConnectionString, Consts.Order))
.Routing(r => r.TypeBased().MapAssemblyOf<StringMessage>(Consts.Order))
.Start();
activator.Bus.Advanced.Topics.Subscribe(Consts.Order);
Console.WriteLine("This is Subscriber 1");
Console.WriteLine("Press ENTER to quit");
Console.ReadLine();
Console.WriteLine("Quitting...");
}
}
}
class Handler : IHandleMessages<StringMessage>, IHandleMessages<DateTimeMessage>, IHandleMessages<TimeSpanMessage>
{
public async Task Handle(StringMessage message)
{
Console.WriteLine("Got string: {0}", message.Text);
}
public async Task Handle(DateTimeMessage message)
{
Console.WriteLine("Got DateTime: {0}", message.DateTime);
}
public async Task Handle(TimeSpanMessage message)
{
Console.WriteLine("Got TimeSpan: {0}", message.TimeSpan);
}
}
The code above create an Order queue, which is not what I want.
I want a topic, and one or two subscriptions.
Rebus pretty much encourages you to use .NET types as topics, which in this case means that the topics
messages_datetimemessage__messages
messages_stringmessage__messages
messages_timespanmessage__messages
get created, because you publish events of types DateTimeMessage, StringMessage, and TimeSpanMessage (all residing in the Messages namespace in the Messages assembly).
If that is not what you want, Rebus allows you to publish to custom topics like so:
await bus.Advanced.Publish("my-topic", new DateTimeMessage(...))`;
which in this case will result in the creation of a single topic: my-topic.
The subscriber(s) will need to subscribe accordingly:
await bus.Advanced.Subscribe("my-topic");
Keep in mind that there's no filtering of the types of events published to a topic, so the subscriber above will receive anything published to the my-topic topic, regardless of whether it is capable of handling it.
I am looking into Rebus and to use it with Azure Service Bus. Using it with regalure Queues was easy, but when I want to use Topic instead I can't get it to work.
Is there any here that have a done a setup and use it with Topic/Subscription. This is what I have so far.
static void Main(string[] args)
{
_bus1 = InitializeBus(System.Environment.MachineName);
_bus2 = InitializeBus(System.Environment.MachineName + "_2");
_bus3 = InitializeBus();
Run();
Console.WriteLine("Press Enter to exit!");
Console.ReadLine();
}
private static void Run()
{
try
{
_bus1.Handle<string>((b, c, m) => { Console.WriteLine(m); return null; });
_bus2.Handle<string>((b, c, m) => { Console.WriteLine(m); return null; });
_bus1.Bus.Subscribe<string>();
_bus2.Bus.Subscribe<string>();
_bus3.Bus.Publish("Publish test message");
}
catch (Exception ex)
{
throw;
}
}
private static BuiltinHandlerActivator InitializeBus(string queueName = null)
{
var activator = new BuiltinHandlerActivator();
if(string.IsNullOrEmpty(queueName))
Configure.With(activator)
.Transport(t => t.UseAzureServiceBusAsOneWayClient(connectionString))
.Options(o => { o.SetNumberOfWorkers(10); o.SetMaxParallelism(10); })
.Start();
else
Configure.With(activator)
.Transport(t => t.UseAzureServiceBus(connectionString, queueName).EnablePartitioning().DoNotCreateQueues())
.Options(o => { o.SetNumberOfWorkers(10); o.SetMaxParallelism(10); })
.Start();
return activator;
}
First I create all the buses. I am using DontCreateQueues() since I don't want the queues to be duplicated created in my root but only under the Topic as Subscription.
Then I set up the buses and the Publish works fine, there is one Topic created and 2 subscriptions created under this Topic, and there is 1 message in each of this subscriptions. But the messages is never collected.
If I remove the DontCreateQueues() method in the Configuration the code work, but then 2 queues are created in the root togheter with the topic and it's 2 subscriptions, but I can't have it like that.
Best Regards
Magnus
Rebus uses topics by creating a subscription for each topic you subscribe to, and then configure the subscription to forward received messages to the input queue of the bus.
If the bus does not have an input queue with the expected name (either one created by Rebus, or one you created manually), things will not work.
The reason DontCreateQueues() exists is to allow for expert users to configure their queue settings beyond what Rebus is capable of (and willing) to do. It requires a pretty detailed knowledge about how Rebus expects your queue entities to be laid out though, so I would recommend almost anyone to NOT create anything manually, and then simply let Rebus set things up.
I'm using ASP.NET Web API 2.2 along with Owin to build a web service and I observed each call to the controller will be served by a separate thread running on the server side, that's nothing surprising and is the behavior I expected.
One issue I'm having now is that because the server side actions are very memory intense so if more than X number of users are calling in at the same time there is a good chance the server code will throw an out-of-memory exception.
Is it possible to set a global "maximum action count" so that Web Api can queue (not reject) the incoming calls and only proceed when there's an empty slot.
I can't run the web service in 64bit because some of the referenced libraries won't support that.
I also looked at libraries like https://github.com/stefanprodan/WebApiThrottle but it can only throttle based on the frequency of calls.
Thanks
You could add a piece of OwinMiddleware along these lines (influenced by the WebApiThrottle you linked to):
public class MaxConccurrentMiddleware : OwinMiddleware
{
private readonly int maxConcurrentRequests;
private int currentRequestCount;
public MaxConccurrentMiddleware(int maxConcurrentRequests)
{
this.maxConcurrentRequests = maxConcurrentRequests;
}
public override async Task Invoke(IOwinContext context)
{
try
{
if (Interlocked.Increment(ref currentRequestCount) > maxConcurrentRequests)
{
var response = context.Response;
response.OnSendingHeaders(state =>
{
var resp = (OwinResponse)state;
resp.StatusCode = 429; // 429 Too Many Requests
}, response);
return Task.FromResult(0);
}
await Next.Invoke(context);
}
finally
{
Interlocked.Decrement(ref currentRequestCount);
}
}
}