Rebus Saga: Message is not arrived in correct sequence when subscriber(s) sent to publisher - rebus

Recently, we have upgraded rebus version from 1 to 5 and then after we are having problem with Saga handler. Now we are not getting response(s) in correct sequence from different subscriber(s).
We are having different sources to authenticate a request and for that we have coordinator to handle all responses from different authentication sources, but now problem is: All "SearchStarted" messages are not arrived first in coordinator from all authentication sources, due to that we can not check how many authentication sources started authenticating.
Tried to send message in different ways, like
1. Used SEND method instead of REPLY.
2. Tried with our without await keyword before sending response.
3. Tried .Wait() method with Send/Reply method(s).
AuthenticationCoordinator:
public class AuthenticationSaga : Saga<AuthenticationSagaData>, IAmInitiatedBy<AuthenticationRequest>, IHandleMessages<SearchStarted>, IHandleMessages<SearchCompleted>, IHandleMessages<AuthenticationResponse>
{
private readonly IBus _bus;
public IBus Bus
{
get { return _bus; }
}
public AuthenticationSaga(IBus bus)
{
_bus = bus;
}
public async Task Handle(AuthenticationRequest message)
{
if (!IsNew) return;
Data.Id = new Guid(MessageContext.Current.Headers[Rebus.Messages.Headers.CorrelationId]);
Data.ReturnAddress = MessageContext.Current.Headers[Rebus.Messages.Headers.ReturnAddress];
message.UniqueId = Data.Id.ToString();
Data.RequestMessage = message;
Bus.Publish(message);
}
public async Task Handle(SearchStarted message)
{
}
public async Task Handle(SearchCompleted message)
{
}
public async Task Handle(AuthenticationResponse message)
{
}
protected override void CorrelateMessages(ICorrelationConfig<AuthenticationSagaData> config)
{
config.Correlate<AuthenticationRequest>(m => m.UniqueId, d => d.Id);
config.Correlate<SearchStarted>(m => m.UniqueId, d => d.Id);
config.Correlate<AuthenticationResponse>(m => m.UniqueId, d => d.Id);
config.Correlate<SearchCompleted>(m => m.UniqueId, d => d.Id);
}
}
AuthenticationLdap:
public class AuthenticationLdapHandler : IHandleMessages
{
private readonly IBus _bus;
public IBus bus
{
get { return _bus; }
}
public AuthenticationLdapHandler(IBus bus)
{
_bus = bus;
}
public async Task Handle(AuthenticationRequest message)
{
await bus.Reply(new SearchStarted { MessageId = MessageContext.Current.Headers[Rebus.Messages.Headers.CorrelationId], UniqueId = message.UniqueId });
var response = AuthenticateLdap(message); await bus.Reply(response);
await bus.Reply(new SearchCompleted { MessageId = MessageContext.Current.Headers[Rebus.Messages.Headers.CorrelationId], UniqueId = message.UniqueId });
}
}
AuthenticationNative:
public class AuthenticationNativeHandler : IHandleMessages
{
private readonly IBus _bus;
public IBus bus
{
get { return _bus; }
}
public AuthenticationNativeHandler(IBus bus)
{
_bus = bus;
}
public async Task Handle(AuthenticationRequest message)
{
await bus.Reply(new SearchStarted { MessageId = MessageContext.Current.Headers[Rebus.Messages.Headers.CorrelationId], UniqueId = message.UniqueId });
var response = AuthenticateNative(message); await bus.Reply(response);
await bus.Reply(new SearchCompleted { MessageId = MessageContext.Current.Headers[Rebus.Messages.Headers.CorrelationId], UniqueId = message.UniqueId });
}
}
We are expecting response(s) in AuthenticationCoordinator as below sequence:
SearchStarted message from Ldap
SearchStarted message from Native
AuthenticationResponse message from Ldap
SearchCompleted message from Ldap
AuthenticationResponse message from Native
SearchCompleted message from Native
But now we are getting response(s) in below sequence:
SearchStarted message from Ldap
AuthenticationResponse message from Ldap
SearchCompleted message from Ldap
SearchStarted message from Native
AuthenticationResponse message from Native
SearchCompleted message from Native
Can we set priority to message? how we can achieve above expected responses in rebus 5.

What you're seeing is most likely a consequence of the fact that Rebus ensures that all outgoing messages are sent AFTER your handler has finished executing.
It does this by enlisting all bus operations in its transaction context, which gets committed only after your handler code is done.
This means that code like
public async Task Handle(string message)
{
await bus.Reply("this is message 1");
await Task.Delay(TimeSpan.FromSeconds(1));
await bus.Reply("this is message 2");
await Task.Delay(TimeSpan.FromSeconds(1));
await bus.Reply("this is message 3");
}
will result in messages 1, 2, and 3 being sent at the same time, when the Rebus transaction context is committed, which means that the recipient will receive them in random order.
If you messages to be sent immediately from your handler, you can "dismantle" the transaction context like this:
var transactionContext = AmbientTransactionContext.Current;
AmbientTransactionContext.SetCurrent(null);
try
{
// current transaction will never know....
await bus.Send(whee);
}
finally
{
AmbientTransactionContext.SetCurrent(transactionContext);
}
which I suggest you wrap in an implementation of IDisposable which enables usage like this:
using(new RebusTransactionContextDismantler())
{
// current transaction will never know....
await bus.Publish(whee);
}

Related

Different threads using same instance of DbContext, called from within DelegatingHandler

I have .Net Core 3.1 application that is using EF Core 3.1.9. During a specific process I am getting the following error:
A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913
I am using Dependency Injection for the DbContext and have gone through all the flows to make sure everything is properly and immediately await'ed.
The error occurs within LtiUserRepository.cs which will be shown below.
That process starts with an external http call using an HttpClient that has a custom MessageHandler, registered in Startup.cs:
services.AddHttpClient<MyRepository>("MyCustomUserClient", client =>
{
var canvasUrl = Configuration.GetSection("Urls:Removed").Value ?? "https://example.com/";
client.BaseAddress = new System.Uri(removed);
}).AddHttpMessageHandler<LtiUserApiAuthenticationHttpClientHandler>();
The code that initiates the HTTP Call is:
public async Task<PlatformQuizSubmissions> GetUserQuiz(string courseId, string quizId)
{
var path = $"api/v1/courses/{courseId}/quizzes/{quizId}/submission";
var response = await _myCustomUserClient.GetAsync(path);
// some stuff
var responseContent = await response.Content.ReadAsStringAsync();
// Some other stuff
}
The purpose of the custom MessageHandler is to check for a header, get some data, and append a query parameter to each request
public sealed class LtiUserApiAuthenticationHttpClientHandler : DelegatingHandler
{
private readonly IHttpContextAccessor _accessor;
private readonly ILtiUserService _userService;
public LtiUserApiAuthenticationHttpClientHandler(IHttpContextAccessor accessor, ILtiUserService ltiUserService)
{
_accessor = accessor;
_userService = ltiUserService;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var obo = _accessor.HttpContext.Request.Headers["QT-OBO"];
// THIS IS THE PART THAT QUERIES THE DATABASE
var user = await _userService.Get(new Guid(obo));
var uriBuilder = new UriBuilder(request.RequestUri);
if (string.IsNullOrEmpty(uriBuilder.Query))
{
uriBuilder.Query = $"as_user_id={user.PlatformUserId}";
}
else
{
uriBuilder.Query = $"{uriBuilder.Query}&as_user_id={user.PlatformUserId}";
}
request.RequestUri = uriBuilder.Uri;
return await base.SendAsync(request, cancellationToken);
}
}
You can see above that the MessageHandler calls _userservice.Get, which is this:
public async Task<LtiUser> Get(Guid guid)
{
return await _ltiUserRepository.Get(guid);
}
That simply returns from the repository, which is this:
public class LtiUserRepository : ILtiUserRepository
{
private readonly SqlDbContext _db;
private readonly IMapper _mapper;
private readonly ILogger<LtiUserRepository> _logger;
public LtiUserRepository(SqlDbContext sqlDbContext, IMapper mapper, ILoggerFactory logger)
{
_db = sqlDbContext;
_mapper = mapper;
_logger = logger != null ? logger.CreateLogger<LtiUserRepository>() : throw new ArgumentNullException(nameof(logger));
}
public async Task<LtiUser> Get(Guid guid)
{
try
{
return await _db.LtiUsers
.AsNoTracking()
.Where(l => l.UUID == guid)
.ProjectTo<LtiUser>(_mapper.ConfigurationProvider)
.SingleOrDefaultAsync();
}
catch (Exception ex)
{
// This is where the error is caught.
_logger.LogCritical($"Could not get LtiUser via (UUID) {guid} : {ex.Message}");
return null;
}
}
}
The database is registered in Startup.cs with:
protected virtual void ConfigureDatabaseServices(IServiceCollection services)
{
services.AddDbContext<SqlDbContext>(
o => o.UseSqlServer(Configuration.GetConnectionString("DbConnectionString")),
ServiceLifetime.Transient);
}
When I hit this endpoint using ApacheBench with 20 requests, concurrency of 2 I get this error anywhere from 2 to 10 times. However, looking at the following snippet from the MessageHandler (LtiUserApiAuthenticationHttpClientHandler) again:
var user = await _userService.Get(new Guid(obo));
if (string.IsNullOrEmpty(uriBuilder.Query))
{
uriBuilder.Query = $"as_user_id={user.PlatformUserId}";
}
else
{
uriBuilder.Query = $"{uriBuilder.Query}&as_user_id={user.PlatformUserId}";
}
If I replace user.PlatformUserId with a hardcoded, known value, (and comment out the call to _userService.Get) I can use AB with 1000 requests and a concurrency of 20 and have 0 occurrences of the issue. That leads me to believe I have it narrowed down to the offending flow, but am not sure of the correct way to do this.

SignalR core console client not receiving notifications

I have looked around at some of the Similar questions and didn't figure this problem out. I have a simple Hub in my .NET core Web API project. Here is the Hub:
public class NotificationHub : Hub<INotificationClient>
{
public async Task SendMessage(string user, string msg)
{
await Clients.All.ReceiveMessage(user, msg);
}
public Task SendMessageToCaller(string msg)
{
return Clients.Caller.ReceiveMessage(msg);
}
public Task SendMessageToPartner(string user, string msg)
{
return Clients.Client(user).ReceiveMessageToPartner(msg);
}
}
Here is the Interface:
public interface INotificationClient
{
Task ReceiveMessage(string user, string msg);
Task ReceiveMessage(string msg);
Task ReceiveMessageToPartner( string msg);
}
Here is the code from the controller:
[Route("[controller]")]
[ApiController]
public class NotificationsController : ControllerBase
{
private IHubContext<NotificationHub> _hub;
public NotificationsController(IHubContext<NotificationHub> hub)
{
_hub = hub;
}
[HttpGet]
public async Task<IActionResult> Get()
{
var msg = new NotificationData { ClientId = "12345", Notification = "Somone just connected" };
await _hub.Clients.All.SendAsync("Notification", msg);
return Ok(new { Message = "Request complete" });
}
}
Lastly here is the console client code:
Console.WriteLine("Press a key to start listening");
Console.ReadKey();
Console.WriteLine("Client Listening!");
var connection = new HubConnectionBuilder()
.WithUrl("http://localhost:61514/notifications")
.Build();
connection.On<NotificationData>("Notification", (notificationData) =>
Console.WriteLine($"Somebody connected: {notificationData.ClientId}"));
connection.StartAsync().GetAwaiter().GetResult();
Console.WriteLine("Listening. Press a key to quit");
Console.ReadKey();
Here is the startup of the web app with the mappings:
public void ConfigureServices(IServiceCollection services)
{
services.AddSignalR();
services.AddControllers();
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseRouting();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
endpoints.MapHub<NotificationHub>("/Notifications");
});
}
I keep getting this error: System.IO.IOException: 'The server disconnected before the handshake could be started.' I must be missing something along the way here.
Update:
Turned on the logs and got this error:
dbug: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[8]
Establishing connection with server at 'http://localhost:61514/notifications'.
dbug: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[9]
Established connection 'BdROAEEQnGUeDYAW5EspRA' with the server.
dbug: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[7]
Starting transport 'ServerSentEvents' with Url: http://localhost:61514/notifications.
info: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[1]
Starting transport. Transfer mode: Text.
dbug: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[3]
Starting receive loop.
dbug: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[9]
Received 30 bytes. Parsing SSE frame.
dbug: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[4]
Receive loop stopped.
dbug: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[100]
Starting the send loop.
dbug: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[18]
Transport 'ServerSentEvents' started.
dbug: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[102]
Send loop canceled.
dbug: Microsoft.AspNetCore.Http.Connections.Client.Internal.ServerSentEventsTransport[101]
Send loop stopped.
Unhandled exception. info: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[3]
HttpConnection Started.
info: Microsoft.AspNetCore.SignalR.Client.HubConnection[24]
Using HubProtocol 'json v1'.
System.IO.IOException: The server disconnected before the handshake could be started.
at Microsoft.AspNetCore.SignalR.Client.HubConnection.HandshakeAsync(ConnectionState startingConnectionState, CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore(CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore(CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncInner(CancellationToken cancellationToken)
at System.Threading.Tasks.ForceAsyncAwaiter.GetResult()
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsync(CancellationToken cancellationToken)
at NotificationClient.Program.Main(String[] args)
So after chonking on this all day, I found out what the problem was, so I thought I'd post the solution here just in case someone else is having this problem.
The Hub Endpoint was pointing to the controller. So when the client was linking it was hitting the controller twice causing a the server to close the connection. So I changed this line:
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
endpoints.MapHub<NotificationHub>("/Notify");
});
from "/Notifications" which was hitting the controller, to "Notify" as in the code above, re-pointed the client from Notifications to Notify
var connection = new HubConnectionBuilder()
.WithUrl("http://localhost:61514/Notify")
.ConfigureLogging(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Debug);
})
.Build();
and messages started to flow in.

Hot to get count of connection in SignalR Core Group

How can I get count of connections in specific SignalR Core group?
Something like that:
this.Clients.Group("Something").Count();
It would be also good if SendAsync would return how many clients send message:
var count = await this.Clients.Client(this.Context.ConnectionId).SendAsync("msg", msg);
I need this to determine if group is empty. Data should be correct if user disconnects from SignalR Core with force (unplug net cable)...
You can use the OnConnected and OnDisconnected Events and save the connection / connectionIds.
I use something like this with additional informations:
internal static ConcurrentDictionary<string, ConnectionInfos> Users = new ConcurrentDictionary<string, ConnectionInfos>();
public override async Task OnConnectedAsync()
{
var connId = Context.ConnectionId;
if (!Users.Keys.Any(x => x == connId))
{
Users.TryAdd(connId, new ConnectionInfos { /*...*/});
}
await base.OnConnectedAsync();
}
public override async Task OnDisconnectedAsync(Exception exception)
{
var id = Context.ConnectionId;
if (Users.TryRemove(id, out ConnectionInfos ci))
{
//Connection removed
}
await base.OnDisconnectedAsync(exception);
}

Passing JWT Token as QueryString to SignalR Hub

Trying to follow the suggestions in the link below to pass a JWT token to my SignalR hub but so far it's not working. In particular, see David Fowler's suggestion on July 22, 2017. https://github.com/aspnet/SignalR/issues/130
My frontend is React so I'm simply adding the token to the querystring as follows where _token has my JWT token value:
const connection = new signalR.HubConnectionBuilder()
.withUrl("/myhub?AUTHORIZATION=" + _token)
.configureLogging(signalR.LogLevel.Information)
.build();
In the ConfigureServices() method of my Startup.cs, I have the following configuration for Jwt tokens:
services.AddAuthentication(options => {
options.DefaultScheme = JwtBearerDefaults.AuthenticationScheme;
})
.AddJwtBearer(jwtOptions => {
jwtOptions.Authority = $"https://login.microsoftonline.com/tfp/{Configuration["AzureAdB2C:Tenant"]}/{Configuration["AzureAdB2C:Policy"]}/v2.0/";
jwtOptions.Audience = Configuration["AzureAdB2C:ClientId"];
jwtOptions.Events = new JwtBearerEvents
{
OnMessageReceived = context =>
{
if(context.HttpContext.WebSockets.IsWebSocketRequest)
context.Token = context.Request.Query["AUTHORIZATION"];
return Task.CompletedTask;
}
};
});
And this is what my Hub looks like:
[Authorize]
public class MyHub : Hub
{
private IBackendService _backendService;
public MyHub(IBackendService backendService)
{
_backendService = backendService;
}
public async Task SendMessage(string message)
{
// Regular SignalR stuff
// SignalR will now send the message to all connected users...
}
}
Basically, I'm getting the 401 Unauthorized error.
I put a break point where I check to see if the request is a web sockets request but I'm not hitting it. Looks like something in the pipeline is determining that the user is not authenticated.
What am I doing wrong in my code?
You can solve this by using custom middleware to handle grabbing the authentication token from the query string.
public class SignalRQueryStringAuthMiddleware
{
private readonly RequestDelegate _next;
public SignalRQueryStringAuthMiddleware(RequestDelegate next)
{
_next = next;
}
// Convert incomming qs auth token to a Authorization header so the rest of the chain
// can authorize the request correctly
public async Task Invoke(HttpContext context)
{
if (context.Request.Headers["Connection"] == "Upgrade" &&
context.Request.Query.TryGetValue("authToken", out var token))
{
context.Request.Headers.Add("Authorization", "Bearer " + token.First());
}
await _next.Invoke(context);
}
}
public static class SignalRQueryStringAuthExtensions
{
public static IApplicationBuilder UseSignalRQueryStringAuth(this IApplicationBuilder builder)
{
return builder.UseMiddleware<SignalRQueryStringAuthMiddleware>();
}
}
This will try to get the query string value "authToken" and it will set the heads so you can leverage your authentication middleware. You need to call this before the authentication middleware in the pipeline like so:
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
//...
app.UseSignalRQueryStringAuth();
app.UseAuthentication();
//...
}
EDIT
on a side note you should only append the token if the user is logged in:
if (accessToken) {
hubUrl += '?authToken' +'=' + accessToken;
}
this._hubConnection = new HubConnectionBuilder()
.withUrl(hubUrl)
.build();
I have also implemented this in my project. The shortest way of doing this is to add a middleware to your Configure method.
app.Use(async (context, next) =>
{
var accessToken = context.Request.Query["access_token"];
if (!string.IsNullOrEmpty(accessToken))
{
context.Request.Headers["authorization"] = "Bearer " + accessToken;
}
await next.Invoke().ConfigureAwait(false);
});
It does the same thing as mentioned in the other answer. It adds the token to the header by reading it from the query string. Of course, you can separate out the implementation of custom middleware in a separate file.

Are SignalR connectionIDs hub-specific?

If I have several hubs, and connect a single JavaScript client to all of them, will the context ConnectionID be the same between them?
Interesting question. I didn't know the answer, so I tested it using this example by changing it a bit.
The Hub classes:
public class ChatHub : Hub {
public void Send(string name, string message) {
string cid = Context.ConnectionId;
Clients.All.sendMessage(name, message);
}
}
public class ChatHub2 : Hub
{
public void Send(string name, string message)
{
string cid = Context.ConnectionId;
Clients.All.sendMessage(name, message);
}
}
The page.html connecting to the hubs:
var chat = $.connection.chatHub;
var chat2 = $.connection.chatHub2;
$.connection.hub.start().done(function () {
// Call the Send method on the hub.
chat.server.send('Me', 'Message to 1');
chat2.server.send('Me', 'Message to 2');
});
I set breakpoints on the Hub methods and both are called, and Context.ConnectionId are the same. That's what I was expecting. Give it a try!
It makes sense, it supposed to use the same connection to send the message over.

Resources