Rebus saga data state issue - rebus

I'm trying out the simple deferred message timeout pattern in Rebus along the outline here http://mookid.dk/oncode/archives/3043 in order to have alternative behaviors depending on whether or not I receive a timely response from an other service hooked up to the same bus. Code has been modified to use async/await.
(yes, I know that article is really about unit testing. I'm just trying out the same timeout thing)
The handler in my case is a saga. Two message send operations are awaited as the last two calls of the handler of the message that starts the saga. First message uses request/reply from the external service and the reply is also eventually handled in the same saga.
Second message is a deferred message that is supposed to enforce the alternative action in case timeout occurs, just like in the Rebus unit test example. I've verified that messages are sent from and received by the saga/handler without any problems. It looks something like this:
public class TestSaga : Saga<TestSagaData>, IAmInitiatedBy<SomeMessage>, IHandleMessages<SomeReply>, IHandleMessages<TimeOutMessage>
{
private readonly IBus _bus;
public TestSaga(IBus bus)
{
_bus = bus;
}
protected override void CorrelateMessages(ICorrelationConfig<TestSagaData> config)
{
config.Correlate<SomeMessage>(s => s.Identifier, d => d.OriginalMessageIdentifier);
config.Correlate<SomeMessage>(s => s.Tag, d => d.CorrelationIdentifier);
config.Correlate<SomeReply>(s => s.Tag, d => d.CorrelationIdentifier);
config.Correlate<TimeOutMessage>(s => s.Tag, d => d.CorrelationIdentifier);
}
public async Task Handle(SomeMessage message)
{
if (!IsNew)
return;
Data.CorrelationIdentifier = message.Tag;
Data.OriginalMessageIdentifier = message.Identifier;
Data.ReplyReceived = false;
await _bus.Send(new SomeRequest {Tag = message.Tag});
await _bus.Defer(TimeSpan.FromSeconds(30), new TimeOutMessage() {Tag = message.Tag});
}
public async Task Handle(SomeReply message)
{
// Even if we would get here loooong before...
Data.ReplyReceived = true;
await DoStuffIfNotTimedout();
}
public async Task Handle(TimeOutMessage message)
{
// ...this, DoStuffIfTimeout below is always called
// since state is preserved from the _bus.Defer call. Correct?
if (!Data.ReplyReceived)
await DoStuffIfTimedout();
}
private async Task DoStuffIfNotTimedout()
{
// some more async stuff here
MarkAsComplete();
}
private async Task DoStuffIfTimedout()
{
// some more async stuff here
MarkAsComplete();
}
}
I have added a boolean saga data flag/property to indicate that the reply to the first message was received first, setting it to false initially before the both await Send/Defer calls
and setting it to true immediately in the message handler of the reply.
The flag was supposed to be used to prevent the timeout actions to start in case the deferred timeout thing was received after the first reply but before the subsequent actions were done with and the saga marked completed.
However, in this case the state of the saga seems to 'follow' the message received. So if the first message reply handler is entered first and sets the saga data flag to true. Then when the deferred message handler is entered later,
something has reset the flag again, seemingly ignoring the action taken in the first reply handler (setting the flag true). Not sure whether the 'Revision' number of the saga is supposed to change or not but it remains unchanged (zero) all the time it seems.
Also noted that it doesn't matter if the timeout occurs long after the reply handler is entered, when the timeout handler is entered, the the flag is 'false'.
Is this saga state behavior by design? I thought saga state would somehow be persisted between message handler calls. If it's the wrong behavior, what could possibly cause it?
I quite new to Rebus so I'm sure I've misunderstood something here but in that case I would like to know what :).
Transport used under the hood is RabbitMQ.
Test code: saga state test

Related

Waiting for multiple external events in Durable Azure Functions and act on them as they occur

I'm using Durable Functions in a process to collect data from 12 different people. For this I chose the WaitForExternalEvent method. I need to get notified of those external events as they happen. All events must be received in the next 1h.
I have created the following orchestration. The behaviour is odd however. The orchestration neither completes, nor fails. I am using Durable Functions Monitor (dfMon) to inspect the logs. As you can see in the execution history all 12 events are in fact received (before the 1h timeout). Still the orchestrator:
didn't execute the Fxbm_notifyOfNewReport activity function after each received event
also didn't exit the while loop after all 12 events
Also, more than 1h has elapsed and every timeout timer has fired. Still, no exception was thrown. The orchestration is still in a running state.
I took inspiration for this from this doc and this blog.
Does someone know why I am not seeing the expected behaviour?
public class Fxbm_workflow
{
[FunctionName(nameof(Fxbm_workflow))]
public async Task Run([OrchestrationTrigger] IDurableOrchestrationContext ctx, ILogger log)
{
var id = ctx.InstanceId;
var trigger = ctx.GetInput<Trigger<OrchestrationInput2>>();
// sub-orchestration to distribute data to people
// return value is int[], these are the ids of the people
var input = (trigger, id);
var childWorkflowId = $"{id}_{nameof(Fxbm_prep_workflow)}";
var requiredCompanies = await ctx.CallSubOrchestratorAsync<int[]>(nameof(Fxbm_prep_workflow), instanceId: childWorkflowId, input: input);
// to every distributed data package, a string response is expected in 1h at the latest
var expiresIn = TimeSpan.FromHours(1);
var responseTasks = requiredCompanies.Select(id => ctx.WaitForExternalEvent<string>(id.ToString(), expiresIn)).ToList();
// all tasks need to complete or a timeout exception must be thrown
// I want to get notified of responses as they come in
// therefore Task.WhenAll() is not suitable
while (responseTasks.Any())
{
try
{
// responses as they occur
var receivedResponse = await Task.WhenAny(responseTasks);
responseTasks.Remove(receivedResponse);
var stringResponse = await receivedResponse;
// notify via mail
await ctx.CallActivityAsync(nameof(Fxbm_notifyOfNewReport), stringResponse);
}
catch (TimeoutException)
{
// break;
throw;
}
}
}
}

SignalR return value from client method

Hello I'm developing a Server-Client application that communicate with SignalR. What I have to implement is a mechanism that will allow my server to call method on client and get a result of that call. Both applications are developed with .Net Core.
My concept is, Server invokes a method on Client providing Id of that invocation, the client executes the method and in response calls the method on the Server with method result and provided Id so the Server can match the Invocation with the result.
Usage is looking like this:
var invocationResult = await Clients
.Client(connectionId)
.GetName(id)
.AwaitInvocationResult<string>(ClientInvocationHelper._invocationResults, id);
AwaitInvocationResult - is a extension method to Task
public static Task<TResultType> AwaitInvocationResult<TResultType>(this Task invoke, ConcurrentDictionary<string, object> lookupDirectory, InvocationId id)
{
return Task.Run(() =>
{
while (!ClientInvocationHelper._invocationResults.ContainsKey(id.Value)
|| ClientInvocationHelper._invocationResults[id.Value] == null)
{
Thread.Sleep(500);
}
try
{
object data;
var stingifyData = lookupDirectory[id.Value].ToString();
//First we should check if invocation response contains exception
if (IsClientInvocationException(stingifyData, out ClientInvocationException exception))
{
throw exception;
}
if (typeof(TResultType) == typeof(string))
{
data = lookupDirectory[id.Value].ToString();
}
else
{
data = JsonConvert.DeserializeObject<TResultType>(stingifyData);
}
var result = (TResultType)data;
return Task.FromResult(result);
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
});
}
As you can see basically I have a dictionary where key is invocation Id and value is a result of that invocation that the client can report. In a while loop I'm checking if the result is already available for server to consume, if it is, the result is converted to specific type.
This mechanism is working pretty well but I'm observing weird behaviour that I don't understand.
If I call this method with await modifier the method in Hub that is responsible to receive a result from client is never invoked.
///This method gets called by the client to return a value of specific invocation
public Task OnInvocationResult(InvocationId invocationId, object data)
{
ClientInvocationHelper._invocationResults[invocationId.Value] = data;
return Task.CompletedTask;
}
In result the while loop of AwaitInvocationResult never ends and the Hub is blocked.
Maby someone can explain this behaviour to me so I can change my approach or improve my code.
As it was mentioned in the answer by Brennan, before ASP.NET Core 5.0 SignalR connection was only able to handle one not streaming invocation of hub method at time. And since your invocation was blocked, server wasn't able to handle next invocation.
But in this case you probably can try to handle client responses in separate hub like below.
public class InvocationResultHandlerHub : Hub
{
public Task HandleResult(int invocationId, string result)
{
InvoctionHelper.SetResult(invocationId, result);
return Task.CompletedTask;
}
}
While hub method invocation is blocked, no other hub methods can be invoked by caller connection. But since client have separate connection for each hub, he will be able to invoke methods on other hubs. Probably not the best way, because client won't be able to reach first hub until response will be posted.
Other way you can try is streaming invocations. Currently SignalR doesn't await them to handle next message, so server will handle invocations and other messages between streaming calls.
You can check this behavior here in Invoke method, invocation isn't awaited when it is stream
https://github.com/dotnet/aspnetcore/blob/c8994712d8c3c982111e4f1a09061998a81d68aa/src/SignalR/server/Core/src/Internal/DefaultHubDispatcher.cs#L371
So you can try to add some dummy streaming parameter that you will not use:
public async Task TriggerRequestWithResult(string resultToSend, IAsyncEnumerable<int> stream)
{
var invocationId = InvoctionHelper.ResolveInvocationId();
await Clients.Caller.SendAsync("returnProvidedString", invocationId, resultToSend);
var result = await InvoctionHelper.ActiveWaitForInvocationResult<string>(invocationId);
Debug.WriteLine(result);
}
and on the client side you will also need to create and populate this parameter:
var stringResult = document.getElementById("syncCallString").value;
var dummySubject = new signalR.Subject();
resultsConnection.invoke("TriggerRequestWithResult", stringResult, dummySubject);
dummySubject.complete();
More details: https://learn.microsoft.com/en-us/aspnet/core/signalr/streaming?view=aspnetcore-5.0
If you can use ASP.NET Core 5, you can try to use new MaximumParallelInvocationsPerClient hub option. It will allow several invocations to execute in parallel for one connection. But if your client will call too much hub methods without providing result, connection will hang.
More details: https://learn.microsoft.com/en-us/aspnet/core/signalr/configuration?view=aspnetcore-5.0&tabs=dotnet
Actually, since returning values from client invocations isn't implemented by SignalR, maybe you can try to look into streams to return values into hubs?
This is supported in .NET 7 now https://learn.microsoft.com/en-us/aspnet/core/signalr/hubs?view=aspnetcore-7.0#client-results
By default a client can only have one hub method running at a time on the server. This means that when you wait for a result in the first hub method, the second hub method will never run since the first hub method is blocking the processing loop.
It would be better if the OnInvocationResult method ran the logic in your AwaitInvocationResult extension and the first hub method just registers the id and calls the client.

Making Azure Function signalR payloads camel case

We have an HttpTriggred function according to the following code snippet:
[FunctionName("commandcompleted")]
public static Task SendMessage(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = "commandcompleted/{userId}")]
object message,
string userId,
[SignalR(HubName = Negotitate.HubName)]IAsyncCollector<SignalRMessage>
signalRMessages,
ILogger log)
{
return signalRMessages.AddAsync(
new SignalRMessage
{
UserId = userId,
Target = "CommandCompleted",
Arguments = new[] { message }
});
}
The client app which is, in fact, a signalR client receives a notification upon completion an operation once the mentioned trigger is invoked.
It's observed that the payload received by the client app is always in Pascal Case. How can we augment the function's code so that it broadcasts the payload in camel case format? Please note that decorating the object's properties by [JsonProperty("camelCasePropertyName")] is not an option and we'd like to do away from it.
The application tier which prepares message object must take care of serializing it in camel case format before submitting it to the http-triggered function.

SignalR Long Running Process

I have setup a SignalR hub which has the following method:
public void SomeFunction(int SomeID)
{
try
{
Thread.Sleep(600000);
Clients.Caller.sendComplete("Complete");
}
catch (Exception ex)
{
// Exception Handling
}
finally
{
// Some Actions
}
m_Logger.Trace("*****Trying To Exit*****");
}
The issue I am having is that SignalR initiates and defaults to Server Sent Events and then hangs. Even though the function/method exits minutes later (10 minutes) the method is initiated again ( > 3 minutes) even when the sendComplete and hub.stop() methods are initiated/called on the client prior. Should the user stay on the page the initial "/send?" request stays open indefinitely. Any assistance is greatly appreciated.
To avoid blocking the method for so long, you could use a Taskand call the client method asynchronously.
public void SomeFunction(Int32 id)
{
var connectionId = this.Context.ConnectionId;
Task.Delay(600000).ContinueWith(t =>
{
var message = String.Format("The operation has completed. The ID was: {0}.", id);
var context = GlobalHost.ConnectionManager.GetHubContext<SomeHub>();
context.Clients.Client(connectionId).SendComplete(message);
});
}
Hubs are created when request arrives and destroyed after response is sent down the wire, so in the continuation task, you need to create a new context for yourself to be able to work with a client by their connection identifier, since the original hub instance will no longer be around to provide you with the Clients method.
Also note that you can leverage the nicer syntax that uses async and await keywords for describing asynchronous program flow. See examples at The ASP.NET Site's SignalR Hubs API Guide.

Parallel exceptions are being caught

somehow my exceptions seem to being caught by method they are executing in. Here is the code to call the method. As you can see I create a cancellation token with a time out. I register a method to call when the cancellation token fires and then I start a new task. The cancellation token appears to be working OK. As does the registered method.
var cancellationToken = new CancellationTokenSource(subscriber.TimeToExpire).Token;
cancellationToken.Register(() =>
{
subscriber.Abort();
});
var task = Task<bool>.Factory.StartNew(() =>
{
subscriber.RunAsync((T)messagePacket.Body, cancellationToken);
return true;
})
.ContinueWith(anticedant =>
{
if (anticedant.IsCanceled)
{
Counter.Increment(12);
Trace.WriteLine("Request was canceled");
}
if (anticedant.IsFaulted)
{
Counter.Increment(13);
Trace.WriteLine("Request was canceled");
}
if (anticedant.IsCompleted)
{
Counter.Increment(14);
}
The next piece of code is the method that seems to be not only throwing the excetion (expected behavior. but also catching the exception.
public async override Task<bool> ProcessAsync(Message input, CancellationToken cancellationToken)
{
Random r = new Random();
Thread.Sleep(r.Next(90, 110));
cancellationToken.ThrowIfCancellationRequested();
return await DoSomethingAsync(input);
}
The exception is being thrown by the cancellation token but according to intellitrace it is being caught at the end of the method. I have tried a number of different options including throwing my own exception, but no matter what the continuewith function always executes the IsComleted or ran to completion code.
Any ideas on what I am doing wrong?
Thanks
I assume that RunAsync is the same as ProcessAsync.
The exception is being thrown by the cancellation token but according to intellitrace it is being caught at the end of the method.
Yup. Any async method will catch its own exceptions and place them on its returned Task. This is by design.
no matter what the continuewith function always executes the IsComleted or ran to completion code.
Well, let's take another look at the code:
var task = Task<bool>.Factory.StartNew(() =>
{
subscriber.RunAsync((T)messagePacket.Body, cancellationToken);
return true;
})
.ContinueWith(
Consider the lambda passed to StartNew: it calls RunAsync, it ignores the Task that it returns, and then it returns true (successfully). So the Task returned by StartNew will always return successfully. This is why the ContinueWith always executes for a successfully-completed task.
What you really want is to await the Task returned by RunAsync. So, something like this:
var task = Task.Run(async () =>
{
await subscriber.RunAsync((T)messagePacket.Body, cancellationToken);
return true;
})
.ContinueWith(
You're still ignoring the return value of RunAsync (the bool it returns is ignored), but you're not ignoring the Task itself (including cancellation/exception information).
P.S. I'm assuming there's a lot of code you're not showing us; using StartNew/Run to just kick off some async work and return a value is very expensive.
P.P.S. Use await Task.Delay instead of Thread.Sleep.

Resources