SignalR just terminate single long running task among many within a specific client - signalr

I have been searching for right methodology to achieve this scenario without any luck.
This is regarding multiple long running tasks to a specific client, let's say there has been three tasks running in single client asynchronously at time t0, t1 and t2.
Clients.Client(TheConnectionID).Task1(GUID1) //at t0
Clients.Client(TheConnectionID).Task1(GUID2) //at t1
Clients.Client(TheConnectionID).Task1(GUID3) //at t2
Later, from server(Hub) user's request was sent to Task started running from t1.
Server know the GUID sent for the Task1 started at t1.
Clients.Client(TheConnectionID).Cancel(GUID2) //To cancel task started at t1
I tried to pair the GUID with CancellationToken and somehow throw operation cancelled exception with no luck. Somehow the thread where the exception thrown seems not the correct thread where t1 Task is running..
Please someone shed me light what I am doing wrong or if this is never possible with signalr invoke methods on client side.

Answering my own question seems awkward but I think this could be valuable to whom in similar situation. Also this solution with my observations could bring further discussion shedding some light.
Here is what I have found:
Based on the invoke of user request:
Clients.Client(TheConnectionID).Cancel(GUID)
You can do a handling of cancellation routine per the specific task in the client side:
try
{
concurrentCTSDictionary.TryGetValue(GUID, out CancellationTokenSource cts);
cts.Cancel();
}
The actual event handler, which was running task and token fed:
async Task SomeMethodUserWantCancel([]args, CancellationToken ct)
{
// Option #1 or #2
}
Now here is what gets a little strange,
When I used option #1, the thread containing throwing exception was not running on the same thread where actual task was running and task kept running even when exception thrown. That's why I posted question to stackoverflow.
Option #1:
using (var ctr = ct.Register(async () =>
{
ct.ThrowIfCancellationRequested();
}))
{
while (longtaskrunning)
{
await longTask();
}
}
Option #2 worked and solved my issue:
while (longtaskrunning)
{
ct.ThrowIfCancellationRequested();
await longTask();
}

Related

ASP.NET Core multithreaded background threads

Using ASP.NET Core .NET 5. Running on Windows.
Users upload large workbooks that need to be converted to a different format. Each conversion process is CPU intensive and takes around a minute to complete.
The idea is to use a pattern where the requests are queued in a background queue and then processed by background tasks.
So, I followed this Microsoft article
The queuing part worked well but the issue was that workbooks were executing sequentially in the background:
private async Task BackgroundProcessing(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var workItem =
await TaskQueue.DequeueAsync(stoppingToken);
try
{
await workItem(stoppingToken);
}
catch (Exception ex)
{
_logger.LogError(ex,
"Error occurred executing {WorkItem}.", nameof(workItem));
}
}
}
If I queued 10 workbooks. Workbook 2 wouldn't start until workbook 1 is done. Workbook 3 wouldn't start until workbook 2 is done, etc.
So, I modified the code to run tasks without await and hid the warning with the discard operator (please note workItem is now Action, not Task):
while (!stoppingToken.IsCancellationRequested)
{
var workItem = await TaskQueue.DequeueAsync(stoppingToken);
_ = Task.Factory.StartNew(() =>
{
try
{
workItem(stoppingToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error occurred executing {WorkItem}.", nameof(workItem));
}
}, TaskCreationOptions.LongRunning);
}
That works -- I get all workbooks starting processing around the same time, and then they complete around the same time too. But, I am not sure if doing this is dangerous and can lead to bugs, crashes, etc.
Is the second version a workable solution, or will it lead to some disaster in the future? Is there a better way to implement parallel workloads on the background threads in ASP.NET?
Thanks.
Using an external queue has some advantages over in-memory queueing. In particular, the queue message are stored in a reliable external store with features around retries, multiple consumers, etc. If your app crashes, the queue item remains and can be tried again.
In Azure, you can use several services including Azure Storage Queues and Service Bus. I like Service Bus because it uses push-based behavior to avoid the need for a polling loop in your code. Either way, you can create an instance of IHostedService that will watch the queue and process the work items in a separate thread with configurable parallelization.
Look for examples on using within ASP.NET Core, for example:
https://damienbod.com/2019/04/23/using-azure-service-bus-queues-with-asp-net-core-services/
The idea is to use a pattern where the requests are queued in a background queue and then processed by background tasks.
The proper solution for request-extrinsic code is to use a durable queue with a separate backend processor. Any in-memory solution will lose that work any time the application is shut down (e.g., during a rolling upgrade).

C# Tasks created by async/await are not creating separate Threads. How else does it work to have a picture in mind?

According to
If async-await doesn't create any additional threads, then how does it make applications responsive?
a C# task, executed by await ... doesn't create a separate thread for the target Task. However, I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I still do not understand, what's going on.
public class TestProgram
{
private static async Task HandleClient(TcpClient clt)
{
using NetworkStream ns = clt.GetStream();
using StreamReader sr = new StreamReader(ns);
while (true)
{
string msg = await sr.ReadLineAsync();
Console.WriteLine($"Received in {System.Threading.Thread.CurrentThread.ManagedThreadId} :({msg.Length} bytes):\n{msg}");
}
}
private static async Task AcceptConnections(int port)
{
TcpListener listener = new TcpListener(IPAddress.Parse("127.0.0.1"), port);
listener.Start();
while(true)
{
var client = await listener.AcceptTcpClientAsync().ConfigureAwait(false);
Console.WriteLine($"Accepted connection for port {port}");
var task = HandleClient(client);
}
}
public async static Task Main(string[] args)
{
var task1=AcceptConnections(5000);
var task2=AcceptConnections(5001);
await Task.WhenAll(task1, task2).ConfigureAwait(false);
}
}
This example code creates two listeners for ports 5000 and 5001. Each of it can accept multiple connections and read independently from the socket created.
Maybe it is not "nice", but it works and I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
Accepted connection for port 5000
Accepted connection for port 5000
Accepted connection for port 5001
Received new message in 5 :(17 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(20 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5001 Message from socket-3
Received new message in 8 :(17 bytes):
Port-5001 Message from socket-3
(texts manually edit for clarity, byte lengths are not valid)
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
a C# task, executed by await ... doesn't create a separate thread for the target Task.
One important correction: a task is not "executed" by await. Asynchronous tasks are already in-progress by the time they're returned. await is used by the consuming code to perform an "asynchronous wait"; i.e., pause the current method and resume it when that task has completed.
I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
The task isn't "executed" anywhere. But the code in the async method does have to run, and it has to run on a thread. await captures a "context" when it pauses the method, and when the task completes it uses that context to resume executing the method. Console apps don't have a context, so the method resumes on any available thread pool thread.
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
You usually do not have to know; as long as your code isn't blocking thread pool threads you're generally fine. It's important to note that zero threads are being used while doing I/O, e.g., while listening/accepting a new TCP socket. There's no thread being blocked there. Thread pool threads are only borrowed when they're needed.
For the most part, you don't have to worry about it. But if you need to, the thread pool has several knobs for tweaking.

ChangeFeedProcessorBuilder checkpointing after unsuccessful processing

I was investigating the behavior of a ChangeFeedProcessorBuilder processor1 that throws an exception or goes down while processing the particular change. Upon recovery, the same change will not be picked up anymore. Is there any way to checkpoint only after the successful processing of the notification?
The delegate is as follows:
var builder = container.GetChangeFeedProcessorBuilder("migrationProcessor",
(IReadOnlyCollection<object> input, CancellationToken cancellationToken) =>
{
Console.WriteLine(input.Count + " Changes Received by " + a);
// just first try will fail (static variable)
if (a++ == 0)
{
throw new Exception();
}
return Task.CompletedTask;
});
Thank you!
The default behavior of the Change Feed Processor is to checkpoint after a successful delegate execution: https://learn.microsoft.com/azure/cosmos-db/change-feed-processor#processing-life-cycle
The normal life cycle of a host instance is:
Read the change feed.
If there are no changes, sleep for a predefined amount of time (customizable with WithPollInterval in the Builder) and go to #1.
If there are changes, send them to the delegate.
When the delegate finishes processing the changes successfully, update the lease store with the latest processed point in time and go to #1.
If your delegate handler throws an unhandled exception, there is no checkpoint.
Adding from comments: The only scenario where the batch might not be retried is if the batch that throws is the first ever (lease has no Continuation). Because when the host picks up the lease again to reprocess, it has no point in time to retry from. Based on the official documentation, one lease is owned by a single instance, so there is no way that other instance could have picked up the same lease and be processing it in parallel (within the same Deployment Unit context).

Nullreference exception does not show up when async Task discarded

I have an async Task with a method signature defined like this:
public async Task<bool> HandleFooAsync()
When executing this task in an async way and discarding the results, exceptions happening in this task do not show up in our logs.
_ = _slackResponseService.HandleFooAsync();
When I await the execution of the task I see the error in our logs
var result = await _slackResponseService.HandleFooAsync();
Is this expected behaviour? Is there a way to achieve a solution in between: "do not wait for the result, but log errors nevertheless.." ? We invested hours debugging our logging setup, just to learn that our logging setup is correct, but discard means in dotnet that everything is discarded - even logs. Which is a quite a new perspective for us, coming from a Python background.
Our logging setup follows the default logging setup for dotnet core 3 https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-3.1
Yes, it is an expected behavior. Call in that way can be considered like an anti-pattern. You can read about it C# Async Antipatterns
You need something which is called as "Fire and forget". One of its implementation can be find in repo AsyncAwaitBestPractices (nuget available too).
A Task in .net and netcore is meant to be awaited. If it is not awaited, the scope might be destroyed before the async method has finished.
If you want to run tasks in the background and not wait for a result you can use BackgroundService in .netcore or a third party such as Hangfire which supports fire and forget jobs out of the box
https://medium.com/#daniel.sagita/backgroundservice-for-a-long-running-work-3debe8f8d25b
https://www.hangfire.io/
One solution is to subscribe to the TaskScheduler.UnobservedTaskException event. It is not ideal because the event is raised when the faulted Task is garbage collected, which may happen long after the actual fault.
Another solution could be to use an extension method every time a task is fired and forgotten. Like this:
_slackResponseService.HandleFooAsync().FireAndForget();
Here is a basic implementation of the FireAndForget method:
public async static void FireAndForget(this Task task)
{
try
{
await task;
}
catch (Exception ex)
{
// log the exception here
}
}

How can I cancel/abort a zone in Dart?

I have an http web server that I'm trying to detect long-running requests and abort them. The following code successfully returns to the client upon timeout, but the async zone still continues to run to completion. How can I actually kill the request handler?
var zone = runZoned(() {
var timer = new Timer(new Duration(seconds: Config.longRequestTimeoutSeconds), () {
if (!completer.isCompleted) { // -- not already completed
log.severe('request timed out');
// TODO: This successfully responds to the client early, but it does nothing to abort the zone/handler that is already running.
// Even though the client will never see the result (and won't have to wait for it), the zone/handler will continue to run to completion as normal.
// TODO: Find a way to kill/abort/cancel the zone
completer.complete(new shelf.Response(HttpStatus.SERVICE_UNAVAILABLE, body: 'The server timed out while processing the request'));
}
});
return innerHandler(request) // -- handle request as normal (this may consist of several async futures within)
.then((shelf.Response response) {
timer.cancel(); // -- prevent the timeout intercept
if (!completer.isCompleted) { // -- not already completed (not timed out)
completer.complete(response);
}
})
.catchError(completer.completeError);
});
Nothing can kill running code except itself. If you want code to be interruptible, you need some way to tell it, and the code itself needs to terminate (likely by throwing an error). In this case, the innerHandler needs to be able to interrupt itself when requested. If it's not your code, that might not be possible.
You can write a zone that stops execution of asynchronous events when a flag is set (by modifying Zone.run etc.), but you must be very careful about that - it might never get to an asynchronous finally block and release resources if you start throwing away asynchronous events. So, that's not recommended as a general solution, only for very careful people willing to do manual resource management.

Resources