Consider the normal scenario where an ASP.NET Core Web API application executes the service Controller action, but instead of executing all the work under the same thread (thread pool thread) until the response is created, I would like to use non-pooled threads (ideally pre-created) to execute the main work, either by scheduling one of these threads from the initial action pooled thread and free the pooled thread for serving other incoming requests, or passing the job to a pre-created non-pooled thread.
Among other reasons, the main reason to have these non-pooled and long running threads is that some requests may be prioritized and their threads put on hold (synchronized), thus it would not block new incoming requests to the API due to thread pool starvation, but older requests on hold (non-pooled threads) may be waked up and rejected and some sort of call back to the thread pool to return the web response back to the clients.
In summary, the ideal solution would be using a synchronization mechanism (like .NET RegisterWaitForSingleObject) where the pooled thread would hook to the waitHandle but be freed up for other thread pool work, and a new non-pooled thread would be created or used to carry on the execution. Ideally from a list of pre-created and idle non-pooled threads.
Seems async-await only works with Tasks and threads from the .NET thread pool, not with other threads. Also most techniques to create non-pooled threads do not allow the pooled thread to be free and return to the pool.
Any ideas? I'm using .NET Core and latest versions of tools and frameworks.
Thank you for the comments provided. The suggestion to check TaskCompletionSource was fundamental. So my goal was to have potentially hundreds or thousands of API requests on ASP.NET Core and being able to serve only a portion of them at a given time frame (due to backend constraints), choosing which ones should be served first and hold the others until backends are free or reject them later. Doing all this with thread pool threads is bad: blocking/holding and having to accept thousands in short time (thread pool size growing).
The design goal was the request jobs to move their processing from the ASP.NET threads to non pooled threads. I plan to to have these pre-created in reasonable numbers to avoid the overhead of creating them all the time. These threads implement a generic request processing engine and can be reused for subsequent requests. Blocking these threads to manage request prioritization is not a problem (using synchronization), most of them will not use CPU at all time and the memory footprint is manageable. The most important is that the thread pool threads will only be used on the very start of the request and released right away, to be only be used once the request is completed and return a response to the remote clients.
The solution is to have a TaskCompletionSource object created and passed to an available non-pooled thread to process the request. This can be done by queuing the request data together with the TaskCompletetionSource object on the right queue depending the type of service and priority of the client, or just passing it to a newly created thread if none available. The ASP.NET controller action will await on the TaskCompletionSouce.Task and once the main processing thread sets the result on this object, the rest of the code from the controller action will be executed by a pooled thread and return the response to the client. Meanwhile, the main processing thread can either be terminated or go get more request jobs from the queues.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
namespace MyApi.Controllers
{
[Route("api/[controller]")]
public class ValuesController : Controller
{
public static readonly object locker = new object();
public static DateTime time;
public static volatile TaskCompletionSource<string> tcs;
// GET api/values
[HttpGet]
public async Task<string> Get()
{
time = DateTime.Now;
ShowThreads("Starting Get Action...");
// Using await will free the pooled thread until a Task result is available, basically
// returns a Task to the ASP.NET, which is a "promise" to have a result in the future.
string result = await CreateTaskCompletionSource();
// This code is only executed once a Task result is available: the non-pooled thread
// completes processing and signals (TrySetResult) the TaskCompletionSource object
ShowThreads($"Signaled... Result: {result}");
Thread.Sleep(2_000);
ShowThreads("End Get Action!");
return result;
}
public static Task<string> CreateTaskCompletionSource()
{
ShowThreads($"Start Task Completion...");
string data = "Data";
tcs = new TaskCompletionSource<string>();
// Create a non-pooled thread (LongRunning), alternatively place the job data into a queue
// or similar and not create a thread because these would already have been pre-created and
// waiting for jobs from queues. The point is that is not mandatory to create a thread here.
Task.Factory.StartNew(s => Workload(data), tcs,
CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
ShowThreads($"Task Completion created...");
return tcs.Task;
}
public static void Workload(object data)
{
// I have put this Sleep here to give some time to show that the ASP.NET pooled
// thread was freed and gone back to the pool when the workload starts.
Thread.Sleep(100);
ShowThreads($"Started Workload... Data is: {(string)data}");
Thread.Sleep(10_000);
ShowThreads($"Going to signal...");
// Signal the TaskCompletionSource that work has finished, wich will force a pooled thread
// to be scheduled to execute the final part of the APS.NET controller action and finish.
// tcs.TrySetResult("Done!");
Task.Run((() => tcs.TrySetResult("Done!")));
// The only reason I show the TrySetResult into a task is to free this non-pooled thread
// imediately, otherwise the following line would only be executed after ASP.NET have
// finished processing the response. This briefly activates a pooled thread just execute
// the TrySetResult. If there is no problem to wait for ASP.NET to complete the response,
// we do it synchronosly and avoi using another pooled thread.
Thread.Sleep(1_000);
ShowThreads("End Workload");
}
public static void ShowThreads(string message = null)
{
int maxWorkers, maxIos, minWorkers, minIos, freeWorkers, freeIos;
lock (locker)
{
double elapsed = DateTime.Now.Subtract(time).TotalSeconds;
ThreadPool.GetMaxThreads(out maxWorkers, out maxIos);
ThreadPool.GetMinThreads(out minWorkers, out minIos);
ThreadPool.GetAvailableThreads(out freeWorkers, out freeIos);
Console.WriteLine($"Used WT: {maxWorkers - freeWorkers}, Used IoT: {maxIos - freeIos} - "+
$"+{elapsed.ToString("0.000 s")} : {message}");
}
}
}
}
I have placed the whole sample code so anyone can easily create as ASP.NET Core API project and test it without any changes. Here is the resulting output:
MyApi> Now listening on: http://localhost:23145
MyApi> Application started. Press Ctrl+C to shut down.
MyApi> Used WT: 1, Used IoT: 0 - +0.012 s : Starting Get Action...
MyApi> Used WT: 1, Used IoT: 0 - +0.015 s : Start Task Completion...
MyApi> Used WT: 1, Used IoT: 0 - +0.035 s : Task Completion created...
MyApi> Used WT: 0, Used IoT: 0 - +0.135 s : Started Workload... Data is: Data
MyApi> Used WT: 0, Used IoT: 0 - +10.135 s : Going to signal...
MyApi> Used WT: 2, Used IoT: 0 - +10.136 s : Signaled... Result: Done!
MyApi> Used WT: 1, Used IoT: 0 - +11.142 s : End Workload
MyApi> Used WT: 1, Used IoT: 0 - +12.136 s : End Get Action!
As you can see the pooled thread runs until the await on the TaskCompletionSource creation, and by the time the Workload starts to process the request on the non-pooled thread there is ZERO ThreadPool threads being used and remains using no pooled threads for the entire duration of the processing. When the Run.Task executes the TrySetResult fires a pooled thread for a brief moment to trigger the rest of the controller action code, reason the Worker thread count is 2 for a moment, then a fresh pooled thread runs the rest of the ASP.NET controller action to finish with the response.
Related
We have a .NET application which is calling over OpenRia services on the server (IIS). This web service call is running a heavy calculation, where we are loading over LoadLibrary some DLL's, which we need to solve some linear systems. We need to go over a list of 1000 events. Every single event is a separate calculation and can be run independently from each other.
What we are doing is, that we create on a 64-core machine 60 tasks and every task is taking one event => run the calculation => take the next event => run the calculation and so on until the list is empty.
As soon the list is empty our calculation is finished.
We have now the strange behaviour that on the first run the calculation seems to run fast, but when we run the same calculation again it's getting slower on every run.
If we restart the server the calculation is running fast again.
We have done an analysis with PerfView and we have seen that on the second/third/fourth run the used threads from the IIS worker process are less than at the beginning.
On the first run the IIS worker process is using 60 threads (as we have defined) and on the second the process is using less than 60. On every run the actual threads used are less and less.
The first run the calculation needs around 3min. The second run we need 6min and the third run we are already around 15min.
What could be the problem? I have tried to use the ThreadPool, but I have the same effect as with the Tasks.
Here is some sample code:
//This part of code is called after the web service call
ConcurrentStack<int> events = new ConcurrentStack<int>();//This is a list of 1000 entries
ParallelOptions options = new ParallelOptions();
int interfacesDone = 0;
Task[] tasks = new Task[options.MaxDegreeOfParallelism];
for (int i = 0; i < options.MaxDegreeOfParallelism; i++)
{
tasks[i] = Task.Run(() =>
{
StartAnalysis(events);
});
}
Task.WaitAll(tasks);
private void StartAnalysis(ConcurrentStack<int> events)
{
while (!events.IsEmpty)
{
int index;
if (events.TryPop(out index))
{
DoHeavyCalculation();
}
}
}
ASP.NET processes requests by using threads from the .NET thread pool. The thread pool maintains a pool of threads that have already incurred the thread initialization costs.
Therefore, these threads are easy to reuse. The .NET thread pool is also self-tuning. It monitors CPU and other resource utilization, and it adds new threads or trims the thread pool size as needed.
I have a Quarkus application where I use the event bus.
the code in question looks like this:
#ConsumeEvent(value = "execution-request", blocking = true)
#Transactional
#TransactionConfiguration(timeout = 3600)
public void consume(final Message<ExecutionRequest> msg) {
try {
execute(...);
} catch (final Exception e) {
// some logging
}
}
private void execute(...)
throws InterruptedException {
// it actually runs a long running task, but for
// this example this has the same effect
Thread.sleep(65000);
}
Why do I still get a
WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 63066 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked
I'm I doing something wrong? Is the blocking parameter at the ConsumeEvent annotation not enough to let that handle in a separate Worker?
Your annotation is working as designed; the method is running in a worker thread. You can tell by both the name of the thread "vert.x-worker-thread-0", and by the 60 second timeout before the warnings were logged. The eventloop thread only has a 3 second timeout, I believe.
The default Vert.x worker thread pool is not designed for "very" long running blocking code, as stated in their docs:
Warning:
Blocking code should block for a reasonable amount of time (i.e no more than a few seconds). Long blocking operations or polling operations (i.e a thread that spin in a loop polling events in a blocking fashion) are precluded. When the blocking operation lasts more than the 10 seconds, a message will be printed on the console by the blocked thread checker. Long blocking operations should use a dedicated thread managed by the application, which can interact with verticles using the event-bus or runOnContext
That message mentions blocking for more than 10 seconds triggers a warning, but I think that's a typo; the default is actually 60.
To avoid the warning, you'll need to create a dedicated WorkerExecutor (via vertx.createSharedWorkerExecutor) configured with a very high maxExcecuteTime. However, it does not appear you can tell the #ConsumeEvent annotation to use it instead of the default worker pool, so you'd need to manually create an event bus consumer, as well, or use a regular #ConsumeEvent annotation, but call workerExectur.executeBlocking inside of it.
According to
If async-await doesn't create any additional threads, then how does it make applications responsive?
a C# task, executed by await ... doesn't create a separate thread for the target Task. However, I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I still do not understand, what's going on.
public class TestProgram
{
private static async Task HandleClient(TcpClient clt)
{
using NetworkStream ns = clt.GetStream();
using StreamReader sr = new StreamReader(ns);
while (true)
{
string msg = await sr.ReadLineAsync();
Console.WriteLine($"Received in {System.Threading.Thread.CurrentThread.ManagedThreadId} :({msg.Length} bytes):\n{msg}");
}
}
private static async Task AcceptConnections(int port)
{
TcpListener listener = new TcpListener(IPAddress.Parse("127.0.0.1"), port);
listener.Start();
while(true)
{
var client = await listener.AcceptTcpClientAsync().ConfigureAwait(false);
Console.WriteLine($"Accepted connection for port {port}");
var task = HandleClient(client);
}
}
public async static Task Main(string[] args)
{
var task1=AcceptConnections(5000);
var task2=AcceptConnections(5001);
await Task.WhenAll(task1, task2).ConfigureAwait(false);
}
}
This example code creates two listeners for ports 5000 and 5001. Each of it can accept multiple connections and read independently from the socket created.
Maybe it is not "nice", but it works and I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
Accepted connection for port 5000
Accepted connection for port 5000
Accepted connection for port 5001
Received new message in 5 :(17 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(20 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5001 Message from socket-3
Received new message in 8 :(17 bytes):
Port-5001 Message from socket-3
(texts manually edit for clarity, byte lengths are not valid)
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
a C# task, executed by await ... doesn't create a separate thread for the target Task.
One important correction: a task is not "executed" by await. Asynchronous tasks are already in-progress by the time they're returned. await is used by the consuming code to perform an "asynchronous wait"; i.e., pause the current method and resume it when that task has completed.
I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
The task isn't "executed" anywhere. But the code in the async method does have to run, and it has to run on a thread. await captures a "context" when it pauses the method, and when the task completes it uses that context to resume executing the method. Console apps don't have a context, so the method resumes on any available thread pool thread.
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
You usually do not have to know; as long as your code isn't blocking thread pool threads you're generally fine. It's important to note that zero threads are being used while doing I/O, e.g., while listening/accepting a new TCP socket. There's no thread being blocked there. Thread pool threads are only borrowed when they're needed.
For the most part, you don't have to worry about it. But if you need to, the thread pool has several knobs for tweaking.
I need to use a Service which starts a Task more than once (= the same Service must run several parallelised Task). I read the JavaFX documentation, and they seem to say that a Service can run only one Task at once.
So if I call twice start with my Service object, the first Task returned by its createTask method would be stopped, as if I used restart after the first start.
However, that's not clear. As I told you, the documentation seems to tell that.
Indeed :
A Service creates and manages a Task that performs the work on the background thread.
Note that I could think they also say that a Service can have several Task started at the same time. Indeed :
a Service can be constructed declaratively and restarted on demand.
My question is : if I use N start in a row, will N Tasks be created AND KEEP EACH RUNNING ?
"If I use N start in a row, will N Tasks be created AND KEEP EACH RUNNING ?
In short, no.
"If I call start twice with my Service object..."
From the Javadocs:
public void start()
Starts this Service. The Service must be in the READY state to succeed in this call.
So if you call start() a second time without previously calling reset(), you will just get an exception. You can only call reset() if the Service is not in a RUNNING or SCHEDULED state. You can call restart(), which will have the effect of first canceling any current task, and then restarting the service. (This is what is meant by the documentation that says the "service can be restarted on demand".)
The net result of this is that a service cannot have two currently running tasks at the same time, since there is no sequence of calls that can get to that situation without throwing an IllegalStateException.
If you want multiple tasks running at once, simply create them yourself and submit them to an executor (or run each in its own thread, but an executor is preferred):
private final Executor exec = Executors.newCachedThreadPool(runnable -> {
Thread t = new Thread(runnable);
t.setDaemon(true);
return t ;
});
// ...
private void launchTask() {
Task<MyDataType> task = new Task<MyDataType>(){
#Override
protected Something call() {
// do work...
return new MyDataType(...);
}
};
task.setOnSucceeded(e -> { /* update UI ... */ });
task.setOnFailed(e -> { /* handle error ... */ });
exec.execute(task);
}
I am working on an asp.net mvc 5 web application , deployed inside IIS-8, and i have a method inside my application to perform a long running task which mainly scans our network for servers & VMs and update our database with the scan results. method execution might last between 30-40 minutes to complete on production environment. and i am using a schedule tool named Hangfire which will call this method 2 times a day.
here is the job definition inside the startup.cs file, which will call the method at 8:01 am & 8:01 pm:-
public void Configuration(IAppBuilder app)
{
var options = new SqlServerStorageOptions
{
PrepareSchemaIfNecessary = false
};
GlobalConfiguration.Configuration.UseSqlServerStorage("scanservice",options);
RecurringJob.AddOrUpdate(() => ss.Scan(), "01 8,20 ***");
}
and here is the method which is being called twice a day by the schedule tool:-
public void Scan()
{
Service ss = new Service();
ss.NetworkScan().Wait();
}
Finally the method which do the real scan is (i only provide a high level description of what the method will do):-
public async Task<ScanResult> NetworkScan()
{
// retrieve the server info from the DB
// loop over all servers & then execute some power shell commands to scan the network & retrieve the info for each server one by one...
// after the shell command completed for each server, i will update the related server info inside the DB
currently i did some tests on our test environment and every thing worked well ,, where the scan took around 25 seconds to scan 2 test servers.but now we are planning to move the application to production and we have around 120++ servers to scan. so i estimate the method execution to take around 30 -40 minutes to complete on the production environment. so my question is how i can make sure that this execution will never expire , and the ScanNetwork() method will complete till the end?
Instead of worrying about your task timing out, perhaps you could start a new task for each server. In this way each task will be very short lived, and any exceptions caused by scanning a single server will not effect all the others. Additionally, if your application is restarted in IIS any scans which were not yet completed will be resumed. With all scans happening in one sequential task this is not possible. You will likely also see the total time to complete a scan of your entire network plummet, as the majority of time would likely be spent waiting on remote servers.
public void Scan()
{
Service ss = new Service();
foreach (var server in ss.GetServers())
{
BackgroundJob.Enqueue<Service>(s => s.ServerScan(server));
}
}
Now your scheduled task will simply enqueue one new task for each server.