I have a Quarkus application where I use the event bus.
the code in question looks like this:
#ConsumeEvent(value = "execution-request", blocking = true)
#Transactional
#TransactionConfiguration(timeout = 3600)
public void consume(final Message<ExecutionRequest> msg) {
try {
execute(...);
} catch (final Exception e) {
// some logging
}
}
private void execute(...)
throws InterruptedException {
// it actually runs a long running task, but for
// this example this has the same effect
Thread.sleep(65000);
}
Why do I still get a
WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 63066 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked
I'm I doing something wrong? Is the blocking parameter at the ConsumeEvent annotation not enough to let that handle in a separate Worker?
Your annotation is working as designed; the method is running in a worker thread. You can tell by both the name of the thread "vert.x-worker-thread-0", and by the 60 second timeout before the warnings were logged. The eventloop thread only has a 3 second timeout, I believe.
The default Vert.x worker thread pool is not designed for "very" long running blocking code, as stated in their docs:
Warning:
Blocking code should block for a reasonable amount of time (i.e no more than a few seconds). Long blocking operations or polling operations (i.e a thread that spin in a loop polling events in a blocking fashion) are precluded. When the blocking operation lasts more than the 10 seconds, a message will be printed on the console by the blocked thread checker. Long blocking operations should use a dedicated thread managed by the application, which can interact with verticles using the event-bus or runOnContext
That message mentions blocking for more than 10 seconds triggers a warning, but I think that's a typo; the default is actually 60.
To avoid the warning, you'll need to create a dedicated WorkerExecutor (via vertx.createSharedWorkerExecutor) configured with a very high maxExcecuteTime. However, it does not appear you can tell the #ConsumeEvent annotation to use it instead of the default worker pool, so you'd need to manually create an event bus consumer, as well, or use a regular #ConsumeEvent annotation, but call workerExectur.executeBlocking inside of it.
Related
According to
If async-await doesn't create any additional threads, then how does it make applications responsive?
a C# task, executed by await ... doesn't create a separate thread for the target Task. However, I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I still do not understand, what's going on.
public class TestProgram
{
private static async Task HandleClient(TcpClient clt)
{
using NetworkStream ns = clt.GetStream();
using StreamReader sr = new StreamReader(ns);
while (true)
{
string msg = await sr.ReadLineAsync();
Console.WriteLine($"Received in {System.Threading.Thread.CurrentThread.ManagedThreadId} :({msg.Length} bytes):\n{msg}");
}
}
private static async Task AcceptConnections(int port)
{
TcpListener listener = new TcpListener(IPAddress.Parse("127.0.0.1"), port);
listener.Start();
while(true)
{
var client = await listener.AcceptTcpClientAsync().ConfigureAwait(false);
Console.WriteLine($"Accepted connection for port {port}");
var task = HandleClient(client);
}
}
public async static Task Main(string[] args)
{
var task1=AcceptConnections(5000);
var task2=AcceptConnections(5001);
await Task.WhenAll(task1, task2).ConfigureAwait(false);
}
}
This example code creates two listeners for ports 5000 and 5001. Each of it can accept multiple connections and read independently from the socket created.
Maybe it is not "nice", but it works and I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
Accepted connection for port 5000
Accepted connection for port 5000
Accepted connection for port 5001
Received new message in 5 :(17 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(20 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5001 Message from socket-3
Received new message in 8 :(17 bytes):
Port-5001 Message from socket-3
(texts manually edit for clarity, byte lengths are not valid)
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
a C# task, executed by await ... doesn't create a separate thread for the target Task.
One important correction: a task is not "executed" by await. Asynchronous tasks are already in-progress by the time they're returned. await is used by the consuming code to perform an "asynchronous wait"; i.e., pause the current method and resume it when that task has completed.
I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
The task isn't "executed" anywhere. But the code in the async method does have to run, and it has to run on a thread. await captures a "context" when it pauses the method, and when the task completes it uses that context to resume executing the method. Console apps don't have a context, so the method resumes on any available thread pool thread.
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
You usually do not have to know; as long as your code isn't blocking thread pool threads you're generally fine. It's important to note that zero threads are being used while doing I/O, e.g., while listening/accepting a new TCP socket. There's no thread being blocked there. Thread pool threads are only borrowed when they're needed.
For the most part, you don't have to worry about it. But if you need to, the thread pool has several knobs for tweaking.
Consider the normal scenario where an ASP.NET Core Web API application executes the service Controller action, but instead of executing all the work under the same thread (thread pool thread) until the response is created, I would like to use non-pooled threads (ideally pre-created) to execute the main work, either by scheduling one of these threads from the initial action pooled thread and free the pooled thread for serving other incoming requests, or passing the job to a pre-created non-pooled thread.
Among other reasons, the main reason to have these non-pooled and long running threads is that some requests may be prioritized and their threads put on hold (synchronized), thus it would not block new incoming requests to the API due to thread pool starvation, but older requests on hold (non-pooled threads) may be waked up and rejected and some sort of call back to the thread pool to return the web response back to the clients.
In summary, the ideal solution would be using a synchronization mechanism (like .NET RegisterWaitForSingleObject) where the pooled thread would hook to the waitHandle but be freed up for other thread pool work, and a new non-pooled thread would be created or used to carry on the execution. Ideally from a list of pre-created and idle non-pooled threads.
Seems async-await only works with Tasks and threads from the .NET thread pool, not with other threads. Also most techniques to create non-pooled threads do not allow the pooled thread to be free and return to the pool.
Any ideas? I'm using .NET Core and latest versions of tools and frameworks.
Thank you for the comments provided. The suggestion to check TaskCompletionSource was fundamental. So my goal was to have potentially hundreds or thousands of API requests on ASP.NET Core and being able to serve only a portion of them at a given time frame (due to backend constraints), choosing which ones should be served first and hold the others until backends are free or reject them later. Doing all this with thread pool threads is bad: blocking/holding and having to accept thousands in short time (thread pool size growing).
The design goal was the request jobs to move their processing from the ASP.NET threads to non pooled threads. I plan to to have these pre-created in reasonable numbers to avoid the overhead of creating them all the time. These threads implement a generic request processing engine and can be reused for subsequent requests. Blocking these threads to manage request prioritization is not a problem (using synchronization), most of them will not use CPU at all time and the memory footprint is manageable. The most important is that the thread pool threads will only be used on the very start of the request and released right away, to be only be used once the request is completed and return a response to the remote clients.
The solution is to have a TaskCompletionSource object created and passed to an available non-pooled thread to process the request. This can be done by queuing the request data together with the TaskCompletetionSource object on the right queue depending the type of service and priority of the client, or just passing it to a newly created thread if none available. The ASP.NET controller action will await on the TaskCompletionSouce.Task and once the main processing thread sets the result on this object, the rest of the code from the controller action will be executed by a pooled thread and return the response to the client. Meanwhile, the main processing thread can either be terminated or go get more request jobs from the queues.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
namespace MyApi.Controllers
{
[Route("api/[controller]")]
public class ValuesController : Controller
{
public static readonly object locker = new object();
public static DateTime time;
public static volatile TaskCompletionSource<string> tcs;
// GET api/values
[HttpGet]
public async Task<string> Get()
{
time = DateTime.Now;
ShowThreads("Starting Get Action...");
// Using await will free the pooled thread until a Task result is available, basically
// returns a Task to the ASP.NET, which is a "promise" to have a result in the future.
string result = await CreateTaskCompletionSource();
// This code is only executed once a Task result is available: the non-pooled thread
// completes processing and signals (TrySetResult) the TaskCompletionSource object
ShowThreads($"Signaled... Result: {result}");
Thread.Sleep(2_000);
ShowThreads("End Get Action!");
return result;
}
public static Task<string> CreateTaskCompletionSource()
{
ShowThreads($"Start Task Completion...");
string data = "Data";
tcs = new TaskCompletionSource<string>();
// Create a non-pooled thread (LongRunning), alternatively place the job data into a queue
// or similar and not create a thread because these would already have been pre-created and
// waiting for jobs from queues. The point is that is not mandatory to create a thread here.
Task.Factory.StartNew(s => Workload(data), tcs,
CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
ShowThreads($"Task Completion created...");
return tcs.Task;
}
public static void Workload(object data)
{
// I have put this Sleep here to give some time to show that the ASP.NET pooled
// thread was freed and gone back to the pool when the workload starts.
Thread.Sleep(100);
ShowThreads($"Started Workload... Data is: {(string)data}");
Thread.Sleep(10_000);
ShowThreads($"Going to signal...");
// Signal the TaskCompletionSource that work has finished, wich will force a pooled thread
// to be scheduled to execute the final part of the APS.NET controller action and finish.
// tcs.TrySetResult("Done!");
Task.Run((() => tcs.TrySetResult("Done!")));
// The only reason I show the TrySetResult into a task is to free this non-pooled thread
// imediately, otherwise the following line would only be executed after ASP.NET have
// finished processing the response. This briefly activates a pooled thread just execute
// the TrySetResult. If there is no problem to wait for ASP.NET to complete the response,
// we do it synchronosly and avoi using another pooled thread.
Thread.Sleep(1_000);
ShowThreads("End Workload");
}
public static void ShowThreads(string message = null)
{
int maxWorkers, maxIos, minWorkers, minIos, freeWorkers, freeIos;
lock (locker)
{
double elapsed = DateTime.Now.Subtract(time).TotalSeconds;
ThreadPool.GetMaxThreads(out maxWorkers, out maxIos);
ThreadPool.GetMinThreads(out minWorkers, out minIos);
ThreadPool.GetAvailableThreads(out freeWorkers, out freeIos);
Console.WriteLine($"Used WT: {maxWorkers - freeWorkers}, Used IoT: {maxIos - freeIos} - "+
$"+{elapsed.ToString("0.000 s")} : {message}");
}
}
}
}
I have placed the whole sample code so anyone can easily create as ASP.NET Core API project and test it without any changes. Here is the resulting output:
MyApi> Now listening on: http://localhost:23145
MyApi> Application started. Press Ctrl+C to shut down.
MyApi> Used WT: 1, Used IoT: 0 - +0.012 s : Starting Get Action...
MyApi> Used WT: 1, Used IoT: 0 - +0.015 s : Start Task Completion...
MyApi> Used WT: 1, Used IoT: 0 - +0.035 s : Task Completion created...
MyApi> Used WT: 0, Used IoT: 0 - +0.135 s : Started Workload... Data is: Data
MyApi> Used WT: 0, Used IoT: 0 - +10.135 s : Going to signal...
MyApi> Used WT: 2, Used IoT: 0 - +10.136 s : Signaled... Result: Done!
MyApi> Used WT: 1, Used IoT: 0 - +11.142 s : End Workload
MyApi> Used WT: 1, Used IoT: 0 - +12.136 s : End Get Action!
As you can see the pooled thread runs until the await on the TaskCompletionSource creation, and by the time the Workload starts to process the request on the non-pooled thread there is ZERO ThreadPool threads being used and remains using no pooled threads for the entire duration of the processing. When the Run.Task executes the TrySetResult fires a pooled thread for a brief moment to trigger the rest of the controller action code, reason the Worker thread count is 2 for a moment, then a fresh pooled thread runs the rest of the ASP.NET controller action to finish with the response.
We have a Java class that listens to a database (Oracle) queue table and process it if there are records placed in that queue. It worked normally in UAT and development environments. Upon deployment in production, there are times when it cannot read a record from the queue. When a record is inserted, it cannot detect it and the records remain in the queue. This seldom happens but it happens. If I would give statistic, out of 30 records queued in a day, about 8 don't make it. We would need to restart the whole app for it to be able to read the records.
Here is a code snippet of my class..
public class SomeListener implements MessageListener{
public void onMessage(Message msg){
InputStream input = null;
try {
TextMessage txtMsg = (TextMessage) msg;
String text = txtMsg.getText();
input = new ByteArrayInputStream(text.getBytes());
} catch (Exception e1) {
// TODO Auto-generated catch block
logger.error("Parsing from the queue.... failed",e1);
e1.printStackTrace();
}
//process text message
}
}
Weird thing we cant find any traces of exceptions from the logs.
Can anyone help? by the way we set the receiveTimeout to 10 secs
We would need to restart the whole app for it to be able to read the records.
The most common reason for this is the listener thread is "stuck" in user code (//process text message). You can take a thread dump with jstack or jvisualvm or similar to see what the thread is doing.
Another possibility (with low volume apps like this) is the network (most likely a router someplace in the network) silently closes an idle socket because it has not been used for some time. If the container (actually the broker's JMS client library) doesn't know the socket is dead, it will never receive any more messages.
The solution to the first is to fix the code; the solution to the second is to enable some kind of heartbeat or keepalives on the connection so that the network/router does not close the socket when it has no "real" traffic on it.
You would need to consult your broker's documentation about configuring heartbeats/keepalives.
According to the documentation, Task#call() is "invoked when the Task is executed ".
Consider the following program:
import javafx.application.Application;
import javafx.concurrent.Task;
import javafx.stage.Stage;
public class TestTask extends Application {
Long start;
public void start(Stage stage) {
start = System.currentTimeMillis();
new Thread(new Taskus()).start();
}
public static void main(String[] args) {
launch();
}
class Taskus extends Task<Void> {
public Taskus() {
stateProperty().addListener((obs, oldValue, newValue) -> {
try {
System.out.println(newValue + " at " + (System.currentTimeMillis()-start));
} catch (Exception e) {
e.printStackTrace();
}
});
}
public Void call() throws InterruptedException {
for (int i = 0; i < 10000; i++) {
// Could be a lot longer.
}
System.out.println("Some code already executed." + " at " + (System.currentTimeMillis()-start));
Thread.sleep(3000);
return null;
}
}
}
Executing this program gives me the following output:
Some code already executed. after 5 milliseconds
SCHEDULED after 5 milliseconds
RUNNING after 7 milliseconds
SUCCEEDED after 3005 milliseconds
Why is the call() method invoked before the task is even scheduled? This makes no sense to me. In the task where I first saw the issue my task executed a few seconds before the task went into the SCHEDULED state. What if I want to give the user some feedback on the state, and nothing happens until the task has already been executed for a few seconds?
Why is the call() method invoked before the task is even scheduled?
TLDR; version: It's not. It's merely invoked before you get notified that it's been scheduled.
You have two threads running, essentially independently: the thread you explicitly create, and the FX Application Thread. When you start your application thread, it will invoke Taskus.call() on that thread. However, changes to the the task's properties are made on the FX Application Thread via calls to Platform.runLater(...).
So when you call start() on your thread, the following occurs behind the scenes:
A new thread is started
On that thread, an internal call() method in Task is called. That method:
Schedules a runnable to execute on the FX Application Thread, that changes the stateProperty of the task to SCHEDULED
Schedules a runnable to execute on the FX Application Thread, that changes the stateProperty of the task to RUNNING
Invokes your call method
When the FX Application Thread receives the runnable that changes the state of the task from READY to SCHEDULED, and later from SCHEDULED to RUNNING, it effects those changes and notifies any listeners. Since this is on a different thread to the code in your call method, there is no "happens-before" relationship between code in your call method and code in your stateProperty listeners. In other words, there is no guarantee as to which will happen first. In particular, if the FX Application Thread is already busy doing something (rendering the UI, processing user input, processing other Runnables passed to Platform.runLater(...), etc), it will finish those before it makes the changes to the task's stateProperty.
What you are guaranteed is that the changes to SCHEDULED and to RUNNING will be scheduled on the FX Application thread (but not necessarily executed) before your call method is invoked, and that the change to SCHEDULED will be executed before the change to RUNNING is executed.
Here's an analogy. Suppose I take requests from customers to write software. Think of my workflow as the background thread. Suppose I have an admin assistant who communicates with the customers for me. Think of her workflow as the FX Application thread. So when I receive a request from a customer, I tell my admin assistant to email the customer and notify them I received the request (SCHEDULED). My admin assistant dutifully puts that on her "to-do" list. A short while later, I tell my admin assistant to email the customer telling them I have started working on their project (RUNNING), and she adds that to her "to-do" list. I then start working on the project. I do a little work on the project, and then go onto Twitter and post a tweet (your System.out.println("Some code already executed")) "Working on a project for xxx, it's really interesting!". Depending on the number of things already on my assistant's "to-do" list, it's perfectly possible the tweet may appear before she sends the emails to the customer, and so perfectly possible the customer sees that I have started work on the project before seeing the email saying the work is scheduled, even though from the perspective of my workflow, everything occurred in the correct order.
This is typically what you want: the status property is designed to be used to update the UI, so it must run on the FX Application Thread. Since you are running your task on a different thread, you presumably want it to do just that: run in a different thread of execution.
It seems unlikely to me that a change to the scheduled state would be observed a significant amount of time (more than one frame rendering pulse, typically 1/60th second) after the call method actually started executing: if this is happening you are likely blocking the FX Application thread somewhere to prevent it from seeing those changes. In your example, the time delay is clearly minimal (less than a millisecond).
If you want to do something when the task starts, but don't care which thread you do it on, just do that at the beginning of the call method. (In terms of the analogy above, this would be the equivalent of me sending the emails to the customer, instead of requesting that my assistant do it.)
If you really need code in your call method to happen after some user notification has occurred on the FX Application Thread, you need to use the following pattern:
public class Taskus extends Task<Void> {
#Override
public Void call() throws Exception {
FutureTask<Void> uiUpdate = new FutureTask<Void>(() -> {
System.out.println("Task has started");
// do some UI update here...
return null ;
});
Platform.runLater(uiUpdate);
// wait for update:
uiUpdate.get();
for (int i = 0; i < 10000; i++) {
// any VM implementation worth using is going
// to ignore this loop, by the way...
}
System.out.println("Some code already executed." + " at " + (System.currentTimeMillis()-start));
Thread.sleep(3000);
return null ;
}
}
In this example, you are guaranteed to see "Task has started" before you see "Some code already executed". Additionally, since displaying the "Task has started" method happens on the same thread (the FX Application thread) as the changes in state to SCHEDULED and RUNNING, and since displaying the "Task has started" message is scheduled after those changes in state, you are guaranteed to see the transitions to SCHEDULED and RUNNING before you see the "Task has started" message. (In terms of the analogy, this is the same as me asking my assistant to send the emails, and then not starting any work until I know she has sent them.)
Also note that if you replace your original call to
System.out.println("Some code already executed." + " at " + (System.currentTimeMillis()-start));
with
Platform.runLater(() ->
System.out.println("Some code already executed." + " at " + (System.currentTimeMillis()-start)));
then you are also guaranteed to see the calls in the order you are expecting:
SCHEDULED after 5 milliseconds
RUNNING after 7 milliseconds
Some code already executed. after 8 milliseconds
SUCCEEDED after 3008 milliseconds
This last version is the equivalent in the analogy of me asking my assistant to post the tweet for me.
In my application, I'm hosting a fairly CPU-intensive engine on a web server, which is connected to clients via SignalR. From the client, the server will be signalled to do some work (via an AJAX request), and every 200ms will send down a queue of "animation events" which describe the work being done.
This is the code used to set up the connection on the client:
$.connection.hub.start({ transport: ['webSockets', 'serverSentEvents', 'longPolling'] })
And here's the related code in the backend:
private const int PUSH_INTERVAL = 200;
private ManualResetEvent _mrs;
private void SetupTimer(bool running)
{
if (running)
{
UpdateTimer = new Timer(PushEventQueue, null, 0, PUSH_INTERVAL);
}
else
{
/* Lock here to prevent race condition where the final call to PushEventQueue()
* could be followed by the timer calling PushEventQueue() one last time and
* thus the End event would not be the final event to arrive clientside,
* which causes a crash */
_mrs = new ManualResetEvent(false);
UpdateTimer.Dispose(_mrs);
_mrs.WaitOne();
Observer.End();
PushEventQueue(null);
}
}
private void PushEventQueue(object state)
{
SentMessages++;
SignalRConnectionManager<SimulationHub>.PushEventQueueToClient(ConnectionId, new AnimationEventSeries { AnimationPackets = SimulationObserver.EventQueue.FlushQueue(), UpdateTime = DateTime.UtcNow });
}
public static void PushEventQueueToClient(string connectionId, AnimationEventSeries series)
{
HubContext.Clients.Client(connectionId).queue(series);
}
And for completeness' sake, the related Javascript method:
self.hub.client.queue = function(data) {
self.eventQueue.addEvents(data);
};
When testing this functionality on localhost, it works absolutely smoothly, with no delay (as you would expect), using serverSentEvents as a transport method.
However, when used in production, this more often than not takes a very long time to complete. Using SignalR's logging and a bit of my own instrumentation, it can be seen that the first series of events reaches the client within a couple of seconds, which is totally acceptable. However, after that SignalR often gives the following error:
Keep alive has been missed, connection may be dead/slow.
Followed soon after by:
Keep alive timed out. Notifying transport that connection has been lost.
This will happen a few times, and then eventually, up to a minute later, the events will arrive, with my own instrumentation showing that they were sen from the server approximately 200ms apart, as expected. It can also be seen that in production, they were sent with the primary transport method, web sockets.
Is anyone aware of any issues that sending multiple SignalR requests on a timer might cause? Like I say, this primarily seems to happen with web sockets. I've been told that using web sockets is best practice, so I'm keen to keep using them, but if there isn't a workaround to these kinds of issues, then I'm afraid I'll have to remove them permanently.
Edit
I've now removed the option to use web sockets on the live site, and I'm running into the same issues with server sent events - several failed attempts to reconnect after the first queue update arrives.
Summing up our discussion, I don't think there are specific issues with websockets/signalr on azure.
I've sample code here: https://github.com/jonegerton/SignalR.StockTicker which can be used for testing, with some minor tweaks (I'll probably develop it as a test platform at some point).
Its based on the sample project from MS which can be found here: https://github.com/SignalR/SignalR-StockTicker.
I've put an example in azure here (http://stockticker.azurewebsites.net) for testing purposes. It has the default transport configurations enabled (ie websockets >> serversentevents >> longpolling)