ADO: Async all the way down the tubes? - asynchronous

Question

Okay, so "async all the way down" is the mandate. But when is it problematic?
For example, if you have limited access to a resource, as in a DbConnection or a file, when do you stop using async methods in favor of synchronous?
Let's review the complexity of an asynchronous database call:
(Not putting .ConfigureAwait(false) for readability.)
// Step 1: Ok, no big deal, our connection is closed, let's open it and wait.
await connection.OpenAsync();
// Connection is open! Let's do some work.
// Step 2: Acquire a reader.
using(var reader = await command.ExecuteReaderAsync())
{
// Step 3: Start reading results.
while(await reader.ReadAsync())
{
// get the data.
}
}
Steps:
Should be reasonably innocuous and nothing to worry about.
But now we've acquired an open connection in a potentially limited connection pool. What if when waiting for step 2, other long running tasks are at the head of the line in the task scheduler?
Even worse now, we await with an open connection (and most likely added latency).
Aren't we holding open a connection longer than necessary? Isn't this an undesirable result? Wouldn't it be better to use synchronous methods to lessen the overall connection time, ultimately resulting in our data driven application performing better?
Of course I understand that async doesn't mean faster but async methods provide the opportunity for more total throughput. But as I've observed, there can definitely be weirdness when there are tasks scheduled in-between awaits that ultimately delay the operation, and essentially behave like blocking because of the limitations of the underlying resource.
[Note: this question is focused on ADO, but this also applies to file reads and writes.]
Hoping for some deeper insight. Thank you.

Answer 1

There are a few things to consider here:
Database connection pool limits, specifically the "Max Pool Size" which defaults to 100. The database connection pool has upper limit of the maximum number of connections. Besure to set "Max Pool Size=X" where X is the maximum number of database connections you want to have. This applies to either sync or async.
The thread pool settings. The thread pool will not add threads quickly if you load spikes. It will only add a new thread every 500ms or so. See MSDN Threading Guidelines from 2004 and The CLR Thread Pool 'Thread Injection' Algorithm. Here is a capture of the number of busy threads on one of my projects. The load spiked and requests were delayed due to lack of available threads to service the requests. The line increases as new threads were being added. Remember every thread required 1MB of memory for its stack. 1000 threads ~= 1GB of RAM just for threads.
The load characteristics of your project, relates to the thread pool.
The type of system you are providing, I will assume you are talking about a ASP.NET type app/api
The throughput (requests/sec) vs latency (sec/request) requirements. Async will add to latency but increase throughput.
The database/query performance, relates to the 50ms recommendation below
The article The overhead of async/await in NET 4.5 Edit 2018-04-16 the recommendation below applied to WinRT UI based applications.
Avoid using async/await for very short methods or having await
statements in tight loops (run the whole loop asynchronously instead).
Microsoft recommends that any method that might take longer than 50ms
to return should run asynchronously, so you may wish to use this
figure to determine whether it’s worth using the async/await pattern.
Also take a watch Diagnosing issues in ASP.NET Core Applications - David Fowler & Damian Edwards that talks about issues with thread pool and using async, sync, etc.
Hopefully this helps

Answer 2

if you have limited access to a resource, as in a DbConnection or a file, when do you stop using async methods in favor of synchronous?
You shouldn't need to switch to synchronous at all. Generally speaking, async only works if it's used all the way. Async-over-sync is an antipattern.
Consider the asynchronous code:
using (connection)
{
await connection.OpenAsync();
using(var reader = await command.ExecuteReaderAsync())
{
while(await reader.ReadAsync())
{
}
}
}
In this code, the connection is held open while the command is executed and the data is read. Anytime that the code is waiting on the database to respond, the calling thread is freed up to do other work.
Now consider the synchronous equivalent:
using (connection)
{
connection.Open();
using(var reader = command.ExecuteReader())
{
while(reader.Read())
{
}
}
}
In this code, the connection is held open while the command is executed and the data is read. Anytime that the code is waiting on the database to respond, the calling thread is blocked.
With both of these code blocks, the connection is held open while the command is executed and the data is read. The only difference is that with the async code, the calling thread is freed up to do other work.
What if when waiting for step 2, other long running tasks are at the head of the line in the task scheduler?
The time to deal with thread pool exhaustion is when you run into it. In the vast majority of scenarios, it isn't a problem and the default heuristics work fine.
This is particularly true if you use async everywhere and don't mix in blocking code.
For example, this code would be more problematic:
using (connection)
{
await connection.OpenAsync();
using(var reader = command.ExecuteReader())
{
while(reader.Read())
{
}
}
}
Now you have asynchronous code that, when it resumes, blocks a thread pool thread on I/O. Do that a lot, and you can end up in a thread pool exhaustion scenario.
Even worse now, we await with an open connection (and most likely added latency).
The added latency is miniscule. Like sub-millisecond (assuming no thread pool exhaustion). It's immeasurably small compared to random network fluctuations.
Aren't we holding open a connection longer than necessary? Isn't this an undesirable result? Wouldn't it be better to use synchronous methods to lessen the overall connection time, ultimately resulting in our data driven application performing better?
As noted above, synchronous code would hold the connection open just as long. (Well, OK, a sub-millisecond amount less, but that Doesn't Matter).
But as I've observed, there can definitely be weirdness when there are tasks scheduled in-between awaits that ultimately delay the operation, and essentially behave like blocking because of the limitations of the underlying resource.
It would be worrying if you observed this on the thread pool. That would mean you're already at thread pool exhaustion, and you should carefully review your code and remove blocking calls.
It's less worrying if you observed this on a single-thread scheduler (e.g., UI thread or ASP.NET Classic request context). In that case, you're not at thread pool exhaustion (though you still need to carefully review your code and remove blocking calls).
As a concluding note, it sounds as though you're trying to add async the hard way. It's harder to start at a higher level and work your way to a lower level. It's much easier to start at the lower level and work your way up. E.g., start with any I/O-bound APIs like DbConnection.Open / ExecuteReader / Read, and make those asynchronous first, and then let async grow up through your codebase.

Answer 3

Due to the way database connection pooling works at lower levels of protocol, the high level open / close commands don't have a lot of effect on performance. Generally though the internal thread scheduling IO is usually not a bottleneck unless you have some really long running tasks - we're talking something CPU intensive or worse - blocking inside. This will quickly exhaust your thread pool and things will start queuing up.
I would also suggest you investigate http://steeltoe.io, particularly the circuit breaker hystrix implementation. The way it works is it allows you to group your code into commands, and have command execution managed by command groups, which are essentially dedicated and segregated thread pools. The advantage is if you have a noisy, long running command, it can only exhaust it's own's command group thread pool without affecting the rest of the app. There are many other advantages of this portion of the library, primary being circuit breaker implementation, and one of my personal favorite collapsers. Imagine multiple incoming calls for a query GetObjectById being grouped into a single select * where id in(1,2,3) query and then results mapped back on the separate inbound requests. Db call is just an example, can be anything really.

Answer 4

Significant amounts of iteration introduce significant added latency and extra CPU usage
See http://telegra.ph/SqlDataReader-ReadAsync-vs-Read-04-18 for details.
As suspected:
Using async does not come without cost and requires consideration.
Certain types of operations lend themselves well to async, and others are problematic (for what should be obvious reasons).
High volume synchronous/blocking code has it's downsides, but for the most part is well managed by modern threading:
Testing / Profiling
4 x 100 paralleled queries, 1000 records each query.
Performance Profile for Synchronous Query
Average Query: 00:00:00.6731697, Total Time: 00:00:25.1435656
Performance Profile for Async Setup with Synchronous Read
Average Query: 00:00:01.4122918, Total Time: 00:00:30.2188467
Performance Profile for Fully Async Query
Average Query: 00:00:02.6879162, Total Time: 00:00:32.6702872
Assessment
The above results were run on SQL Server 2008 R2 using a .NET Core 2 console application. I invite anyone who has access to a modern instance of SQL Server to replicate these tests to see if there is a reversal in trend. If you find my testing method flawed, please comment so I correct and retest.
As you can easily see in the results. The more asynchronous operations we introduce, the longer the the queries take, and the longer the total time to complete. Even worse, fully asynchronous uses more CPU overhead which is counter productive to the idea that using async tasks would provide more available thread time. This overhead could be due to how I'm running these tests, but it's important to treat each test in a similar way to compare. Again, if anyone has a way to prove that async is better, please do.
I'm proposing here that "async all the way" has it's limitations and should be seriously scrutinized at certain iterative levels (like file, or data access).