How do .NET SQL connection pools scale in a multi-node environment? - asp.net

Question

I have some fairly typical SQL calls in an app that look something like this (Dapper typically in the middle), .NET 6:
var connection = new SqlConnection("constring");
using (connection)
{
await connection.OpenAsync();
var command = new SqlCommand("sql");
await command.ExecuteAsync();
await connection.CloseAsync();
connection.Dispose();
}
A request to the app probably generates a half-dozen calls like this, usually returning in <0 to 10ms. I almost never see any SQL usage (it's SQL Azure) beyond a high of 5%.
The problem comes when a bot hits the app with 50+ simultaneous requests, coming all within the same 300 or so milliseconds. This causes the classic error
InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached
I have the following things in place:
I have the connection string set to a max pool size of 250.
I'm running three nodes as an Azure App Service.
The call stacks are all async.
I do have ARR Affinity on because I'm using SignalR, but I assume the load balancer would spread out the requests as the bot likely isn't sending ARR cookies.
The app services and SQL Server do not break a sweat even with these traffic storms.
Here's the question: How do I scale this? I assume human users don't see this and the connection pool exhaustion heals quickly, but it creates a lot of logging noise. The App Service and SQL Server instance are not at all stressed or working beyond their limits, so it appears it's the connection pool mechanics that are a problem. They're kind of a black box abstraction, but a leaky abstraction since I clearly need to know more about them to make them work right.

Answer 1

Here's the question: How do I scale this?
.NET 6 introduced Rate Limiting, which is really the right solution here. Test how many concurrent requests your API app and database can comfortably handle, and stall or reject additional requests.
Take the analogy of an Emergency Room. Do you want to let everyone into the back who walks in the door? No once all the rooms are full, you make them wait in the waiting room, or send them away.
So put in a request throttle like:
builder.Services.AddRateLimiter(options =>
{
options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(httpContext =>
RateLimitPartition.GetFixedWindowLimiter(
partitionKey: httpContext.Request.QueryString.Value!,
factory: partition => new FixedWindowRateLimiterOptions
{
AutoReplenishment = true,
PermitLimit = 50,
QueueLimit = 10,
Window = TimeSpan.FromSeconds(10)
}));
options.OnRejected = (context, cancellationToken) =>
{
context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
return new ValueTask();
};
});