IIS 7.0 503 errors with generic handler (.ashx) implementing IHttpAsyncHandler - asp.net

Question

I'm running into some performance issues using a generic handler that implements IHttpAsyncHandler. At its simplest, the handler receives a GET request, and 20 seconds later ends the response after writing '< timeout / >' to the response.
When hammering the .ashx with 10000-20000 simultaneous requests, it fails with 503 server unavailable after precisely 5000 requests. When switching to synchronous mode, and ending the request immediately, the problem goes away.
I've tinkered with a number of settings, yet the only thing I've managed to acheive is lower the request threshold at which this error occurs.
Here's a rundown of the settings I've toyed with:
machine.config:
<configuration>
...
<system.web>
...
<processModel enable="true" requestQueueLimit="10000"/>
...
web.config:
<configuration>
...
<system.web>
...
<httpRuntime enable="true" appRequestQueueLimit="10000"/>
...
IIS Manager > ApplicationPools > Advanced Settings
Queue Length : 65535
Although I can't be sure, it seems like these settings work good and fine if the requests are synchronous, but when async, I can't seem to get beyond exactly 5000 requests before the server starts telling me to go away. If I set things lower (can't remember exactly which setting that would be from the above, but I've tried them all), then the 503 count goes up accordingly, but I can never stop it happening beyond 5000 when under serious load.
It seems that there a a number of settings scattered in a myriad of places that might affect this, but the 5000 seems fairly set in stone. I see here that appRequestQueueLimit cannot exceed 5000, but can find no further info about this, and wonder if this is misinformation.
Is there any kind of "flood-control" setting in IIS that might be limiting a single host to no more than 5000 requests? How can I get IIS to handle more that 5000 concurrent asynchronous requests?
Edit2: Are there any counters or other indicators of which limit might be being exceeded, and how would I investigate further?
Edit: Here's the loadgenerator code:
using System;
using System.Net;
using System.Threading;
namespace HammerTime
{
class Program
{
private static int counter = 0;
static void Main(string[] args)
{
var limit = 5000;
ServicePointManager.DefaultConnectionLimit=limit;
for (int i = 0; i < limit;++i )
{
StartWebRequest(i.ToString());
}
Console.ReadLine();
}
private static void StartWebRequest(string channelId)
{
string uri = "http://spender2008/test/Test.ashx?channel="+channelId;
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
request.BeginGetResponse(responseHandler, request);
}
private static void responseHandler(IAsyncResult ar)
{
try
{
HttpWebRequest state = (HttpWebRequest)ar.AsyncState;
HttpWebResponse response = (HttpWebResponse)state.EndGetResponse(ar);
}
catch(Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
Console.WriteLine(Interlocked.Increment(ref counter));
}
}
}
}

Answer 1

OK. Fixed... many thanks to this post for clearing up a few details.
To eliminate the 503 errors required 3 different config changes:
machine.config:
<configuration>
...
<system.web>
...
<processModel enable="true" requestQueueLimit="100000"/>
IIS Manager > ApplicationPools > Advanced Settings
Queue Length : 65535
and finally (the missing piece of the puzzle), the command line:
appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:100000
The web.config setting mentioned in the main post was irrelevant.
10000 concurrent connections, no problems. Thanks for help!