I've tried to set the executionTimeout in the web.config file:
<compilation debug="false" targetFramework="4.5">
<httpRuntime executionTimeout="30"/>
Looking at the IIS Manager Requests page I can see the requests are not being terminated after 30 seconds.
Should I implement a Timer inside my IHttpAsyncHandler?
With the apparent lack of built-in support for IHttpAsyncHandler timeouts, presumably you must manage your own timeout. Perhaps this is by design; after all you are choosing an asynchronous pattern - who does MSFT think they are trying to set a default timeout for your long running task?
What I would do is use ThreadPool.RegisterWaitForSingleObject to manage your polling with an appropriate timeout. Here is a code sample I use to avoid waiting on a web service that never returns:
private const int REQUEST_TIMEOUT = 30000; // miliseconds (30 sec)
private void CallService()
{
try {
string url = "somewebservice.com";
WebRequest request = WebRequest.Create(url);
// Asynchronously fire off the request
IAsyncResult result = request.BeginGetResponse(new AsyncCallback(MyRoutineThatUsesTheResults), request);
// Handle timed-out requests
ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, new WaitOrTimerCallback(RequestTimeout), request, REQUEST_TIMEOUT, true);
}
catch (Exception ex) {
_logger.Error("Error during web service request.", ex);
}
private void RequestTimeout(object state, bool timedOut)
{
if (timedOut) {
WebRequest request = (WebRequest)state;
_logger.WarnFormat("Request to {0} timed out (> {1} sec)", request.RequestUri.ToString(), REQUEST_TIMEOUT / 1000);
request.Abort();
}
}
You will need an IAsyncResult to work with this approach but that's an established pattern you shouldn't have trouble running down samples about.
Also, you will run into issues when IIS decides to recycle your app pool / tear down your app domain while your polling is still running. If that's a condition you want to handle, you can use HostingEnvironment.RegisterObject.
You can try to add this to your web.config file:
<system.web>
<pages asyncTimeout="30" />
</system.web>
Its for PageAsyncTask, but just might be honored for IHttpAsyncHandler too?
If not, you may have more luck with the new HttpTaskAsyncHandler in ASP.Net 4.5 version:
http://www.asp.net/vnext/overview/whitepapers/whats-new#_Toc318097378
You would have to use RegisterAsyncTask check the link below
http://msdn.microsoft.com/en-us/magazine/cc163725.aspx
Related
We have a simple application in ASP.NET Core which calls a website and returns the content. The Controller method looks like this:
[HttpGet("test/get")]
public ActionResult<string> TestGet()
{
var client = new WebClient
{
BaseAddress = "http://v-dev-a"
};
return client.DownloadString("/");
}
The URL which we call is just the default page of an IIS. I am using Apache JMeter to test 1000 requests in 10 seconds. I have always the same issue, after about 300-400 requests it gets stuck for a few minutes and nothing works. The appplication which holds the controller is completely frozen.
In the performance monitor (MMC) I see that the connection are at 100%.
I tried the same code with ASP.NET 4.7.2 and it runs without any issues.
I also read about that the dispose of the WebClient does not work and I should make it static. See here
Also the deactivation of the KeepAlive did not help:
public class QPWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
((HttpWebRequest)request).KeepAlive = false;
}
return request;
}
}
The HttpClient hast the same issue, it does not change anything
With dependency injection like recommended here there is an exception throw that the web client can't handle more request at the same time.
Another unsuccessful try was to change ConnectionLimit and SetTcpKeepAlive in ServicePoint
So I am out of ideas how to solve this issue. Last idea is to move the application to ASP.NET. Does anyone of you have an idea or faced already the same issue?
I'm getting some issues with SignalR (1.1.2) trying to create a basic realtime chat setup and after spending about a week on it (including trawling through the SignalR source) I'm sort of at the end of what I can try...
I have (I think) a rather complicated SignalR setup consisting of:
Load balanced servers
Redis message bus
Two sites on each server (ASP.NET Webforms VB.NET desktop site and MVC3 C# mobile site)
Each of the sites includes the hub of itself and the other site, so each page can send messages to each site.
Looking into the Chrome inspector (in this example on the mobile site), the hubs are both loaded, the negotiate step for mobile is successful but the connect attempt fails after 3 seconds with the error:
EventSource's response has a MIME type ("text/html") that is not "text/event-stream". Aborting the connection.
which is of course our custom 500 error page after Microsoft.Owin.Host.SystemWeb has thrown:
The connection id is in the incorrect format.
Once this happens, most of the time this will then get into some sort of weird loop where it will continue to throw hundreds of these errors and send off lots of pings followed by a longPolling connect
The solution works perfectly well in my development environment (single IIS instance) but moving to the load balanced test environment is where I see the errors.
I don't know if there's anything else I can add that may help but I'm happy to add it.
I've added the following to the web.config files on both sites:
<validation validateIntegratedModeConfiguration="false"/>
<modules runAllManagedModulesForAllRequests="true"/>
and
<add name="Access-Control-Allow-Origin" value="*"></add>
<add name="Access-Control-Allow-Headers" value="Content-Type" />
The global.asax files have:
protected void Application_Start()
{
AreaRegistration.RegisterAllAreas();
RegisterGlobalFilters(GlobalFilters.Filters);
RegisterRoutes(RouteTable.Routes);
RedisScaleoutConfiguration redisConfig = new RedisScaleoutConfiguration([redisIP], [port], String.Empty, "Name");
redisConfig.Database = 9;
GlobalHost.DependencyResolver.UseRedis(redisConfig);
}
public static void RegisterRoutes(RouteCollection routes)
{
routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
HubConfiguration hubConfig = new HubConfiguration();
hubConfig.EnableCrossDomain = true;
hubConfig.EnableDetailedErrors = true;
RouteTable.Routes.MapHubs(hubConfig);
<snip>
}
The JS code I have is along the lines of:
function setUpSignalR() {
//Set up the connections
webConnection = $.hubConnection(pageInfo.webUrl);
mobConnection = $.hubConnection(pageInfo.mobUrl);
//Get the hubs for web and mobile
webHub = webConnection.createHubProxies().messagingHub;
mobHub = mobConnection.createHubProxies().messagingHub;
//Hook up the call back functions
<snip>
//Now, start it up!
mobConnection.logging = true;
mobConnection.start().done(function() {
mobHub.server.joinConversation(pageInfo.conversationGuid, "mobile").fail(function (error) { console.log('JoinConversation for mobile connection failed. Error: ' + error); });
webConnection.start().done(function() {
webHub.server.joinConversation(pageInfo.conversationGuid, "mobile").fail(function (error) { console.log('JoinConversation for web connection failed. Error: ' + error); });
});
});
}
From the SignalR troubleshooting document:
"The connection ID is in the incorrect format" or "The user identity
cannot change during an active SignalR connection" error
This error may be seen if authentication is being used, and the client
is logged out before the connection is stopped. The solution is to
stop the SignalR connection before logging the client out.
I am just about to launch my ASP.NET MVC3 web app to production, however, as a complex app, it takes a LONG time to start up. Obviously, I don't want my users waiting over a minute for their first request to go through after the AppPool has timed out.
From my research, i've found that there are two ways to combat this:
Run a worker role or other process - which poll's the website every 19 minutes preventing the warm up.
Change the timeout from the default 20 minutes - To something much larger.
As Solution 2 seems like the better idea, i just wondered what the disadvantages would be of this, will I run out of memory etc.?
Thanks.
Could you use the auto-start feature of IIS? There is a post here that presents this idea.
You'd have IIS 7.5 and Win2k8 R2 with Azure OS family 2. You'd just need to be able to script/automate any setup steps and configuration.
I do this with a background thread that requests a keepalive URL every 15 minutes. Not only does this keep the app from going idle, but it also warms up the app right away anytime the web role or virtual machine restarts or is rebuilt.
This is all possible because Web Roles really are just Worker Roles that also do IIS stuff. So you can still use all the standard Worker Role startup hooks in a Web Role.
I got the idea from this blog post but tweaked the code to do a few extra warmup tasks.
First, I have a class that inherits from RoleEntryPoint (it does some other things besides this warm up task and I removed them for simplicity):
public class WebRole : RoleEntryPoint
{
// other unrelated member variables appear here...
private WarmUp _warmUp;
public override bool OnStart()
{
// other startup stuff appears here...
_warmUp = new WarmUp();
_warmUp.Start();
return base.OnStart();
}
}
All the actual warm up logic is in this WarmUp class. When it first runs it hits a handful of URLs on the local instance IP address (vs the public, load balanced hostname) to get things in memory so that the first people to use it get the fastest possible response time. Then, it loops and hits a single keepalive URL (again on the local role instance) that doesn't do any work and just serves to make sure that IIS doesn't shut down the application pool as idle.
public class WarmUp
{
private Thread worker;
public void Start()
{
worker = new Thread(new ThreadStart(Run));
worker.IsBackground = true;
worker.Start();
}
private void Run()
{
var endpoint = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["http"]; // "http" has to match the endpointName in your ServiceDefinition.csdef file.
var pages = new string[]
{
"/",
"/help",
"/signin",
"/register",
"/faqs"
};
foreach (var page in pages)
{
try
{
var address = String.Format("{0}://{1}:{2}{3}",
endpoint.Protocol,
endpoint.IPEndpoint.Address,
endpoint.IPEndpoint.Port,
page);
var webClient = new WebClient();
webClient.DownloadString(address);
Debug.WriteLine(string.Format("Warmed {0}", address));
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
}
}
var keepalive = String.Format("{0}://{1}:{2}{3}",
endpoint.Protocol,
endpoint.IPEndpoint.Address,
endpoint.IPEndpoint.Port,
"/keepalive");
while (true)
{
try
{
var webClient = new WebClient();
webClient.DownloadString(keepalive);
Debug.WriteLine(string.Format("Pinged {0}", keepalive));
}
catch (Exception ex)
{
//absorb
}
Thread.Sleep(900000); // 15 minutes
}
}
}
Personally I'd change the timeout, but both should work: effectively they would both have the same effect of preventing the worker processes from shutting down.
I believe the timeout is there to avoid IIS retaining resources that aren't needed for servers with lots of Web sites that are lightly used. Given that heavily used sites (like this one!) don't shut down their worker processes I don't think you'll see any memory issues.
We just recently noticed that executionTimeout has stopped working on our website. It was definitely working ~last year ... hard to say when it stopped.
We are currently running on:
Windows-2008x64
IIS7
32bit binaries
Managed Pipeline Mode = classic
Framework version = v2.0
Web.Config has
<compilation defaultLanguage="vb" debug="false" batch="true">
<httpRuntime executionTimeout="90" />
Any hints on why we are seeing Timetaken all the way up to ~20 minutes. Would compilation options for DebugType (full vs pdbonly) have any effect?
datetime timetaken httpmethod Status Sent Received<BR>
12/19/10 0:10 901338 POST 302 456 24273<BR>
12/19/10 0:18 1817446 POST 302 0 114236<BR>
12/19/10 0:16 246923 POST 400 0 28512<BR>
12/19/10 0:12 220450 POST 302 0 65227<BR>
12/19/10 0:22 400150 GET 200 180835 416<BR>
12/19/10 0:20 335455 POST 400 0 36135<BR>
12/19/10 0:57 213210 POST 302 0 51558<BR>
12/19/10 0:48 352742 POST 302 438 25802<BR>
12/19/10 0:37 958660 POST 400 0 24558<BR>
12/19/10 0:06 202025 POST 302 0 58349<BR>
Execution timeout and time-taken time two different things. Although, the size of the discrepancy is troubling.
time-taken includes all of the network time in the request/response (under certain conditions.). The network transfer time easily outstrips the amount of time a request really takes. Though, normally, I'm used to just seconds of difference not minutes.
Execution timeout refers only to the amount of time the worker process spent processing the request; which is just a subset of time-taken. It only applies if the debug attribute is set to false; which it looks like you have.
Of course, assuming the first request you listed took the full 90 seconds of allowed time out, that still leaves 13.5 minutes left in the time-taken window to transfer essentially 24k of data. That sounds like a serious network issue.
So, either you have a serious transport issue or there is another web.config file somewhere in the tree where the requests are being processed that either sets debug to true or increases the execution timeout to something astronomical.
Another possibility is that the page itself has either the debug attribute set or it's own timeout values.
I have a theory but I'm not sure how to prove it. I've done something similar to cbcolin and logged the time when the request starts from within the BeginRequest event handler. Then when the request times out (1 hour later in our case) it is logged in the database and a timestamp recorded.
So here is the theory: ASP.NET only counts time that the thread is actually executing, not time that it is asleep.
So after BeginRequest the thread goes to sleep until the entire POST body is received by IIS. Then the thread is woken up to do work and the executionTimeout clock starts running. So time spent in the network transmission phase is not counted against the executionTimeout. Eventually the site wide Connection Timeout is hit and IIS closes the connection, resulting in an exception in ASP.NET.
BeginRequest and even PreRequestHandlerExecute all get called before the POST body is transferred to the web server. Then there is a long gap before the request handler is called. So it may look like .NET had the request for 30 minutes but the thread wasn't running that long.
I'm going to start logging the time that the request handler actually starts running and see if it ever goes over the limit I set.
Now as to control how long a request can stay in the transmittions phase like this on a per URL basis I have no idea. On a global level we can set minBytesPerSecond in webLimits for the application. There is no UI for it that I can find. This should kick ultra slow clients in the transmission phase.
That still wont solve the problem for DoS attacks that actually send data.
I came across this article 2 days ago when I had the same problem. I tried everything, it worked on my local machine but did not work on the production server. Today, I have a workaround that fixes the problem and would like to share. Microsoft seems to not apply timeout to IHttpAsyncHandler and I take advantage of that. On my system, I only have 1 handler that is time-consuming, so this solution works for me.
My handler code looks like this:
public class Handler1 : IHttpAsyncHandler
{
public bool IsReusable
{
get { return true; }
}
public void ProcessRequest(HttpContext context)
{ }
public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData)
{
//My business logic is here
AsynchOperation asynch = new AsynchOperation(cb, context, extraData);
asynch.StartAsyncWork();
return asynch;
}
public void EndProcessRequest(IAsyncResult result)
{ }
}
And my helper class:
class AsynchOperation : IAsyncResult
{
private bool _completed;
private Object _state;
private AsyncCallback _callback;
private HttpContext _context;
bool IAsyncResult.IsCompleted { get { return _completed; } }
WaitHandle IAsyncResult.AsyncWaitHandle { get { return null; } }
Object IAsyncResult.AsyncState { get { return _state; } }
bool IAsyncResult.CompletedSynchronously { get { return false; } }
public AsynchOperation(AsyncCallback callback, HttpContext context, Object state)
{
_callback = callback;
_context = context;
_state = state;
_completed = false;
}
public void StartAsyncWork()
{
_completed = true;
_callback(this);
}
}
In this approach, we actually do not do anything asynchronously. AsynchOperation is just a fake async task. All of my business logic is still executed on the main thread which does not change any behavior of the current code.
I'm running into some performance issues using a generic handler that implements IHttpAsyncHandler. At its simplest, the handler receives a GET request, and 20 seconds later ends the response after writing '< timeout / >' to the response.
When hammering the .ashx with 10000-20000 simultaneous requests, it fails with 503 server unavailable after precisely 5000 requests. When switching to synchronous mode, and ending the request immediately, the problem goes away.
I've tinkered with a number of settings, yet the only thing I've managed to acheive is lower the request threshold at which this error occurs.
Here's a rundown of the settings I've toyed with:
machine.config:
<configuration>
...
<system.web>
...
<processModel enable="true" requestQueueLimit="10000"/>
...
web.config:
<configuration>
...
<system.web>
...
<httpRuntime enable="true" appRequestQueueLimit="10000"/>
...
IIS Manager > ApplicationPools > Advanced Settings
Queue Length : 65535
Although I can't be sure, it seems like these settings work good and fine if the requests are synchronous, but when async, I can't seem to get beyond exactly 5000 requests before the server starts telling me to go away. If I set things lower (can't remember exactly which setting that would be from the above, but I've tried them all), then the 503 count goes up accordingly, but I can never stop it happening beyond 5000 when under serious load.
It seems that there a a number of settings scattered in a myriad of places that might affect this, but the 5000 seems fairly set in stone. I see here that appRequestQueueLimit cannot exceed 5000, but can find no further info about this, and wonder if this is misinformation.
Is there any kind of "flood-control" setting in IIS that might be limiting a single host to no more than 5000 requests? How can I get IIS to handle more that 5000 concurrent asynchronous requests?
Edit2: Are there any counters or other indicators of which limit might be being exceeded, and how would I investigate further?
Edit: Here's the loadgenerator code:
using System;
using System.Net;
using System.Threading;
namespace HammerTime
{
class Program
{
private static int counter = 0;
static void Main(string[] args)
{
var limit = 5000;
ServicePointManager.DefaultConnectionLimit=limit;
for (int i = 0; i < limit;++i )
{
StartWebRequest(i.ToString());
}
Console.ReadLine();
}
private static void StartWebRequest(string channelId)
{
string uri = "http://spender2008/test/Test.ashx?channel="+channelId;
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
request.BeginGetResponse(responseHandler, request);
}
private static void responseHandler(IAsyncResult ar)
{
try
{
HttpWebRequest state = (HttpWebRequest)ar.AsyncState;
HttpWebResponse response = (HttpWebResponse)state.EndGetResponse(ar);
}
catch(Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
Console.WriteLine(Interlocked.Increment(ref counter));
}
}
}
}
OK. Fixed... many thanks to this post for clearing up a few details.
To eliminate the 503 errors required 3 different config changes:
machine.config:
<configuration>
...
<system.web>
...
<processModel enable="true" requestQueueLimit="100000"/>
IIS Manager > ApplicationPools > Advanced Settings
Queue Length : 65535
and finally (the missing piece of the puzzle), the command line:
appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:100000
The web.config setting mentioned in the main post was irrelevant.
10000 concurrent connections, no problems. Thanks for help!