I have a backend IIS server that hosts an ASMX web service. During the past months, this web service has been performing very poorly: requests that usually took around 20ms are now taking around 250ms. After investigating the matter in New Relic, I see that the majority of the time (over 90% of the request processing time - well over 200ms) is being spent on this call:
System.Web.HttpApplication+MaterializeHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
In all of my other web services, this call takes no more than 1ms.
I have no idea what is this call and why it is performing so poorly. I tried website recycles and IIS resets, but that didn't help. I also tried to google it for hours, but to no avail.
I should also mention that sometimes, for a few brief hours, this problem disappears and the above call is being executed in less than 1ms, as expected.
Any kind of help will be greatly appreciated.
Related
After deploying my Web Api to Azure, I noticed that I have a very high response latency. I stopwatched a method that awaits a Http request to a controller that just returns a "Hello" string. The times I measure are also not consistent, but most of the time I get something around 0.9 seconds. The problem is, that my database queries take forever, even the least fancy ones take something around two seconds (and when my UI updates multiple elements, it takes up to 4 seconds until the whole thing is loaded).
I have really no idea where to start diagnosing this issue, so any help (even the most basic) would be highly appreciated!
I found my problem, I used Database.EnsureCreated() on every service request, since it was called in the constructor of the DbContext I inject on Startup. Seems pretty unefficient. Whoopsie!
In case of TL;DR - I basically need guidance regarding what tools are available to debug requests which are issued to IIS and which stall inside a module.
I have a problem with an old ASP 2.0 app at the moment whereby it will periodically become unavailable and recycling the app pool (horrible as that may be) doesn't bring it back up 100% of the time.
So first of all it presents itself as requests entering the app pool and being trapped in state 'BeginRequest' in RewriteModule.
It is not a specific request which is always the first to experience this issue. The issue cannot be easily recreated either.
Eventually requests join this backlog and when it becomes 70+ deep the app pool fails to respond to pings from WAS and it forcibly recycles. Predictably it doesn't stop on-time and the old app pool is forced to stop. When the new app pool comes up it either works just fine or it instantly experiences the same issue as the outgoing one and requests begin to queue.
In issues like this all the official guidance is understandably focussed around looking at why the RewriteModule may choke.
I have validated my redirections and though complex there are no obvious issues with syntax (XML validates).
Likewise in inetmgr loading up the URL Rewrite Module seems to parse the configs fine and show them visually.
Basic stuff like permissions is all fine.
When the app is working normally I also used Failed Request Tracing/Logging to look at the request pipeline for a sample URL which stalled and I can confirm that there is no circular logic or weird errors presenting - the request seems to be handled just fine. This also showed me how high up the rewritemodule is invoked and from this I really don't see how the issue could be app-related as .NET isn't invoked at this point.
Annoyingly when an app pool is experiencing this issue and I can throw in requests which just stall Failed Request Tracing is no good because you actually need a request to get to the end of it's journey and fail otherwise it refuses to log anything out.
I resorted to taking process dumps of affected w3wp.exe's and running them through DebugDiag. Unfortunately the only thing I see is that threads are open accessing the rewritemodule but precious little about what they are stuck on.
As anyone else would do I've tried to track the start of the issue back to any recently installed patches or code changes but nothing matches. Likewise this is happening on 3x servers otherwise I would try reinstalling the rewritemodule. Other sites on the same server which invoke rewritemodule are unaffected.
Has anyone else experienced issues like this - the net seems to have relatively little info in this case. Perhaps you can recommend further debugging tools or approaches for IIS which I can adapt to this scenario? This is sort of a cry for help from someone more used to Apache/Nginx - sorry for the long post.
I got website that running on IIS7 that response very slow during weekend night time, after restarting the application pool the website would response fast for certain of time. As time goes (maybe few hours) the website response become slow again.
I open the resource monitor at task manager, and look for the process for the website, the Average CPU is like 6-8, what makes me curious is the Receive B/s go up to 0.8-1MB while the Sent B/s only have 200B, I don't know if that is relevant.
So I was hoping to get an insight of what is the problem here.
Your question is a bit vague. For starters, are we talking a web app, connected to a database? If so, what type of database? There could be any number of causes for this.
But from the sounds of it, I would first start with any database queries you have running. I had a similar issue to yours with a web app, which turned out to be the result of an inefficient SQL Stored Procedure. The site would slow down and grind almost to a halt, but would spring back to life after the app pool was restarted. Rewriting the main Sproc solved the problem.
We have a web page that calls a stored procedure. The stored procedure takes ~ 5 minutes to run. When called from ASP.NET, it times out at ~ 2 minutes and 40 seconds with an HTTP execution timeout error.
I tried setting an HTTP timeout property in my web.config file as:
<httpRuntime executionTimeout="600">
But it didn't help.
Any ideas appreciated. thanks
You should not create a web application with a page that could require such a long response time from the server. As a general rule, anything that you know will take longer than 10 seconds or so should be done as an asynchronous process. You've probably seen websites that display a "please wait" screen for long running processes, most times these pages work by delegating the long-running job to a background process or message queue, then polling until the job either completes successfully or errors out.
I know this may seem like a tall order if you've not done it before, but it really is the professional way to handle the scenario you're faced with. In some cases, your clients may be working from networks with proxy servers set up to abort the HTTP request regardless of what you've set your timeouts to.
This is a dated link, and I believe the .NET framework has introduced other ways of doing this, but I actually still use the following approach today in certain scenarios.
http://www.devx.com/asp/Article/29617
As the title implies, the first "cold" POST to our MVC3 app in the Azure cloud is very slow. Once it "spins up", the normal requests are blazing fast. The first spin-up after a brief period of rest takes a few seconds. Subsequent requests can be measured in milliseconds.
How can we keep this thing awake?
This is probably due to the appplication pool unloading after a period on inactivity. The next request has to take the overhead of starting it up again.
To confirm this, you need to turn on the performance counters and look at the numbers of app domain loads and unloads.
Either way, this blog post explains how to fix it.