After perusing stack exchange & numerous MSDN articles for months, I'm coming to the community looking for help. I'm the IIS Admin for my organization & I've been seeing an issue wherein when a change is made to a .NET application, the page takes a very long time to load when a request is first submitted to the page. When I say a long time, I'm saying anywhere from five minutes to a half hour at times. I've tried a number of items to fix the issue. But it's my belief that there's a configuration setting buried somewhere that is causing the issue. Once the page loads once, it's load time is normal going forward. It seems to happen regardless of the application, although some take longer than others. It's also not consistent. Sometimes it'll take seven minutes to load after a change. That same application may take seventeen minutes to load the next time a change is made... The changes are usually very minor, a new image box is moved in, or a new link added. Nothing major.
If an app takes twenty seconds to a minute to load the first time, we're not concerned. But the ten, fifteen minute, sometimes half hour load time is what is not acceptable.
The issue happens regardless of whether or not it's a static content application, or if it's reaching out to a data source. Any connections to a data source are configured at the app level. We only utilize that on a handful of apps & I verified the connection information is valid in the web.config for those apps that do reach out to a data source. We're using Windows Authentication on every app.
We run a three tiered environment, all running Windows Server 2012 R2 Standard with 16gb of ram & a multicore CPU setup. We're running IIS 8.5 on .NET 4.0.30319. The app pool is utilizing the integrated pipeline with support for 32-bit apps (which the apps are). These are VMware Hosts. The server is rebooted once a week.
The issue does not occur on our test or dev servers. Only our production server. It doesn't matter what time of day the change is made.
In December of 2016, I ported over all of our .NET applications from an older Windows 2003 system running IIS 6.0 to the new Windows 2012 system. While we worked with the developer to alter any hardcoded hostnames in the apps, we ultimately had to install a CNAME host record to re-direct the old hostname to the new one. The issue seems to have started around this time.
One issue I noticed was that the developer was compiling all apps in debug mode. We changed this setting to false, but it only marginally helped in some cases.
I tried the following as well:
Segregated a given app & set the app pool to always run. Also tried changing the application pool identity to run as a network service, or as a service account user.
Adding the application initialization role & configuring in IIS to always run the app pool & enabled preload at the app level. - I backed this out when I saw it made no change.
I married up all the roles between our test/dev servers where the issue doesn't happen with that of production.
Disabled idle timeout at the app pool level.
Compared settings between test & production regarding compilation settings, time-outs, etc
None of these changes made any difference. I can't find anything differentiating the test/dev boxes where there is no issue & the production problematic server. Please let me know what additional information you may need & I appreciate the help in advance!!
Thanks,
Mike
We found the problem. Our AV software was the culprit. Specifically Trend Micro Deep Security.
When a request was made & the program was going to compile in the temp area (C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files) - I noticed it seemed to be taking a very long time for the files to build out. Pre-compiling saw the same behavior. I asked our LAN admin to add an exception for this particular folder. Once that was done, it's only taking 5-10 seconds on first load, which is acceptable. Thanks again for your input.
You want to precompile your pages so they don't get compiled on first run. IIRC it's just a simple web.config addition.
https://msdn.microsoft.com/en-us/library/bb398860.aspx
Though to be honest, compile times of 5 minutes per pages seems like there's a bigger problem going on. I've never seen a new aspx compile take longer than a few seconds.
Neil N's answer makes sense, but I'd definitely check and see if that setting is the same across all of your servers.
If that isn't the culprit, my next stop would definitely be looking at the 'Application' section of Event Viewer. It could be throwing some meaningful warnings to help you get closer to the root of the issue.
Related
My CPU usage is 100% most of the the time in Windows Server 2008-R2 with my own vps, vmware, quad core, and 4GB Ram. When I open windows Task Manager and go to the resource monitor I see that 100% usage is because of workerprocess.exe. I have 3 websites in my IIS.
How can I figure out which web site causes that usage
How can I limit it to 80% usage of CPU?
Could it be a DDOS attack?
Is there any way to prevent DDOS?
I installed eset-nod32 on my vps, but it doesn't show any attack in the logs. I've tried searching about IIS and preventing DDOS, and just found an extension for banning IP addresses, but how can I find which IP address are generating traffic?
The web site is written in ASP.NET and C#. How can I determine what is happening on that web site and which lines of code are causing the CPU usage?
Also, one of my web sites should access administrator's desktop and read and write some files. So because of that I changed its application pool -> identity (Process Model) to local system, and I don't know if it is related with the CPU usage or not.
Diagnosing
In terms of diagnosing what App Pool is causing trouble, you can:
Select the server
Go to IIS > Worker Processes
This should bring up a menu like this so you can determine which App Pool is running amok.
From there you can simply restart the the app pool and 9 times out of 10 that will fix any immediate issues you're having.
Treating
Unless you run some sort of controversial business, this is probably not a DDOS attack. It's likely that some code is just hanging because it couldn't get through to another server or got stuck in a loop or mis-allocated resources or your app pool just hasn't been recycled in a while.
You can deal with this problem programmatically without having to manually identify, log in, and recycle the app pool. Just configure the CPU property on your App Pool. You can have it kill (and automatically restart) your process anytime you reach a CPU threshold for a certain amount of time.
In your case, if you want it to restart at 80%, you can right click on the app pool and go to Advanced Settings and apply the following configurations:
NOTE: As kraken101 pointed out, different IIS GUIs have treated this differently over time. While the config value is always in 1/1000 of a %, sometimes the GUI takes the whole percent.
You can add this to your config section like this:
<applicationPools>
<add name="DefaultAppPool">
<cpu limit="80000" action="KillW3wp" resetInterval="00:01:00" />
</add>
</applicationPools>
Alternatively, you could script it with Powershell's WebAdministration Module like this:
(*make sure web-scripting-tools is enabled)
Import-Module WebAdministration
$appPoolName = "DefaultAppPool"
$appPool = Get-Item "IIS:\AppPools\$appPoolName"
$appPool.cpu.limit = 80000
$appPool.cpu.action = "KillW3wp"
$appPool.cpu.resetInterval = "00:01:00"
$appPool | Set-Item
Preventing
The steps above will help fix some things once they've broken, but won't really solve any underlying issues you have.
Here are some resources on doing performance monitoring:
Troubleshooting High CPU in an IIS 7.x Application Pool
MSDN - Performance Tuning and Monitoring
IIS Extension - Web Performance Monitor
Troubleshooting Failed Requests Using Tracing
PerfMonitor
Log Parser
Well, this can take long time to figure out. Few points to narrow it down:
Identify what is killing the CPU. I recommend Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
Identify what AppPool is causing this
Fix your code
There are a lot of reasons that you can be seeing w3wp.exe high CPU usage.
I have selected six common causes to cover.
High error rates within your ASP.NET web application
Increase in web traffic causing high CPU
Problems with application dependencies
Garbage collection
Requests getting blocked or hung somewhere in the ASP.NET pipeline
Inefficient .NET code that needs to be optimized
Use PerfMon to collect data and DebugDiag to analyse.
Found this link while searching for similar issue.
http://www.iis.net/learn/troubleshoot/performance-issues/troubleshooting-high-cpu-in-an-iis-7x-application-pool
I was facing the same issues recently and found a solution which worked for me and reduced the memory consumption level upto a great extent.
Solution:
First of all find the application which is causing heavy memory usage.
You can find this in the Details section of the Task Manager.
Next.
Open the IIS manager.
Click on Application Pools. You'll find many application pools which your system is using.
Now from the task manager you've found which application is causing the heavy memory consumption. There would be multiple options for that and you need to select the one which is having '1' in it's Application column of your web application.
When you click on the application pool on the right hand side you'll see an option Advance settings under Edit Application pools. Go to Advanced Settings.
5.Now under General category set the Enable 32-bit Applications to True
Restart the IIS server or you can see the consumption goes down in performance section of your Task Manager.
If this solution works for you please add a comment so that I can know.
If it is not necessary turn off 'Enable 32-bit Applications' from your respective application pool of your website.
This worked for me on my local machine
Use procmon to define your problem.
I recently had this problem myself, and once I determined which AppPool was causing the problem, the only way to resolve the issue was remove that app pool completly and create a new one for the site to use.
Some background info:
We have several websites running on a 64-bit machine with IIS6
These websites all have the same core code, but different skins and content
We have a SQL 2005 database which is fairly heavily used throughout the site
Historically we've used SQL stored procs, but have been gradually transitioning to NHibernate. The majority of our code uses NHibernate now, but not all.
These sites have been running fine on our live web server for a while, although we get a few errors a day regarding SQL connectivity / deadlocking.
Last Thursday we noticed the sites going very slow, then checking task manager revealed one of the websites was hogging over 1.6Gb of memory. Ever since then we've been restarting the app and watching it slowly increase in size over the course of the day.
We apparently have a memory leak (or at least, that's the effect), but I'm losing hair trying to work out how to trace it.
It only appears to be happening on this one website, even though as far as I am aware nothing had changed in the code before it started happenning. It is, however, our busiest website so it could be a traffic issue.
Debug Diagnostics hasn't revealed any issues.
Refreshing certain pages very quickly causes the memory to jump up rapidly, then fall slightly, but all the time the gradual progression is upwards.
I cannot replicate the issue on our test servers or locally. Probably because the traffic has something to do with it.
My suspicion is that the problem lies in database connectivity / locking. However, I'm not sure how that would cause the problem specified.
Any ideas?
Edit
Okay so not exactly sure I've found the problem but we're getting closer. It's definately SQL related. The error log reveals lots of errors since last thursday.
It all happened after we ran some windows updates on our servers. One of the updates failed on the SQL server so not sure if this caused some problems.
The warnings we're getting are:
SQL Server has encountered XX occurence(s) of I/O requests taking longer than 15 seconds to complete on file .. tempdb.mdf
Where XX is anything between 17 and 90! Does that sound like a deadlocking issue?
Followed by the following erors:
Unable to complete login process due to delay in opening server connection
These coincide with our log times for when the websites have been "blipping".
We've increased the page file size on SQL server to the recommended size, as it was set to a max of 4Gb, but recommended was 12Gb. I think we may need to roll back the windows updates we did on Thursday if that doesn't fix it.
Unfortunately I can't get into Activity monitor as it tells me Timeout expired!
Edit
Okay after a reboot I'm into Activity monitor. How many sleeping processes would you say would be normal? We have roughly 127 sleeping. That's serving over 10 websites.
If there is a deadlock or timeout issue, will NHibernate not clean up its connections properly?
Okay so in the end it seems it's quite complex. Sql deadlocks and data problems, heightened it seems by anti-virus software that was locking up or choking on a file.
Turning off the anti-virus reduced the problems, but we still need to resolve the underlying data issues.
In continuation to last question - My site goes slow and stops access certain services externally if we check the Process monitor we see that it is normally due to the ‘w3p.exe’ process – which is the background process for running the website – it regularly reaches 99/100% - killing the process/restarting the WebPublishing service reolves tis – my webhost says this can only be due to bad coding ....can someone comment on this ??…
Wanted to know any monitoring software which traces IIS & freely available ...
If you are using Asp.Net then you can use the built in Asp.Net tracing to find out things like the size of your viewstate and the where the time is spent while rendering a page. There are various ways of enabling this depending on what your needs are: see http://authors.aspalliance.com/aspxtreme/webapps/tracefunctionality.aspx
99% CPU is not going to occur if you have an inefficient page or two. 99% CPU utilization happens when you have a bug.
If it does not happen on your local server, but only in the hosted environment, then you will have to resort to the old school detective approaches. Tracing, removing portions of code, and so on - til you find the source of the problem.
I have recently purchased some new windows server 2008 boxes all running IIS7.
I have moved some of my websites across to them, but they seem to be running VERY slow when the app pool first kicks in.
All the sites are precompiled (using webdeploy) and in IIS6 they were lightening fast, I thought it might have been just because the web apps are fairly large but I get the same issue on smaller ones too.
The app pools are set to Classic mode, as we couldnt get the urlMapping to work in integrated mode.
I have seen a few posts about the slowness on the net and one post on here about it and people throwing stupid answers around such as its because your code isnt compiled or its to do with JIT, and both answers are pretty much a word that rhymes with JIT but has an extra letter. I say this because I have run much larger web apps in IIS6 and not experienced this lag.
Thanks for anything that you bring to the table on this.
edit:
just done some timings and it can take up to 28seconds for the first request to return a page...
This was slow due to using Classic Mode, I have moved it to integrated pipes which is faster at around 9 seconds load time, which is still unacceptable, but I believe this is an application issue now not IIS.
I've asked this on server fault already but to no avail. So I'm hoping another ASP.Net Developer would have come across my issue.
I've got a Windows VPS running II6 and I'm migrating my ASP.Net applications across to the server. When I navigate to one of those applications it takes approximately 2 minutes to do anything. I created a hello world application and uploaded that, and that to takes 2 minutes to load the first time. If I restart the app pool I get exactly the same issue.
Any application that shares an app pool with another application will work once one application has loaded successfully in that app pool.
I.e. bob.example.com and jane.example.com are in an app pool called ABC1. If I access bob it will take 2-3 minutes to load. After bob has loaded, Jane will work instantly and bob will work instantly. After 5-10 minutes of being idle, or if I restart the app pool, exactly the same process happens again. During the time when bob and jane and working fine, if dave.example.com in app pool XYZ9 is loaded, I must wait 2-3 minutes for that to load too.
Has anyone come across this issue? I am in absolute panic mode at the moment as I need to get these migrated by the end of the weekend.
Oh, and those sites work instantly on the old server. The only difference is that the old server is 32-bit and the new server is 64-bit.
You say it takes two minutes to start up. What is the system doing during that time? In particular, what is w3wp.exe doing?
I suggest you run SysInternals' Process Monitor to learn a bit about what's going on. It will take a while to come up with a useful filter, but it should be worthwhile.
In particular, the latest versions of this tool can give you information about networking calls being made by the processes you're watching, which will be good if it turns out you've got issues with certificates, proxy servers, or both.
Instead of requesting a .aspx file, what happens if you request a plain .htm file? The difference is that HTML files don't go through ASP.NET, so if you still get the slowness issue, you'll know that it is more of an IIS thing than an ASP.NET thing.
Though not an answer, it's a step towards isolating the issue.
On shared virtual hosts (VPS), I've found that the disk access can be extraordinarily slow, and vary at times of day. When the application pool spins up, the process has to load from disk and it (presumably) needs to load the .Net framework.
(I asked a similar question here.)
Really, my virtual host is shockingly slow at times so I believe that 2 minutes can be attributable to disk I/O speed. And if your virtual server has a low amount of RAM, it will be worse.
I use a server monitoring service to hit a URL every 30 minutes. This helps keep the app loaded. ASP.net 4.0 has some features which promise to help.
I disabled recycling and that fixed everything...ish. Once it's loaded once it's loaded forever. Still doesn't fix the underlying problem, but 1and1 refuse to help regarding issues with IIS.
This could just be the initial appdomain compile startup cost but magnified because you're using a VPS.
What kind of spec have you got on that?
I don't have any clue on this question.
However, I read/heard somewhere that - in such a scenario it could be possible that your request goes out on the internet to check for the domain, when it should check first in the local network (related to DNS maybe).
This is totally based on what I can recollect of a similar problem people discussing on an article/podcast.
Could be that the server is probably having problems validating the signed assemblies certificates revocation chain lists. You can disable CRL checks and see if it performs better.
Update
To disable the Certificate Revocation List (CRL) check:
Open Internet Explorer
Go to Tools —> Internet Options…
Go to the Advanced tab
Locate the Security section and uncheck the Check for publisher’s certificate revocation option.
This disables globally, for the entire server, the CRL check, and has the side effect of disabling the authenticode CRL checks (ie. the ASP app pool checks). I'm not aware of any more fine grained method. Also, I don't know if this has to be done separately for the x86 version of IE (to apply to 32 bit apps) and to the x64 version of IE (for the 64 bit apps).
Does your application share the same application pool with other web applications? In other words, are you using a third-party VPS and have no control over app pool usage? If so, then the application initialization code for the other apps could be what's causing the slow startup time.
Under what identity is the application pool running? Domain, or local?
What forms of authentication are you using? If anonymous, what account is it using (domain or local)>
If you check task manager right after iisreset and hitting the website, does it show w3svc running, or are we waiting for it to start at all?
What about a static file, e.g., .htm?
That's all I have - hope this helps.