My CPU usage is 100% most of the the time in Windows Server 2008-R2 with my own vps, vmware, quad core, and 4GB Ram. When I open windows Task Manager and go to the resource monitor I see that 100% usage is because of workerprocess.exe. I have 3 websites in my IIS.
How can I figure out which web site causes that usage
How can I limit it to 80% usage of CPU?
Could it be a DDOS attack?
Is there any way to prevent DDOS?
I installed eset-nod32 on my vps, but it doesn't show any attack in the logs. I've tried searching about IIS and preventing DDOS, and just found an extension for banning IP addresses, but how can I find which IP address are generating traffic?
The web site is written in ASP.NET and C#. How can I determine what is happening on that web site and which lines of code are causing the CPU usage?
Also, one of my web sites should access administrator's desktop and read and write some files. So because of that I changed its application pool -> identity (Process Model) to local system, and I don't know if it is related with the CPU usage or not.
Diagnosing
In terms of diagnosing what App Pool is causing trouble, you can:
Select the server
Go to IIS > Worker Processes
This should bring up a menu like this so you can determine which App Pool is running amok.
From there you can simply restart the the app pool and 9 times out of 10 that will fix any immediate issues you're having.
Treating
Unless you run some sort of controversial business, this is probably not a DDOS attack. It's likely that some code is just hanging because it couldn't get through to another server or got stuck in a loop or mis-allocated resources or your app pool just hasn't been recycled in a while.
You can deal with this problem programmatically without having to manually identify, log in, and recycle the app pool. Just configure the CPU property on your App Pool. You can have it kill (and automatically restart) your process anytime you reach a CPU threshold for a certain amount of time.
In your case, if you want it to restart at 80%, you can right click on the app pool and go to Advanced Settings and apply the following configurations:
NOTE: As kraken101 pointed out, different IIS GUIs have treated this differently over time. While the config value is always in 1/1000 of a %, sometimes the GUI takes the whole percent.
You can add this to your config section like this:
<applicationPools>
<add name="DefaultAppPool">
<cpu limit="80000" action="KillW3wp" resetInterval="00:01:00" />
</add>
</applicationPools>
Alternatively, you could script it with Powershell's WebAdministration Module like this:
(*make sure web-scripting-tools is enabled)
Import-Module WebAdministration
$appPoolName = "DefaultAppPool"
$appPool = Get-Item "IIS:\AppPools\$appPoolName"
$appPool.cpu.limit = 80000
$appPool.cpu.action = "KillW3wp"
$appPool.cpu.resetInterval = "00:01:00"
$appPool | Set-Item
Preventing
The steps above will help fix some things once they've broken, but won't really solve any underlying issues you have.
Here are some resources on doing performance monitoring:
Troubleshooting High CPU in an IIS 7.x Application Pool
MSDN - Performance Tuning and Monitoring
IIS Extension - Web Performance Monitor
Troubleshooting Failed Requests Using Tracing
PerfMonitor
Log Parser
Well, this can take long time to figure out. Few points to narrow it down:
Identify what is killing the CPU. I recommend Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
Identify what AppPool is causing this
Fix your code
There are a lot of reasons that you can be seeing w3wp.exe high CPU usage.
I have selected six common causes to cover.
High error rates within your ASP.NET web application
Increase in web traffic causing high CPU
Problems with application dependencies
Garbage collection
Requests getting blocked or hung somewhere in the ASP.NET pipeline
Inefficient .NET code that needs to be optimized
Use PerfMon to collect data and DebugDiag to analyse.
Found this link while searching for similar issue.
http://www.iis.net/learn/troubleshoot/performance-issues/troubleshooting-high-cpu-in-an-iis-7x-application-pool
I was facing the same issues recently and found a solution which worked for me and reduced the memory consumption level upto a great extent.
Solution:
First of all find the application which is causing heavy memory usage.
You can find this in the Details section of the Task Manager.
Next.
Open the IIS manager.
Click on Application Pools. You'll find many application pools which your system is using.
Now from the task manager you've found which application is causing the heavy memory consumption. There would be multiple options for that and you need to select the one which is having '1' in it's Application column of your web application.
When you click on the application pool on the right hand side you'll see an option Advance settings under Edit Application pools. Go to Advanced Settings.
5.Now under General category set the Enable 32-bit Applications to True
Restart the IIS server or you can see the consumption goes down in performance section of your Task Manager.
If this solution works for you please add a comment so that I can know.
If it is not necessary turn off 'Enable 32-bit Applications' from your respective application pool of your website.
This worked for me on my local machine
Use procmon to define your problem.
I recently had this problem myself, and once I determined which AppPool was causing the problem, the only way to resolve the issue was remove that app pool completly and create a new one for the site to use.
Related
After perusing stack exchange & numerous MSDN articles for months, I'm coming to the community looking for help. I'm the IIS Admin for my organization & I've been seeing an issue wherein when a change is made to a .NET application, the page takes a very long time to load when a request is first submitted to the page. When I say a long time, I'm saying anywhere from five minutes to a half hour at times. I've tried a number of items to fix the issue. But it's my belief that there's a configuration setting buried somewhere that is causing the issue. Once the page loads once, it's load time is normal going forward. It seems to happen regardless of the application, although some take longer than others. It's also not consistent. Sometimes it'll take seven minutes to load after a change. That same application may take seventeen minutes to load the next time a change is made... The changes are usually very minor, a new image box is moved in, or a new link added. Nothing major.
If an app takes twenty seconds to a minute to load the first time, we're not concerned. But the ten, fifteen minute, sometimes half hour load time is what is not acceptable.
The issue happens regardless of whether or not it's a static content application, or if it's reaching out to a data source. Any connections to a data source are configured at the app level. We only utilize that on a handful of apps & I verified the connection information is valid in the web.config for those apps that do reach out to a data source. We're using Windows Authentication on every app.
We run a three tiered environment, all running Windows Server 2012 R2 Standard with 16gb of ram & a multicore CPU setup. We're running IIS 8.5 on .NET 4.0.30319. The app pool is utilizing the integrated pipeline with support for 32-bit apps (which the apps are). These are VMware Hosts. The server is rebooted once a week.
The issue does not occur on our test or dev servers. Only our production server. It doesn't matter what time of day the change is made.
In December of 2016, I ported over all of our .NET applications from an older Windows 2003 system running IIS 6.0 to the new Windows 2012 system. While we worked with the developer to alter any hardcoded hostnames in the apps, we ultimately had to install a CNAME host record to re-direct the old hostname to the new one. The issue seems to have started around this time.
One issue I noticed was that the developer was compiling all apps in debug mode. We changed this setting to false, but it only marginally helped in some cases.
I tried the following as well:
Segregated a given app & set the app pool to always run. Also tried changing the application pool identity to run as a network service, or as a service account user.
Adding the application initialization role & configuring in IIS to always run the app pool & enabled preload at the app level. - I backed this out when I saw it made no change.
I married up all the roles between our test/dev servers where the issue doesn't happen with that of production.
Disabled idle timeout at the app pool level.
Compared settings between test & production regarding compilation settings, time-outs, etc
None of these changes made any difference. I can't find anything differentiating the test/dev boxes where there is no issue & the production problematic server. Please let me know what additional information you may need & I appreciate the help in advance!!
Thanks,
Mike
We found the problem. Our AV software was the culprit. Specifically Trend Micro Deep Security.
When a request was made & the program was going to compile in the temp area (C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files) - I noticed it seemed to be taking a very long time for the files to build out. Pre-compiling saw the same behavior. I asked our LAN admin to add an exception for this particular folder. Once that was done, it's only taking 5-10 seconds on first load, which is acceptable. Thanks again for your input.
You want to precompile your pages so they don't get compiled on first run. IIRC it's just a simple web.config addition.
https://msdn.microsoft.com/en-us/library/bb398860.aspx
Though to be honest, compile times of 5 minutes per pages seems like there's a bigger problem going on. I've never seen a new aspx compile take longer than a few seconds.
Neil N's answer makes sense, but I'd definitely check and see if that setting is the same across all of your servers.
If that isn't the culprit, my next stop would definitely be looking at the 'Application' section of Event Viewer. It could be throwing some meaningful warnings to help you get closer to the root of the issue.
I have a website application running in it's own application pool on IIS 7.0. The application is an ASP.NET MVC 3 website.
I have noticed the memory usage for this applications corresponding w3wp IIS worker service is quite high ( 800 MB, with some fluctuation ).
I am trying to diagnose the problem and have tried the following:
I have disabled output page caching for the website at IIS level and then recycled the application pool. This causes the w3wp process to restart. The memory usage for this process then slowly creeps up to around 800 MB, it takes around 30 seconds to do so. There are no page requests being handled at this time. When I restart the website from IIS the memory size of the process does not alter.
I have tried running a debug copy of the application from VS 2010, there are no problems with memory usage.
Some ideas I have/questions are:
Is this problem related to the websites code? - Given that the memory rockets before any page requests have been sent/handled, I would assume this is NOT a code problem?
The application built in MVC has no handling of caching written into it.
The website uses real-time displaying of data, it uses ajax requests periodically, and is generally left 'open' for long periods of time.
Why does the memory usage rocket up after the application is recycled and no user requests are being sent? Is this because it is loading old cache information into it's memory from disk?
The application does NOT crash, I'm just concerned about the memory usage, it is not that big of a website...
Any ideas/help with getting to the bottom of this problem would be appreciated.
The best thing to do if you can afford to use a debugger is install the Windows Debugging Tools and use something like WinDbg and SOS.dll to figure out exactly what is it in memory.
once you've installed the tools then you can:
Launch Windbg.exe running elevated (as Administrator)
Use "File->Attach To Process" and choose w3wp.exe for the app you are trying to figure out. If you have many you can use Task Manager and add the command-line column to see the PID or use IIS Manager->Worker Processes to figure it out, and then choose that process in WinDBG.
run:
.loadby sos clr
!dumpheap -stat
At that point you should be able to see all types sorted by the most memory consumption so you can start with the ones at the bottom. (I would recommend exclude Strings, and Object since those are usually a side-effect and not the cause).
Use "!dumpheap -type type-here" to find the instances and use !gcroot for those to figure out why they are in memory, maybe due to a static field, or an event handler leaked, WCF channels not disposed, or things like that are common sources.
I just looked my server and my pools use 900-1000 MB Virtual size Memory, and 380 MB Working set. My sites run smooth with out problem for some years now, and I have checked the sites from all sides. My pool never recycles and the server runs until the next update continuously with 40% stable free physical memory.
If your memory is not continuously growing, then this memory is the code plus the data that you set as static, const, the string, and the possible cache, inside your application.
You can use process explorer to see the working and the virtual size memory.
You can also think to run a profile against your code to see if you have any "memory leak" or other issue. Find one from google: https://www.google.com/search?hl=en&q=asp.net+memory+profiler.
It probably doesn't apply here but thought I would throw it in for good measure. Recently I had a problem where my memory would go right up and max out when it really could of cleaned up 80% of it. Problem: It thought it about 2 more gig than it actually did so the GC was quite lazy. (It was due to a VM ware bug -windows was reporting 8 Gig but physically there was only 6.4). See blog.http://www.worthalook.net/2014/01/give-back-memory/
Something that might help: if you "rewrite" (open/save) the web.config , then your application will reset, you should monitor the memory usage from that point. If it keeps growing during usage, this could mean memory leak or insane caching. You might be able to identify which actions on your site lead to memory increase. During a long time the memory usage of an application should be stable.
An ASP.NET web app running on IIS6 periodically shoots the CPU up to 100%. It's the W3WP that's responsible for nearly all CPU usage during these episodes. The CPU stays pinned at 100% anywhere from a few minutes to over an hour.
This is on a staging server and the site is only getting very light traffic from testers at this point.
We've running ANTS profiler on the server, but it's been unenlightening.
Where can we start finding out what's causing these episodes and what code is keeping the CPU busy during all that time?
Standard Windows performance counters (look for other correlated activity, such as many GET requests, excessive network or disk I/O, etc); you can read them from code as well as from perfmon (to trigger data collection if CPU use exceeds a threshold, for example)
Custom performance counters (particularly to time for off-box requests and other calls where execution time is uncertain)
Load testing, using tools such as Visual Studio Team Test or WCAT
If you can test on or upgrade to IIS 7, you can configure Failed Request Tracing to generate a trace if requests take more a certain amount of time
Use logparser to see which requests arrived at the time of the CPU spike
Code reviews / walk-throughs (in particular, look for loops that may not terminate properly, such as if an error happens, as well as locks and potential threading issues, such as the use of statics)
CPU and memory profiling (can be difficult on a production system)
Process Explorer
Windows Resource Monitor
Detailed error logging
Custom trace logging, including execution time details (perhaps conditional, based on the CPU-use perf counter)
Are the errors happening when the AppPool recycles? If so, it could be a clue.
It's not much of an answer, but you might need to go old school and capture an image snapshot of the IIS process and debug it. You might also want to check out Tess Ferrandez's blog - she is a kick a** microsoft escalation engineer and her blog focuses on debugging windows ASP.NET, but the blog is relevant to windows debugging in general. If you select the ASP.NET tag (which is what I've linked to) then you'll see several items that are similar.
If your CPU is spiking to 100% and staying there, it's quite likely that you either have a deadlock scenario or an infinite loop. A profiler seems like a good choice for finding an infinite loop. Deadlocks are much more difficult to track down, however.
Process Explorer is an excellent tool for troubleshooting. You can try it for finding the problem of high CPU usage. It gives you an insight into the way your application works.
You can also try Procdump to dump the process and analyze what really happened on the CPU.
Also, look at your perfmon counters. They can tell you where a lot of that cpu time is being spent. Here's a link to the most common counters to use:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/852720c8-7589-49c3-a9d1-73fdfc9126f0.mspx?mfr=true
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/be425785-c1a4-432c-837c-a03345f3885e.mspx?mfr=true
We had this on a recursive query that was dumping tons of data to the output - have you double checked everything does exit and no infinite loops exist?
Might try to narrow it down with a single page - we found ANTS to not be much help in that same case either - what we ended up doing was running the site hit a page watch the CPU - hit the next page watch CPU - very methodical and time consuming but if you cant find it with some code tracing you might be out of luck -
We were able to use IIS log files to track it to a set of pages that were suspect -
Hope that helps !
This is a guess at best, but perhaps your development team is building and deploying the application in debug mode, in stead of release mode. This will cause the occurrence of .pdb files. The implication of this is that your application will take up additional resources to collect system state and debugging information during the execution of your system, causing more processor utilization.
So, it would be simple enough to ensure that they are building and deploying in release mode.
This is a very old post, I know, but this is also a common problem. All of the suggested methods are very nice but they will always point to a process, and there are many chances that we already know that our site is making problems, but we just want to know what specific page is spending too much time in processing.
The most precise and simple tool in my opinion is IIS itself.
Just click on your server in the left pane of IIS.
Click on 'Worker Processes' in the main pane. you already see what application pool is taking too much CPU.
Double click on this line (eventually refresh by clicking 'Show All') to see what pages consume too much CPU time ('Time elapsed'
column) in this pool
If you identify a page that takes time to load, use SharePoint's Developer Dashboard to see which component takes time.
I've asked this on server fault already but to no avail. So I'm hoping another ASP.Net Developer would have come across my issue.
I've got a Windows VPS running II6 and I'm migrating my ASP.Net applications across to the server. When I navigate to one of those applications it takes approximately 2 minutes to do anything. I created a hello world application and uploaded that, and that to takes 2 minutes to load the first time. If I restart the app pool I get exactly the same issue.
Any application that shares an app pool with another application will work once one application has loaded successfully in that app pool.
I.e. bob.example.com and jane.example.com are in an app pool called ABC1. If I access bob it will take 2-3 minutes to load. After bob has loaded, Jane will work instantly and bob will work instantly. After 5-10 minutes of being idle, or if I restart the app pool, exactly the same process happens again. During the time when bob and jane and working fine, if dave.example.com in app pool XYZ9 is loaded, I must wait 2-3 minutes for that to load too.
Has anyone come across this issue? I am in absolute panic mode at the moment as I need to get these migrated by the end of the weekend.
Oh, and those sites work instantly on the old server. The only difference is that the old server is 32-bit and the new server is 64-bit.
You say it takes two minutes to start up. What is the system doing during that time? In particular, what is w3wp.exe doing?
I suggest you run SysInternals' Process Monitor to learn a bit about what's going on. It will take a while to come up with a useful filter, but it should be worthwhile.
In particular, the latest versions of this tool can give you information about networking calls being made by the processes you're watching, which will be good if it turns out you've got issues with certificates, proxy servers, or both.
Instead of requesting a .aspx file, what happens if you request a plain .htm file? The difference is that HTML files don't go through ASP.NET, so if you still get the slowness issue, you'll know that it is more of an IIS thing than an ASP.NET thing.
Though not an answer, it's a step towards isolating the issue.
On shared virtual hosts (VPS), I've found that the disk access can be extraordinarily slow, and vary at times of day. When the application pool spins up, the process has to load from disk and it (presumably) needs to load the .Net framework.
(I asked a similar question here.)
Really, my virtual host is shockingly slow at times so I believe that 2 minutes can be attributable to disk I/O speed. And if your virtual server has a low amount of RAM, it will be worse.
I use a server monitoring service to hit a URL every 30 minutes. This helps keep the app loaded. ASP.net 4.0 has some features which promise to help.
I disabled recycling and that fixed everything...ish. Once it's loaded once it's loaded forever. Still doesn't fix the underlying problem, but 1and1 refuse to help regarding issues with IIS.
This could just be the initial appdomain compile startup cost but magnified because you're using a VPS.
What kind of spec have you got on that?
I don't have any clue on this question.
However, I read/heard somewhere that - in such a scenario it could be possible that your request goes out on the internet to check for the domain, when it should check first in the local network (related to DNS maybe).
This is totally based on what I can recollect of a similar problem people discussing on an article/podcast.
Could be that the server is probably having problems validating the signed assemblies certificates revocation chain lists. You can disable CRL checks and see if it performs better.
Update
To disable the Certificate Revocation List (CRL) check:
Open Internet Explorer
Go to Tools —> Internet Options…
Go to the Advanced tab
Locate the Security section and uncheck the Check for publisher’s certificate revocation option.
This disables globally, for the entire server, the CRL check, and has the side effect of disabling the authenticode CRL checks (ie. the ASP app pool checks). I'm not aware of any more fine grained method. Also, I don't know if this has to be done separately for the x86 version of IE (to apply to 32 bit apps) and to the x64 version of IE (for the 64 bit apps).
Does your application share the same application pool with other web applications? In other words, are you using a third-party VPS and have no control over app pool usage? If so, then the application initialization code for the other apps could be what's causing the slow startup time.
Under what identity is the application pool running? Domain, or local?
What forms of authentication are you using? If anonymous, what account is it using (domain or local)>
If you check task manager right after iisreset and hitting the website, does it show w3svc running, or are we waiting for it to start at all?
What about a static file, e.g., .htm?
That's all I have - hope this helps.
We're running a custom application on our intranet and we have found a problem after upgrading it recently where IIS hangs with 100% CPU usage, requiring a reset.
Rather than subject users to the hangs, we've rolled back to the previous release while we determine a solution. The first step is to reproduce the problem -- but we can't.
Here's some background:
Prod has a single virtualized (vmware) web server with two CPUs and 2 GB of RAM. The database server has 4GB, and 2 CPUs as well. It's also on VMWare, but separate physical hardware.
During normal usage the application runs fine. The w3wp.exe process normally uses betwen 5-20% CPU and around 200MB of RAM. CPU and RAM fluctuate slightly under normal use, but nothing unusual.
However, when we start running into problems, the RAM climbs dramatically and the CPU pegs at 98% (or as much as it can get). The site becomes unresponsive, necessitating a IIS restart. Resetting the app pool does nothing in this situation, a full IIS restart is required.
It does not happen during the night (no usage). It happens more when the site is under load, but it has also happened under non-peak periods.
First step to solving this problem is reproducing it. To simulate the load, we starting using JMeter to simulate usage. Our load script is based on actual usage around the time of the crash. Using JMeter, we can ramp the usage up quite high (2-3 times the load during the crash) but the site behaves fine. CPU is up high, and the site does become sluggish, but memory usage is reasonable and nothing is hanging.
Does anyone have any tips on how to reproduce a problem like this in a non-production environment? We'd really like to reproduce the error, determine a solution, then test again to make sure we've resolved it. During the process we've found a number of small things that we've improved that might solve the problem, but I'd really feel a lot more confident if we could reproduce the problem and test the improved version.
Any tools, techniques or theories much appreciated!
You can find some information about troubleshooting this kind of problem at this blog entry. Her blog is generally a good debugging resource.
I have an article about debugging ASP.NET in production which may provide some pointers.
Is your test env the same really as live?
i.e
2 separate vm instances on 2 physical servers - with the network connection and account types?
Is there any other instances on the Database?
Is there any other web applications in IIS?
Is the .Net Config right?
Is the App Pool Config right for service accounts ?
Try look at this - MS Article on II6 Optmising for Performance
Lots of tricks.