One of our web servers is suffering from random w3wp.exe crashing and after a couple of weeks of debugging i simply cannot figure out why. The only thing that has helped so far is reducing the max worker processes from 15 to 5 however this isn't ideal as we are using a multi-cpu machine in the hopes of reducing the total number of servers needed. We serve a large volume of small requests so parallel processing is a requirement.
As far as I am aware all possible sources of parallel processing collision have been addressed using thread locking.
Win 2008 64Bit SP2
IIS7
Dual 3.1Ghz Xeon
4Gb Ram
First error:
Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 70D9CECA (70D40000) with exit code 80131506.
Followed straight away by:
Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module clr.dll, version 4.0.30319.1, time stamp 0x4ba1d9ef, exception code 0xc0000005, fault offset 0x0005ceca, process id 0x%9, application start time 0x%10.
Many Thanks
Edit
Problem eventually solved. It turned out SQL server was unmounting the database straight after every query, so every new query had to wait for it to re-mount. Anyway, telling SQL Server not to do that seems to have solved it, no idea how but it's working so I'm happy
Problem eventually solved. It turned out SQL server was unmounting the database straight after every query, so every new query had to wait for it to re-mount. Anyway, telling SQL Server not to do that seems to have solved it, no idea how but it's working so I'm happy
Exception code 0xc0000005 generally points to memory access violation. Look for any unsafe component you may be using.
You are in for a doozy of a ride. These exceptions are extremely tricky to track down and correct.
The first step is to get the IIS Debug Diagnostic Tool (v1.1). Once you have installed this, you'll need to set up some tracking projects and then attach the debugger to your running processes. Keep in mind, this tool collects a LOT of data (it can be in excess of 1GB of stuff), so combing through it may be a hassle, but it has a good potential of telling you what modules are causing the crash and what modules are interfering.
The reason w3wp.exe is crashing, though, is that an unhandle-able exception is occurring during phases of the transaction that your code/health-monitoring/etc are already completed.
In my own personal case, I found that decoupling the session from the process solved the problem. I never discovered the full reason, but the best guess we had was that the memory requirements for paging were too great for the w3wp.exe to handle all at the same time. Once we decoupled into an external session state server, the problem went away.
It may be time to re-think your web-garden. Scott Forsyth has an interesting 11 minute vLog on why webgardens are counterproductive: http://dotnetslackers.com/articles/iis/Why-You-Shouldnt-Use-Web-Gardens-in-IIS-Week-24.aspx
Links to articles he mentions in his VLog:
Tuning recommendations for IIS6 and IIS7 -- read the whole article: http://support.microsoft.com/kb/821268 Further information http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx
His bottom line is if you have performance problems that are resolved by web gardens—use the web gardens as a great crutch until the underlying performance issue (usually resource contention) is resolved
Related
I am trying to run this website I have been working on at work locally at home, but I can't get it to work. I have tried recycling the dedicated app pool, restarting the site, killing the w3wp.exe processes, recompiling the site, deleting temporary ASP.net files from "C:\Windows...".
Making a new request from the browser brings up the worker process, which just sits there after a few seconds of low cpu usage. Then its memory usage continues to fluctuate by a few kilobytes though.
When I introduce a compilation error, the YSOD appears swiftly. Eliminating the compilation error takes me back to square one - the requests keep waiting and ultimately times out.
I am clueless as to what's going on. This is not the first time I've had this problem and before calling quits on microsoft stack, courtesy such frustrations, I am shouting out here - yep, am really stressed about this! Any Ideas guys? Or pointers on how to determine what's going on/debug? Framework version is 4.0, integrated mode.
Thanks
I manage an ASP.NET MVC 3 website with multiple online transactions. In the website, customers can place orders, pay bills while vendors can bill customers. All this can happen simultaneously so I have semaphores to ensure thread safety.
What I have noticed is that about once a week, the website stalls for ten minutes. My first tought was for deadlocks in the semaphores, but after putting in place a semaphore log and analysing the results, there seems to be no deadlocks. Also, the website comes back by itself after ten minutes.
While investigating, I noticed that the entire website becomes irresponsive and not just the parts using the semaphores. They all use the database tough. That is why my primary suspect is the database.
What is stranger is that every time, the website freeze for ten minutes almost to the second. Could SQL Server have a scheduled maintenance or anything that could explain this delay? If not, do you have any idea what could cause this?
The answer to your question seems to be "yes". Something is happening in the environment that is locking things up.
Have you run sp_who2 to see what is running when it stalls?
If that is inconvenient, then set up a job to dump sp_who2 output into a table every five minutes. When it stalls, you can see what is running and work from there.
I have faced what may be a similar problem, where the master database seems to be getting locked up. As a consequence, renaming databases does not work. Fortunately, this is not in a live transaction environemnt, so waiting five minutes and trying again does the trick.
ASP.NET Hangs can happen for a variety of reasons. Typically you get Connection or Command timeouts when you have problems with SQL not hangs.
You're much better off
Grabbing the Debugging Tools for Windows
Use AdPlus to grab a memory dump (adplus -hang -pn processname.exe) or DebugDiag and setup a dump rule
Use WinDbg (or VS 2010 for 4.0 framework) (after you set up a symbol cache) and start examining what's happening using
!threads or !dumpheap -stat to inspect the threads and the heap objects.
Please note debugging production issues is very hard and WinDbg is not a friendly tool but guessing and looking at logs is even less so.
I have a website application running in it's own application pool on IIS 7.0. The application is an ASP.NET MVC 3 website.
I have noticed the memory usage for this applications corresponding w3wp IIS worker service is quite high ( 800 MB, with some fluctuation ).
I am trying to diagnose the problem and have tried the following:
I have disabled output page caching for the website at IIS level and then recycled the application pool. This causes the w3wp process to restart. The memory usage for this process then slowly creeps up to around 800 MB, it takes around 30 seconds to do so. There are no page requests being handled at this time. When I restart the website from IIS the memory size of the process does not alter.
I have tried running a debug copy of the application from VS 2010, there are no problems with memory usage.
Some ideas I have/questions are:
Is this problem related to the websites code? - Given that the memory rockets before any page requests have been sent/handled, I would assume this is NOT a code problem?
The application built in MVC has no handling of caching written into it.
The website uses real-time displaying of data, it uses ajax requests periodically, and is generally left 'open' for long periods of time.
Why does the memory usage rocket up after the application is recycled and no user requests are being sent? Is this because it is loading old cache information into it's memory from disk?
The application does NOT crash, I'm just concerned about the memory usage, it is not that big of a website...
Any ideas/help with getting to the bottom of this problem would be appreciated.
The best thing to do if you can afford to use a debugger is install the Windows Debugging Tools and use something like WinDbg and SOS.dll to figure out exactly what is it in memory.
once you've installed the tools then you can:
Launch Windbg.exe running elevated (as Administrator)
Use "File->Attach To Process" and choose w3wp.exe for the app you are trying to figure out. If you have many you can use Task Manager and add the command-line column to see the PID or use IIS Manager->Worker Processes to figure it out, and then choose that process in WinDBG.
run:
.loadby sos clr
!dumpheap -stat
At that point you should be able to see all types sorted by the most memory consumption so you can start with the ones at the bottom. (I would recommend exclude Strings, and Object since those are usually a side-effect and not the cause).
Use "!dumpheap -type type-here" to find the instances and use !gcroot for those to figure out why they are in memory, maybe due to a static field, or an event handler leaked, WCF channels not disposed, or things like that are common sources.
I just looked my server and my pools use 900-1000 MB Virtual size Memory, and 380 MB Working set. My sites run smooth with out problem for some years now, and I have checked the sites from all sides. My pool never recycles and the server runs until the next update continuously with 40% stable free physical memory.
If your memory is not continuously growing, then this memory is the code plus the data that you set as static, const, the string, and the possible cache, inside your application.
You can use process explorer to see the working and the virtual size memory.
You can also think to run a profile against your code to see if you have any "memory leak" or other issue. Find one from google: https://www.google.com/search?hl=en&q=asp.net+memory+profiler.
It probably doesn't apply here but thought I would throw it in for good measure. Recently I had a problem where my memory would go right up and max out when it really could of cleaned up 80% of it. Problem: It thought it about 2 more gig than it actually did so the GC was quite lazy. (It was due to a VM ware bug -windows was reporting 8 Gig but physically there was only 6.4). See blog.http://www.worthalook.net/2014/01/give-back-memory/
Something that might help: if you "rewrite" (open/save) the web.config , then your application will reset, you should monitor the memory usage from that point. If it keeps growing during usage, this could mean memory leak or insane caching. You might be able to identify which actions on your site lead to memory increase. During a long time the memory usage of an application should be stable.
Some background info:
We have several websites running on a 64-bit machine with IIS6
These websites all have the same core code, but different skins and content
We have a SQL 2005 database which is fairly heavily used throughout the site
Historically we've used SQL stored procs, but have been gradually transitioning to NHibernate. The majority of our code uses NHibernate now, but not all.
These sites have been running fine on our live web server for a while, although we get a few errors a day regarding SQL connectivity / deadlocking.
Last Thursday we noticed the sites going very slow, then checking task manager revealed one of the websites was hogging over 1.6Gb of memory. Ever since then we've been restarting the app and watching it slowly increase in size over the course of the day.
We apparently have a memory leak (or at least, that's the effect), but I'm losing hair trying to work out how to trace it.
It only appears to be happening on this one website, even though as far as I am aware nothing had changed in the code before it started happenning. It is, however, our busiest website so it could be a traffic issue.
Debug Diagnostics hasn't revealed any issues.
Refreshing certain pages very quickly causes the memory to jump up rapidly, then fall slightly, but all the time the gradual progression is upwards.
I cannot replicate the issue on our test servers or locally. Probably because the traffic has something to do with it.
My suspicion is that the problem lies in database connectivity / locking. However, I'm not sure how that would cause the problem specified.
Any ideas?
Edit
Okay so not exactly sure I've found the problem but we're getting closer. It's definately SQL related. The error log reveals lots of errors since last thursday.
It all happened after we ran some windows updates on our servers. One of the updates failed on the SQL server so not sure if this caused some problems.
The warnings we're getting are:
SQL Server has encountered XX occurence(s) of I/O requests taking longer than 15 seconds to complete on file .. tempdb.mdf
Where XX is anything between 17 and 90! Does that sound like a deadlocking issue?
Followed by the following erors:
Unable to complete login process due to delay in opening server connection
These coincide with our log times for when the websites have been "blipping".
We've increased the page file size on SQL server to the recommended size, as it was set to a max of 4Gb, but recommended was 12Gb. I think we may need to roll back the windows updates we did on Thursday if that doesn't fix it.
Unfortunately I can't get into Activity monitor as it tells me Timeout expired!
Edit
Okay after a reboot I'm into Activity monitor. How many sleeping processes would you say would be normal? We have roughly 127 sleeping. That's serving over 10 websites.
If there is a deadlock or timeout issue, will NHibernate not clean up its connections properly?
Okay so in the end it seems it's quite complex. Sql deadlocks and data problems, heightened it seems by anti-virus software that was locking up or choking on a file.
Turning off the anti-virus reduced the problems, but we still need to resolve the underlying data issues.
An ASP.NET web app running on IIS6 periodically shoots the CPU up to 100%. It's the W3WP that's responsible for nearly all CPU usage during these episodes. The CPU stays pinned at 100% anywhere from a few minutes to over an hour.
This is on a staging server and the site is only getting very light traffic from testers at this point.
We've running ANTS profiler on the server, but it's been unenlightening.
Where can we start finding out what's causing these episodes and what code is keeping the CPU busy during all that time?
Standard Windows performance counters (look for other correlated activity, such as many GET requests, excessive network or disk I/O, etc); you can read them from code as well as from perfmon (to trigger data collection if CPU use exceeds a threshold, for example)
Custom performance counters (particularly to time for off-box requests and other calls where execution time is uncertain)
Load testing, using tools such as Visual Studio Team Test or WCAT
If you can test on or upgrade to IIS 7, you can configure Failed Request Tracing to generate a trace if requests take more a certain amount of time
Use logparser to see which requests arrived at the time of the CPU spike
Code reviews / walk-throughs (in particular, look for loops that may not terminate properly, such as if an error happens, as well as locks and potential threading issues, such as the use of statics)
CPU and memory profiling (can be difficult on a production system)
Process Explorer
Windows Resource Monitor
Detailed error logging
Custom trace logging, including execution time details (perhaps conditional, based on the CPU-use perf counter)
Are the errors happening when the AppPool recycles? If so, it could be a clue.
It's not much of an answer, but you might need to go old school and capture an image snapshot of the IIS process and debug it. You might also want to check out Tess Ferrandez's blog - she is a kick a** microsoft escalation engineer and her blog focuses on debugging windows ASP.NET, but the blog is relevant to windows debugging in general. If you select the ASP.NET tag (which is what I've linked to) then you'll see several items that are similar.
If your CPU is spiking to 100% and staying there, it's quite likely that you either have a deadlock scenario or an infinite loop. A profiler seems like a good choice for finding an infinite loop. Deadlocks are much more difficult to track down, however.
Process Explorer is an excellent tool for troubleshooting. You can try it for finding the problem of high CPU usage. It gives you an insight into the way your application works.
You can also try Procdump to dump the process and analyze what really happened on the CPU.
Also, look at your perfmon counters. They can tell you where a lot of that cpu time is being spent. Here's a link to the most common counters to use:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/852720c8-7589-49c3-a9d1-73fdfc9126f0.mspx?mfr=true
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/be425785-c1a4-432c-837c-a03345f3885e.mspx?mfr=true
We had this on a recursive query that was dumping tons of data to the output - have you double checked everything does exit and no infinite loops exist?
Might try to narrow it down with a single page - we found ANTS to not be much help in that same case either - what we ended up doing was running the site hit a page watch the CPU - hit the next page watch CPU - very methodical and time consuming but if you cant find it with some code tracing you might be out of luck -
We were able to use IIS log files to track it to a set of pages that were suspect -
Hope that helps !
This is a guess at best, but perhaps your development team is building and deploying the application in debug mode, in stead of release mode. This will cause the occurrence of .pdb files. The implication of this is that your application will take up additional resources to collect system state and debugging information during the execution of your system, causing more processor utilization.
So, it would be simple enough to ensure that they are building and deploying in release mode.
This is a very old post, I know, but this is also a common problem. All of the suggested methods are very nice but they will always point to a process, and there are many chances that we already know that our site is making problems, but we just want to know what specific page is spending too much time in processing.
The most precise and simple tool in my opinion is IIS itself.
Just click on your server in the left pane of IIS.
Click on 'Worker Processes' in the main pane. you already see what application pool is taking too much CPU.
Double click on this line (eventually refresh by clicking 'Show All') to see what pages consume too much CPU time ('Time elapsed'
column) in this pool
If you identify a page that takes time to load, use SharePoint's Developer Dashboard to see which component takes time.