We have an ASP.net website built in C#/VB built on reporting services. Some of these are local (rdlc) and some sit on a reporting server (rdl).
The problem we are running into is that about every two weeks the server starts reporting OOM errors and the IIS worker process is running away with memory. The quick fix seems to be restarting iis, but this requires manual interaction and is usually reported by users.
It seems like a memory leak somewhere, but most of the reports are really simple data pulls and connections are all closed, at this point we don't really know how to debug it further.
Any ideas?
You can run the IIS Debug Diagnostic Tool to see what the offending handle is http://support.microsoft.com/kb/919789
Related
When I get a call there was an error in our web application when running on a windows server I would simply go to Windows Logs and scroll down the application logs and find the Warning. Easy to read and made it quick to solve the problem.
Now I'm testing out Azure Websites to run asp.net applications and i'm racking my brain trying to figure out how to have the same kind of quick error find a fix process.
Viewing the application logging in Azure's application diagnostics is utterly useless.
It would take me forever to dig through those logs. rewriting the application to handle errors differently is not an option for me.
So who else is having this problem? any solutions?
The easiest solution is to offload your error logs to a separate service like Raygun. This will result in a (small) monthly fee but is a 5 minute setup and you are good to go.
Otherwise you can go and use Log4net to log your error to the blob storage but then you will have to poll that one for your logs.
Try the Azure Website Log Browser site extension, you can install it from the new Azure Portal on your website (as a site extension) and it makes it easier to view all of your logs.
Read more about it here.
We have a function in C# that uses the ICSharpCode SharpZipLib BZip2 decompression method to uncompress some XML we get from a database. We have noticed an issue on two of our webservers (Win 2K and Win 2003 Svr) that this code takes a really long time to execute and causes the CPU utilization to max out on these servers. We have isolated the code and put it into a Winforms app for testing and when running this same code in the winforms app on the same machines the code runs much faster. This is with the same SharpZipLib assembly.
So far we are at a loss as to why the code runs so much slower under the asp.net process.
Any suggestions, ideas ?
Thanks in advance!
Use a profiler. Problem might be somewhere outside of the decrompession code. You, I and the rest of the folks at SO won't give any more info than one profiler session will give.
Have you tried watching the code in a Profiler like RedGate ANTS Performance Profiler to see what kind of things the code is doing?
Code hosted inside of an ASP.NET application is run under completely different conditions than a WinForm application. You could be running into ThreadPool issues when loading through the IIS worker process.
You could also be running into an issue where your code is not pre-compiled in your ASP.NET application.
All that being said, running your application through a Profiler is going to provide you with the best view into how things are working.
Make sure you are creating a web project and compile it for release build. (Build -> configuration manager)
Working with one of our partners, we have developed now two separate sets of web services for their use. The first one was a simple "post to an https URL" style web service, which we facilitated by building a web page in ASP.NET that inspected the arguments in the URL, and then acted accordingly. This "web service" (if you can call it that) has been very stable.
At some point, the partner asked us to begin using SOAP based web services. At their request, we built them a new set of web services largely based on the previous objects, reimplemented as an actual "Web Service". This web service has not been very stable: around once a week, Nagios will alert us that our web service is not responding - and a quick iisreset does the trick.
Analyzing the log output and working in a debugger has not led us to anything concrete. The volume on this new web service is actually much lower than the HTTP web service. I think this could be a code problem or a platform problem, or of course something in between.
We've tried, with little improvement:
To duplicate the behavior in the lab
Debugging in the Visual Studio debugger
Tinkering with IIS options to give it its own application pool
My question, what are the next steps for troubleshooting?
Environment:
Windows Server 2003 Standard Edition R2 Service Pack 2 32 bit, Visual Studio 2005, MS SQL 2005, .NET Framework 2.0.50727
You may get some answers by profiling your webservices and understanding how they are using their resources. perfmon and procmon are both very useful tools in this regard.
EDIT: Since you say errors happen after about a week, the only thing I can think of is resource usage. Ensure your DB connections are being cleaned up, and any opened files (system call to the exe) are being closed.
Also, if your webservices can tolerate it, IIS has a setting that triggers a periodic recycle of an App Pool to handle cases where performance degrades over time. Its dirty, but it may work well for your case.
Since there isn't much to go on - here's another odd issue we came up against regarding our web services.
When the web service stops responding how is memory utilization? We have experienced issues with memory and memory fragmentation relating to busy web services on a system (there was also other things running causing additional fragmentation). When we re-factored the web services to load from smaller dll's and depend on other libraries (instead of one large library) we were able to resolve the memory fragmentation.
To identify what was occurring we would take a dump from the offending iis worker process where the app pool resided and then reviewed that using WinDbg.
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
Additionally we used DebugDiag to take the postmortem dumps.
http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1286
Hope this provides another direction to look at.
I've a a customer using IIS and an application, developed by us, in ASP.NET 1.1.
Monday, for 4 times in a row the following error has appeared:
A process serving application pool 'xxxx' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was 'yyyy'. The data field contains the error number.
Any idea about how to diagnose this? The only link I've found talks about installing low level debug tools but before proceeding in this kind of low level analysis I would know if someone has a better idea or a suitable alternative.
The problem (from what I can see) is something in the customer environment, because the same application it's installed at other customer sites on at least 20 or 30 different servers and the problem do not happen.
Had the same issue on IIS 7, had few reports all worked except one report that was very long and it never worked on IIS7 (it was fine on low spec server).
on IIS7 in application pool's Advance Setting I set "Enable 32-Bit Application" to true and all worked nicely
I'm getting the same error, here are some more details:
running: Windows Server 2003, IIS 6.0 / ASP 3.0,
2.13 GHz, 1 GB ram
My web site is in Beta, so I hardly have any visitors to the site.
According to the event viewer I'm getting this warning 3 times every 3 minutes,
then it stops for a few hours.
Then sometimes I get the error:
A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '3900'. The process exit code was '0x800703e9'.
which follow by:
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Which then causes a 'Service Unavailable' message when browsing to the website.
after reading too many post about this issue, I did the following steps:
I read that it might be registry access right, so I installed a monitor and trace all the W3SVC Access Denied errors and grant permition
I read that the 0x800703e9 error means stack over flow which causes a w3wp.exe crash and I should install a Debug tool and try to get a memory dump.
I did that but I didn't get any dump, so I installed a new debugging tool, but didn't got a crash yet.
My web site is doing some data mining which keeps the server busy.
Conclusion:
I don't know what is going on there... but I do know that my server machine is way to slow on resource, so I'm going to upgrade and reinstall it, I'm certain that it will solve the problem...
The problem happens all the time, even when my .net code is idle, therefore it is a problem in the server, and not in my code.
I think that the first warnning “A process serving application pool ..." happens every some time, and every now and then it causes the Application Pool to restart, therefore a attaching a debugger doesn't help - the process keeps restarting and the debugger is not effective any longer...
I think that the 0x800703e9 error (which causes the Service Unavailable) maybe happens when the app pool restart, I guess that it needs a lot of resource and since my machine is too slow it gets the 0x800703e9 ... as said before this is a stack over flow, but I think that it is caused by low resource and not by an endless recursion.
I think that the 'registry access right' which is claimed by Microsoft to be the issue, is nonsense, but I didn't got 'Service Unavailable' since so it might helped (thought I still get the warnning “A process serving application pool ..." ).
Hope this help someone...
I'm sure you already know this, but the App Pool contains 1.1 applications only right? I don't remember the error you get when the pool dies by trying to mix frameworks (something like Server Unavailable), but it's more common then I thought in the wild so I would double check.
While not likely the case, it's somewhere to start.
Edit: This KB article also had the error message you described relating to registry permissions, what version of IIS is the client running?
I had this same problem when a website was deployed to a client's web server. This Microsoft support article says:
"This issue may occur if the NT AUTHORITY\NETWORK SERVICE account does not have the permissions to the required registry keys."
And the resolution is: "set the permissions to the required registry keys, and then restart IIS 6.0."
The linked article has the steps for doing this.
One more popular reason (as in my case) - one of windows logs is full.
After deploying a new build (mostly changed DLL's) of an ASP.NET web app the CPU on the server is now jumping to 100% every few seconds and the culprit is lsass.exe. Do you think that the deployment of the asp.net web app to the server and this problem are related? (or a coincidence that it happened at the same time?)
More info:
This is the first time that I've done the build on a Server 2008 x64 machine. Previous the builds were done on a Server 2003 x86 machine. Target is "Any CPU" so should work on either. Deployed to server is Server 2003 x86.
I've searched the web for more info on this and have confirmed that the process is lsass.exe (first character a lower case L and not an upper case i) so ruled out the virus version. Found some docs relating to a Server 2000 bug but doesn't apply here.
I eventually isolated the problem to an ASP forum running "under" that ASP.NET web app. Using the admin page on the forum I took the forum down and then brought it back up again and the problem disappeared. I find this very frustrating because the problem has now gone but I don't know what caused it and as such it could easily return.
I also installed this Microsoft Hotfix and rebooted this server but that didn't work.
Have you checked the System and Application event logs for anything unusual?
Have you updated to use Active Directory role provider? I've seen issues where enumerating groups to do role checking pegs the CPU and really slows down the app. I actually implemented a customized provider that allowed me to specify a particular OU and set of groups that I actually care about to get around this issue.
The xperf tools distributed in the Windows Performance Toolkit will tell you exactly what is usin CPU time or disk bandwith. These tools are free and work on any retail build of WS2008 or Vista. Here is series of posts on the xperf tools from myself.