I've been meaning to get on here for ages, alas, my recent problem is motivating me, hoping to contribute and not just sponge, but hopefully someone can assist with this issue.
We have two 2003 web servers (looking to upgrade) running IIS6 running multiple ASP.net applications on one web site (via Microsoft NLB).
We have a persistent issue with one application, that is, the white screen of death, the application just hangs, no errors.
Things I have tried / observed:
Taken servers out of NLB
Checked HTTPERR for errors at time of issue, some bad requests and connection abandoned by app pool present
IIS logs show requests coming in
Checked database for locking PID's or performance, everything ok
Checked network connections and links to other API's all good
Checked other websites on server - not working ok
Event app logs no errors found at exact time of issue
Split out app pools, found the problem application
IIS reset and shutting down all IIS process and services gets it working again
So next when application hung, recycling that specific app pool gets it working for a short period
'SOME' parts of the same application will work, even though the main function hangs and gives the white screen of death
Other things to note:
Server has 4GB ram, terrible I know, 4gb page file, 1.7gb total in use
CPU usage always <12%
Resources not 'really' a major factor
Only one w3wp for this app pool (been running like this for years)
After setting the app pool to recycle on an hourly basis and tweaking some app pool settings, the issue seemed to go away for at least six months. Now it's back for whatever reason, no updates, no changes in application or infrastructure.
I've upped the worker process for the app pool to 2, the issue still persists after a period of time
I observed THREE worker process at random intervals when checking not the TWO set up, could be part of the recycle process (one starts up, other shuts down) and I caught it at that moment.
Killing the highest RAM usage W3WP re-instates application instantly when new process starts up for that app
Common RAM usage when site failing appears to be apprx 210mb on one of the applications W3WP
I have now set recycling rules instead of hourly to recycle at a more purposeful 200mb. Why this would be a limiting factor I do not know.
I'm struggling to get to the bottom of it, but it doesn't appear to be network, hardware, NLB, Database, or any other core factors. It is specific to a worker process for this app pool hanging / crashing and not processing requests for a certain part of the site.
I am running an effort to upgrade to 2008 in the next few months (version of ESX does not support 2012 and that upgrade is not in scope). However, I'd really like to work out what the issue is rather than using web deploy to deploy a dodgy installation of a legacy application.
Any advice or help appreciated.
Related
I have an ASP.NET MVC 5 application running on Windows Server 2012 on IIS 8.
Every so often my site gets ridiculously sluggish to the point where it takes at least 5 seconds just to load the basic About page. It tends to feel as though it's running in Debug mode, even though my deploy process is automated, so I'm not accidentally deploying in Debug mode, and I can find no signs that it's in Debug mode. I've sometimes wondered if it's a memory issue, but I don't see the server memory getting out of control. I've also wondered if it's something weird with IIS settings such as Application Pool recycling (or something to that effect). I've been dealing with this issue for over a year, not knowing where to look next. I have MiniProfiler running in my development and staging environments, so I'm pretty familiar with the performance of my various site pages etc. Also, I have another web app on the exact same IIS server that doesn't seem to suffer from the same problem. The only difference I can think of between the 2 is that for one site I use Ninject for dependency injection, but I don't for the other site.
I'm really at a loss, but the only "cure" when the site gets bogged down is to completely delete the application files for the site and redeploy. This does the trick every time and then the site will act snappy again (loading the About page in under 1 second). I seem to have to repeat this ritual every week or 2, or at least once a month (whenever I notice the site is at a crawl).
I have an application that has a application pool that shuts down a couple of times a day and I can't figure out why. the website is hosted on a VM with SQL Server R2. the version of IIS is 7 and the website has a dedicated application pool using .NET framework v4.0.30319.
At the application level the website is using a target framework of 4.5 and seems to run just fine for a 24-48 hour period and then for some reason the application pool just stops. We do have other .NET 4.5 applications running on this machine and their application pools do not have this problem. The only thing slightly different about this application is that we initiate a separate thread to send a order confirmation email when an order is completed. We average about a couple of hundred orders a day for this website.
According to the event log the error reported prior to the shutdown is
"A process serving application pool 'RetailSupport' suffered a fatal communication error with the Windows Process Activation Service"
Please help
I would check, if your code could hit under some circumstances an endless loop or a recursive
call of methods. This is often the reason for this error.
What always helps me in such cases is to add debug logs (e.g. log4net) in your case especially before the email is sent, status info about resources, etc. They can be more informative then the IIS logs.
If this does not help, check http://support.microsoft.com/kb/919789/en how to use the debug Diagnostic tool, to get more details for the error.
Look also for IIS: Web Application hangs periodically needs system reboot it could help you.
Also: what recycle strategy on IIS do you have?
I developed a project in ASP.Net MVC 3, my hosting is using iis7 (Win Web Serv 2008 R2), and the first request after the website sit's idle (during about 1-2 hours) is very slow.
I use VPS with 512Mb RAM. Can this be related with a too little RAM?
Can anyone help me with possible causes of such behaviour?
After a certain amount of inactivity IIS unloads the AppDomain. And then the first request loads the application once again which is slower. You could try to configure this period in the properties of IIS but there might also be other causes that an application unloads such as for example a certain threshold of CPU or memory usage is reached. Those thresholds are also configurable in IIS.
That's not something specific for ASP.NET MVC. It's true for all ASP.NET applications in general.
We had also this problem with ruby and passenger that takes the app out of memory after a while, but I found a nice application that fixed this issue for us without changing anything in the server configuration, the app is called wekkars, and you can find it here: http://www.wekkars.com
I've read recommendations that we should create separate application pools for each asp.net application on our Win2008 server.
We have about 20 apps that would be on the same server. I know this would create 20 separate worker processes which seems very wasteful.
Is it good practice to create separate application pools for each application?
Reposted from ServerFault, "Why add additional application pools in IIS?"
AppPools can run as different identities, so you can restrict permissions this way.
You can assign a different identity to each app pool so that when you run task manager, you know which w3wp.exe is which.
You can recycle/restart one app pool without affecting the sites that are running in different app pools.
If you have a website that has a memory leak or generally misbehaves, you can place it in an app pool so it doesn't affect the other web sites
If you have a website that is very CPU-intensive (like resizing photos, for instance), you can place it in its own app pool and throttle its CPU utilization
If you have multiple websites that each have their own SQL database, you can use active directory authentication instead of storing usernames/passwords in web.config.
Separating apps in pools is a good thing to do when there's a reason, and there are a number of good reasons listed above. There are, however, good reasons not to separate apps into different pools, too.
Apps using the same access, .NET version, etc. will run more efficiently in a single pool and be more easily maintained. Most annoyingly, IIS will kill idle app pools, requiring the pool be recreated on each use. If you isolate infrequently used apps you'll impose an unnecessary startup cost on users. Combining these apps into a single pool will make for happier users when they don't pay the startup cost, happier servers when they don't give memory to multiple processes and CPU slices for them, and happier admins when they have to manage fewer app pools.
I used to have 58 .Net websites and 17 old classic ASP websites on the same IIS7.5 server using separate app pools for each site. I noticed that the IIS compression started failing intermittently, causing the style sheets to be corrupted about 5% of the time. Looking at the task manager on the server, I could see that the server was approaching it's 4GB ram limit- each w3wp.exe process was taking anything up to 100 MB memory depending on how much traffic the site was getting. I then moved all the websites into just 2 application pools (one for .net 4 websites and one for the old classic ASP sites) and the total memory used after doing that dropped from 3.8GB to just under 2.8GB - saving me over 1GB memory space on the server. After the change (and leaving the server running for a couple of hours to get back to normal levels of traffic), the w3wp processes were using 300MB for all the .net websites websites and 20MB for the classic ASP websites. I could re-enable IIS compression again without a problem.
Using separate APP pools is a great idea for many of the reasons mentioned by the other posts above, but it also in my experience causes a much higher memory overhead if you are hosting a fair number of websites on the same server.
I guess it's a trade-off between hardware restrictions and security whether you want to use separate app pools. It's a good idea if you have the resources to do it.
yes, it is a good idea even for 20 applications.
Security. Different app pools running under different accounts.
Isolation. One crashing app won't take down other apps.
Memory (if you are running 32bit). Each app pool will have its own address space. Thus you can address much more memory than a maximum of approximately 2.7GB of usable space for 1 process.
You may choose to periodically restart one not-so-well behaving app without affecting other applications.
If your apps are stable and don't use much memory, then I would say that it's fine to put them in the same app pool. App Pools give you isolation between your applications.
One main reason I consider when creating app pools is the process management. There are other reasons such as security etc.
When an IIS-hosted application crashes it takes down its host process as well. In previous versions of IIS this meant all web activity crashed together. With app pools you can isolate your applications from one another. If one has a memory leak and keeps crashing your other apps will continue to function.
The main benefit of creating different application pools is that you can provide each pool with other credentials. Your 20 applications may communicate with 20 different databases that all need another login. The best practice then is to run each application using a different service account.
I wouldn't worry too much about performance. Most time will probably be spent inside each web application, no matter what process each application is in.
This really depends on the applications, your security model, and how much you trust the applications.
Here are a few things that I always tell people to consider when working with application pools.
Use separate application pools if each application needs different access to system resources. (Can use multiple process accounts)
If an application is a resource hog, mission critical, or an "unknown", it is best to put it in to its own pool to isolate it from the rest of the system
We work as outside contractors for a large corporate client. We are not on-site, so we do not have connectivity to all of their systems. Sometimes during development it is necessary for me to debug an application directly on their development servers. To do that I must be able to attach to w3wp process. When I attach and start debugging, entire process stalls, which affects all of the applications that are in the same process/application pool.
By creating a dedicated application pool and moving my development there, I can easily debug without making anyone's life miserable.
Scenario: I've an n-Tier enterprise ASP.NET application deployed using Web Deployment Projects. All tiers produce independent assemblies that is consumed by the ASP.NET application.
Problem: When I run the app. for the first time after deployment it takes lot of time to load dependent assemblies in memory. But once loaded its lighting fast app. In case if there are no users accessing the app, IIS unloads the assemblies from the memory and when a user tried to access the app on a later instance it goes on loading all the assemblies once again taking the same amount of time to load as it takes to do so for the first time.
I'm looking for a solution that enables me to keep my assemblies loaded into memory persistently overriding the volatile nature of assemblies towards memory residency.
Or any other solution that lets my users happily use the app resolving the mentioned problem.
In IIS 6, go to the Application Pools section, and right-click > Properties on the pool which hosts the ASP.NET application in question. Go to the Performance tab and uncheck "Shutdown worker processes after being idle for:"
In IIS 7, go to the Connections pane and find Application Pools, and select Advanced Settings for the pool which hosts your application. Find the "Idle Timeout" property and set it to "0" (this disables it).
The default is 20 minutes of inactivity. By unchecking the box, once your AppDomain is loaded by the worker process, it will never die (unless you kill the process or something of course). By default, IIS will recycle the process when it reaches some limit, such as a memory cap, but it will also start a new one and "phase over" all incoming requests until the old one is unused, so as to minimize disruption.
I've also written a small c# class which will keep your ASP.NET application alive (alternate archived version) under normal circumstances. Since it runs within the application, obviously it can't stop IIS or anything else from explicitly killing the process, but it will keep the application "hot", e.g. the app will never go idle long enough for IIS to decide to shut it off.
If you do not have direct control over your IIS configuration (shared host, for example) your best bet is to have a small application running on a separate system - say, an always-on workstation - which hits your site every x minutes to keep the application pool from timing out. Nothing fancy - a simple WebRequest and a while() loop in a console application will do.
One of the advantages of ASP .net is the posibility to create static (shared) instances of objects.
To avoid the necessity of an external process you can create an static timer in (per example) global.asax which calls for a page on the domain with a simple WebRequest. On this way the site keeps alive himself until a manual reset of the pool is done.
I wrote a little C# console application that keeps my 4 sites alive every 10 mins via windows task scheduler. Life is once again good. We do not run the app from 2-5am just so the serves can do any cleanups of memory, if it even matters. for our sites there is rarely anyone on at those hours anyway.