I maintain a Sitefinity 5 site that is fairly heavily customized, which I think plays a part in my problem, that takes upwards of 10-20+ minutes to become usable again after an app pool recycle is initiated. Sometimes I'll even have to kill the w3wp process for that specific app pool.
I'm wondering if there isn't a setting somewhere on the site/app pool that is causing problems.
So, I decided to create a new site, same codebase (different folder) with a non-routing hostname so only I can hit it, and it loads/recycles relatively quickly (1 - 2 minutes).
Does amount of traffic play into how quickly an app pool recycles?
Any other ideas?
I think the amount of traffic plays here, because once the app pool restarts, all cache is cleared and if there are many requests at the same time - they will all try to fetch the data from the DB because the cache is still empty.
In the recent version (7.3) there will be a switch that will control this. It will work in a way that the first request will enter a lock section and only that request will be able to get the data from the DB, then it will populate the cache and release the lock.
The other queued requests will then read from the cache.
More details are here: http://www.sitefinity.com/blogs/boyan-barnevs-blog-/2014/11/28/improve-the-application-start-up-and-the-effects-of-page-templates-changes-for-websites-with-heavy-load
Related
I am having some performance issues with my iis webserver. It is hanging randomly and I am trying to figure out how to speed up the server. I enabled Failed request tracing on the server and set it to generate a log when the request is over 3 seconds.
The resulting logs(xml) dont show much but there is a point in the compact performance log that indicates what part of the log the server is hanging on. Below is the part of the log where the large time loss is occurring.
65. i GENERAL_GET_URL_METADATA PhysicalPath="", AccessPerms="513" 17:46:32.577
66. i HANDLER_CHANGED OldHandlerName="", NewHandlerName="ExtensionlessUrlHandler-Integrated-4.0", NewHandlerModules="ManagedPipelineHandler", NewHandlerScriptProcessor="", NewHandlerType="System.Web.Handlers.TransferRequestHandler" 17:46:32.577
67. i VIRTUAL_MODULE_UNRESOLVED Name="FormsAuthentication", Type="System.Web.Security.FormsAuthenticationModule" 17:46:47.771
I am not sure what Handler changed is but it is taking a long time, any tips would be great on where to start looking.
It is hard to come up with a solution without having any piece of code in sight. Here are some general hints/tips you can follow in order to have great performances with an ASP.NET application.
The fastest way to do a request is to not do it in the first place. Try caching everything that can be cached. There are server-side caches and client-side caches. Each have their own uses, but you are not limited to only one type.
Make sure you do not cache and/or keep references of any request-related objects into memory. ASP.NET have a limited number of concurrent requests and keeping a request reference in memory will hang your server if it runs out of threads
Close the request as soon as you are done with it
Everything that is not needed by the client at the time of the request should be done in the background
Make sure you have no memory leak in your application. Garbage Collections are often the cause of hangs in ASP.NET application. When garbage collecting, all running threads are paused. This is especially true for Gen 2 garbage collections. You can enable background generation 2 garbage collections.
Isolate the problematic code. Use a profiler and see which type of request is CPU-intensive. Then dig deeper and see what inside that request makes it slow.
In any well-balanced application, objects should either be short-lived and live forever. In the case of an ASP.NET application, the objects created during the course of a request should ideally die within that request or during the next GC gen 0.
Consider object pooling for large objects and objects that are long to initialize
Make sure your app pool doesn't totally crash and restarts (look the IIS logs and/or the Windows Events)
Some useful debugging tools you can use:
LeanSentry. Great for diagnosing ASP.NET server hangs
windbg. High learning curve but by far the most powerful debugging tool you can use
PerfView. Useful for analyzing ETW events like I/O or CPU usage
There are many ways to improve server performance. But before that you should start with checking CPU usage during the "hang". An infinite loop in the application code may cause this behavior. Unless there is I/O, locking, or sleeps in the loop, you will be able to see it from the CPU usage as you will get exactly one full core's worth of CPU usage for each infinite loop.
Help link to improve server performance
More Info:
I can see entry related to VIRTUAL MODULE UNRESOLVED: which is related to bad use of Response.Redirect(url); Also make sure you have deployed your app on integrated mode on IIS.
here's a simple checklist you might want to reconsider:
Always pre-compiling your site, as opposed to copying it! you might gain a significant performance boost compiling your website before deployment: ASP.NET Precompilation Overview
Do not run the production application with debug="true" enabled, when debug flag is true in your web.config, Much more memory is used within the application at runtime, and since some additional debug paths are enabled, codes can execute much slower
Check your Web.config file to ensure trace is disabled in the section
IIS 7.5 comes with the Auto-Start Feature. WAS (Windows Process Activation Service) starts all the application pools that are configured to start automatically, ensure that your application pool is configured to AlwaysRunning in the IIS 7.5 applicationHost.config, check out here for more detail.
Every asp.net server can be well configured by aspnet.config file located in the root of the framework folder. Ensure that Publisher Evidence for Code Access Security (CAS) is set to false in your aspnet.config file, This might increase the initial page load when you restart the ASP.NET app pool. you can read more about it here.
Also you might want to try Application Initialization Module for IIS 7.5, this module also available on IIS 8.0 can decrease the response time for first requests by pre-loading worker processes
My shared hosting provider set up IIS recycle app pool every 3 minutes for idle.
So my session factory often recreates (at application startup). As I have about 70-100 entities it takes about 2-5 seconds to construct factory. So cold start of my application is rather long. I haven't access to IIS setting.
You can offset a lot of the cost of setting up your factory by generating your proxies at build-time instead of runtime. This article explains the steps how.
Being realistic, the simplest change is to ask that the app-pool isn't recycled so frequently (since this is an expensive operation for your application). I'm sure they've set the timeout very low as a "performance" setting, but really this is generating work and slowing things down.
You might not have access to the IIS settings directly, but this shouldn't stop you from contacting your supplier's technical support and getting it resolved.
If you are in a full trust environment (doubtful, but provider may be willing to work with you on this), you can try serializing your configuration so it doesn't need to be rebuilt each time. Merging all your entity mappings into a single XML doc can help also (just do this as build step so its not a nightmare to work with mappings).
More info here: http://nhibernate.info/blog/2009/03/13/an-improvement-on-sessionfactory-initialization.html
Have you tried to stop your site from being idle in the first place? I use uptime robot that is FREE and pings your site every 5 minutes. The benefit of this service is that it only requests the headers of the page you set up as a monitor and therefore does not affect logging such as Google Analytics.
However said you will need to test this to see when your app does indeed recycle to see if uptime robot works with your shared hosting provider. The best way is to log every time the session factory is re-built.
not much you can do. app pool recycle shuts down your app...
I guess you could try to fool the recycler by having the application do something every 2:45.
I'm trying to implement some fail safes on a client's web server which is running two of their most important sites (ASP.NET on IIS7). I'm going to set up application pool limiting so that if any w3wp process uses 90%+ CPU for longer than a minute then it gets killed (producing a temporary 503 Service Unavailable message to any visitors), and based on my local testing will be restarted within a minute - a much better solution than having one CPU-hogging process taking down the whole server for any length of time.
This seems to work, however during my fiddling on my local IIS7 instance I've noticed that if a request calls my "Kill.aspx", even when the site comes back up IIS will not serve the session that caused it to hang. I can only restart the test site from a different session - but as soon as I clear my cookies on the "killer" browser I can get to the site again.
So, whatever malicious behaviour IIS is trying to curb with this would not work against an even slightly determined opponent. In most cases, if excrement does hit fan it will be coding/configuration error and not the fault of the user who happened to request a page at that time.
Therefore, I'd like to turn this feature off as the theoretical user would have no idea that they need to clear their cookies before they can access the site again. I would really appreciate any ideas on how this might be possible.
Yous should be using ASP.Net Session StateServer instead of In-Proc (see msdn for details). That way, you session will run in different process and won't be affected by IIS crash.
Turn what "feature" off? If the worker process is reset (and your using in-proc session) then the session is blown away on a reset.
You might want to investigate moving your session storage to a state server or some other out of process scenario.
Also, you might want to set the application pool to use several worker processes (aka: web garden) this way if one process is killed the others continue serving content.
Next, as another option you might want to set up multiple web servers and load balance them.
Finally, you might want to profile the app to see exactly how they are causing it to spin into nothingness. My guess is that there are a number of code issues you are simply covering up with this idea.
As best I can tell when a worker process recycles:
a) a new one spins up before the old one shuts down
b) the old one shuts down once all the active requests its servicing completes
Is the above accurate?
If so, I have data that I store in SQL once Application_End() fires from the global.ascx file. I pull this data back in when Application_Start() fires.
The problem is based on my testing, the new worker process fires the Application_Start() before my old worker process gets a chance to complete its Application_End().
What are best practices for handling this situation?
cheers in advance
edit: I just noticed a feature on IIS 7 'Disabled Overlapped Recycle' - I'm guessing this is the best route
Your description of overlapped recycling is accurate, yes (1); and there is a setting for disabling it, but it's intended to prevent HTTP errors which you would be re-introducing. App pool recycles are a normal occurrence for managed apps (stems from, among other things, a CLR limitation that prevents the unloading of assemblies in the same memory space) that you must design for.
Your technique would be difficult to manage in a web-farm or web-garden scenario.
I think a better design would be to rely on out-of-process storage for the data (using distributed cache products like ScaleOut, App Fabric, and the like) so that all app pools have the same view of the cached data.
(1) -
http://mvolo.com/blogs/serverside/archive/2008/02/25/Starting_2C00_-stopping-and-recycling-IIS-7.0-Web-sites-and-application-pools.aspx
A website that I'm working on is making extensive use of the HttpContext.Current.Application store to cache lookup data from the database. (This isn't a lot of data - just simple things that are used in drop down lists etc..)
A few questions about this approach:
Does this store have a lifetime (in the same way that a user's session will time-out after 20 minutes, does the application store timeout)?
Do events in IIS or events in the same App Domain as the website cause the Application store to refresh? (I understand that changing a value in the web.config file causes the store to be re-created.. are there other things that do this?)
Is there a better way to do this?
Web.config causes the application to restart, and it is that action which is clearing out HttpContext.Current.Application.
Anything that restarts the application, touching web.config, manually doing it in IIS, having the application pool recycled by IIS will "clear" that data.
With that in mind it is still the most pernament in-memory store available, so if you need something more pernament you will have to persist your data to disk or to a database somewhere.
You probably don't really need to use it though (we rarely do), usually the best tactic is to use the application cache and rebuild resources as required. As long as your server isnt under high memory stress then cache will not be dropped either (unelss an expiry is set).
My understanding this is available for the application lifetime, or until the application pool is reset.
There is a good article here on all the various reasons the application/session pool in IIS would be recycled.
There are settings in both IIS and also web.config which can change the behavior of the application store availability.