I have a number of ASP.NET (4.0) web applications that appear to leak (a small amount) of memory during each request. It is such a small amount, that for most use-cases, it will not grow to become a problem for weeks or even months at a time. I generally try to be good with closing any connections managed by the application, avoiding state-variables (or instance variables for my singleton), etc.
My question is this - is this normal behavior for ASP.NET applications? I had turned off the default (IIS 7) behavior of recycling the app pool after 20 minutes of being idle. I do this since the application takes a few minutes to build its internal cache, and I want to avoid negatively impacting the user experience (and having them wait for the application to start when they issue the request).
I am aware this could be mitigated by serializing the cache or speeding up the cache generation process but my question has more to do with the principle of it: I personally consider relying on the IIS auto-recycle feature as a bandaid approach. Am I wrong? am I just not seeing the garbage collector at work because the application's memory usage is not high enough compared to the amount of available memory? or should I dig deeper into the memory issues?
Any insight would be appreciated.
Unfortunately it is normal, though mostly dues to shoddy application writers than anything else.
IIS by default configures newly created application pools to recycle every 1740 minutes for this reason.
As you said, this is a band-aid. A well written application that clears up all of its resources (including dangling event handlers), shouldn't leak at all.
See this blog post about the subject.
Related
I am having some performance issues with my iis webserver. It is hanging randomly and I am trying to figure out how to speed up the server. I enabled Failed request tracing on the server and set it to generate a log when the request is over 3 seconds.
The resulting logs(xml) dont show much but there is a point in the compact performance log that indicates what part of the log the server is hanging on. Below is the part of the log where the large time loss is occurring.
65. i GENERAL_GET_URL_METADATA PhysicalPath="", AccessPerms="513" 17:46:32.577
66. i HANDLER_CHANGED OldHandlerName="", NewHandlerName="ExtensionlessUrlHandler-Integrated-4.0", NewHandlerModules="ManagedPipelineHandler", NewHandlerScriptProcessor="", NewHandlerType="System.Web.Handlers.TransferRequestHandler" 17:46:32.577
67. i VIRTUAL_MODULE_UNRESOLVED Name="FormsAuthentication", Type="System.Web.Security.FormsAuthenticationModule" 17:46:47.771
I am not sure what Handler changed is but it is taking a long time, any tips would be great on where to start looking.
It is hard to come up with a solution without having any piece of code in sight. Here are some general hints/tips you can follow in order to have great performances with an ASP.NET application.
The fastest way to do a request is to not do it in the first place. Try caching everything that can be cached. There are server-side caches and client-side caches. Each have their own uses, but you are not limited to only one type.
Make sure you do not cache and/or keep references of any request-related objects into memory. ASP.NET have a limited number of concurrent requests and keeping a request reference in memory will hang your server if it runs out of threads
Close the request as soon as you are done with it
Everything that is not needed by the client at the time of the request should be done in the background
Make sure you have no memory leak in your application. Garbage Collections are often the cause of hangs in ASP.NET application. When garbage collecting, all running threads are paused. This is especially true for Gen 2 garbage collections. You can enable background generation 2 garbage collections.
Isolate the problematic code. Use a profiler and see which type of request is CPU-intensive. Then dig deeper and see what inside that request makes it slow.
In any well-balanced application, objects should either be short-lived and live forever. In the case of an ASP.NET application, the objects created during the course of a request should ideally die within that request or during the next GC gen 0.
Consider object pooling for large objects and objects that are long to initialize
Make sure your app pool doesn't totally crash and restarts (look the IIS logs and/or the Windows Events)
Some useful debugging tools you can use:
LeanSentry. Great for diagnosing ASP.NET server hangs
windbg. High learning curve but by far the most powerful debugging tool you can use
PerfView. Useful for analyzing ETW events like I/O or CPU usage
There are many ways to improve server performance. But before that you should start with checking CPU usage during the "hang". An infinite loop in the application code may cause this behavior. Unless there is I/O, locking, or sleeps in the loop, you will be able to see it from the CPU usage as you will get exactly one full core's worth of CPU usage for each infinite loop.
Help link to improve server performance
More Info:
I can see entry related to VIRTUAL MODULE UNRESOLVED: which is related to bad use of Response.Redirect(url); Also make sure you have deployed your app on integrated mode on IIS.
here's a simple checklist you might want to reconsider:
Always pre-compiling your site, as opposed to copying it! you might gain a significant performance boost compiling your website before deployment: ASP.NET Precompilation Overview
Do not run the production application with debug="true" enabled, when debug flag is true in your web.config, Much more memory is used within the application at runtime, and since some additional debug paths are enabled, codes can execute much slower
Check your Web.config file to ensure trace is disabled in the section
IIS 7.5 comes with the Auto-Start Feature. WAS (Windows Process Activation Service) starts all the application pools that are configured to start automatically, ensure that your application pool is configured to AlwaysRunning in the IIS 7.5 applicationHost.config, check out here for more detail.
Every asp.net server can be well configured by aspnet.config file located in the root of the framework folder. Ensure that Publisher Evidence for Code Access Security (CAS) is set to false in your aspnet.config file, This might increase the initial page load when you restart the ASP.NET app pool. you can read more about it here.
Also you might want to try Application Initialization Module for IIS 7.5, this module also available on IIS 8.0 can decrease the response time for first requests by pre-loading worker processes
I've heard that only 1 cpu core can be used per asp.net 4.0 website in IIS 7. Using more cpu cores means entering webfarm territory (as a result session management should be done respectively). But I could not find any references confirming this.
So are there any limitations on cpu core count that can be used per website, where session still can be in-proc? Any references?
Absolut wrong.
The IIS7 and asp.net use all cpu and the power of the server. Also include in that and the SQL server that run in parallel with asp.net/iis and also use all the cpus and (all) the memory. Also if you make any system Thread, you also potential use a different cpu.
What you have "hear" probably is the session is blocking the asynchronous processing of the pages that is not totally bad you know, is help in many case. Again that is not mean that is use one CPU, only that the calls on the same site are synchronous (the one wait the other)
Few words about the session. Personally I have totally replace it with my custom made session handler, but for any beginner site and small sites the current session is perfect because is help you to synchronize the calls.
Without the current session module, you need to handle the synchronization by case manually - that is not so easy. If you not do that, the results is usually double and triple submissions of the same data (all that from experience).
Now if you design your site for web garden, and design it good, you take care of the synchronization of the calls and make it work fast, and correct.
Read about the session blocking on:
Web app blocked while processing another web app on sharing same session
Replacing ASP.Net's session entirely
What perfmon counters are useful for identifying ASP.NET bottlenecks?
Trying to make Web Method Asynchronous
I have a web application that uses ASP.NET with "InProc" session handling. Normally, everything works fine, but a few hundred requests each day take significantly longer to run than normal. In the IIS logs, I can see that these pages (which usually require 2-5 seconds to run) are running for 20+ seconds.
I enabled Failed Request Tracing in Verbose mode, and found that the delay is happening in the AspNetSessionData section. In the example shown below, there was a 39-second gap between AspNetSessionDataBegin and AspNetSessionDataEnd.
I'm not sure what to do next. I can't find any reason for this delay, and I can't find any more logging features that could be enabled to tell me what's happening here. Does anyone know why this is happening, or have any suggestions for additional steps I can take to find the problem?
My app usually stores 1-5MB in session for each user, mostly cached data for searches. The server has plenty of available memory, and only runs about 50 users.
It could be caused by lock contention for the session state. Take a look at the last paragraph of MSDN's ASP.NET Session State Overview. See also K. Scott Allen's helpful post on this subject.
If a page is annotated with EnableSessionState="True" (or inherits the web.config default), then all requests for that page will acquire a write lock on the session state. All other requests that use session state -- even if they do not acquire a write lock -- are blocked until that request finishes.
If a page is annotated with EnableSessionState="ReadOnly", then the page will not acquire a write lock and so will not block other requests. (Though it may be blocked by another request holding the write lock.)
To eliminate this lock contention, you may want to implement your own [finer grained] locking around the HttpContext.Cache object or static WeakReferences. The latter is probably more efficient. (See pp. 118-122 of Ultra-Fast ASP.NET by Richard Kiessig.)
There is chance your are running up against the maximum amount of memory that Application Pool is allowed to consume, which causes a restart of the Application Pool (which would account for the delay you are seeing in accessing the session). The amount of memory on the server doesn't impact the amount of memory ASP.NET can use, this is controlled in the machine.config in the memoryLimit property and in IIS 6.0 later in IIS itself using the "Maximum memory used" property. Beyond that, have you considered alternatives to each user using 5 MB of session memory? This will not scale well at all and can cause a lot of issues while under load. Might caching be a more effective solution? Do the searches take so long that you need to do this, could the SQL/Database Setup be optimized to speed up your queries?
When adding items using the default insert method Insert(key, value) and as such using the default behavior of HttpRuntime.Cache, when does the scavenging take place?
The limit on the application pool that hosts my application is 0 (unlimited)... but I see no way to find out how big the cache can get before scavenging occurs? The problem is I am caching a potentially very large amount of data but I have limited memory on the server and I want to avoid recycling of the app pool.
I am asking in the context of IIS6 and IIS7.
Thank you.
The short answer is that the cache begins to scavenge at Cache.EffectivePrivateBytesLimit.
The default limit is calculated using a number of factors, and generally should keep your process from excessive recycling. Ideally an application won't be affected by a recycle, but I realize there are many reasons why that isn't always possible. If needed, you should be able to set memory limits for the cache in coordination with the app pool recycle settings.
Thomas Marquardt from the asp.net team wrote a very interesting post explaining the details.
I'm seeing consistently high CPU usage for my ASP.NET web application (on the live production box only, naturally....!) and I'm trying to narrow down the cause - it's basically maxing out a quad core Xeon box and there's no way it should be able to do that!
The CPU usage of the web process is generally higher than that of the DB process - which rings alarm bells to me on its own (?).
However, using the standard profiling tools (dotTrace, Red Gate etc) only show you the time spent in individual methods (rather than actual CPU usage) - and ultimately still highlight methods that are DB-bound. While this might indicate opportunities for caching or better indexes, I don't see how that in itself would result in high CPU usage of the web application process?
Any suggestions or tips as to how I can narrow this down?
Thanks!
Some suggestions to try at the first place.
1.Deploy with Release Build
Check whether the deployed product is in release mode. By running in debug mode, lot of time is wasted loading the pdbs along with the assemblies.
2.Disable ViewState
Disable viewstate if its not required. ViewState is nothing but data stored in hidden fields to be persisted between requests. it increases the total payload of the page both when served and when requested. There is also an additional overhead incurred when serializing or deserializing view state data that is posted back to the server. Lastly, view state increases the memory allocations on the server.
3.Disable Session State:
If you are not going to use it disable Session State. By default it’s on. You can actually turn this off for specific pages or for the whole application.
There are some basic ASP.NET application performance monitoring, check these two MSDN articles
"Monitoring ASP.NET Application Performance" and Performance Counters for ASP.NET
Can you set up some unit tests to call various methods and see what their impact is on processor usage? Visual Studio has some testing tools built in if you're using Team System, but even if you're not, you could write a multithreaded tester to call particular functions hundreds of times.
If you'd like some pointers on how to do this, I can help you build some basic unit testing.
are you recording/reporting unhandled exception? If not do so and check if any of them correspond with your high CPD spikes you may have a stack overflow causing the spikes.
http://msdn.microsoft.com/en-us/library/ms998306.aspx
You could also look into recoding the time of each request by using a HttpModule and checking which requests are taking up the most time which may indicate the pages that are causing the issue.
As Pradeepno notes, the place to start with is really performance counters--they can give you a very good idea of what is consuming what part of the CPU.
The web app usage being higher than DB usage isn't entirely suprising. If you have decent db design, most web apps are barely going to cause a decently powered DB server to break a sweat.