I have one website on my server, and my IIS Worker Process is using 4GB RAM consistently. What should I be checking?
c:\windows\system32\inetsrv\w3wp.exe
I would check the CLR Tuning Section in the document Gulzar mentioned.
As the other posters pointed out, any object that implements IDispose should have Dispose() called on it when it's finished with, preferably using the using construct.
Fire up perfmon.exe and add these counters:
Process\Private Bytes
.NET CLR Memory# Bytes in all Heaps
Process\Working Set
.NET CLR Memory\Large Object Heap size
An increase in Private Bytes while the
number of Bytes in all Heaps counter remains the same indicates unmanaged
memory consumption.
An increase in
both counters indicates managed memory
consumption
check the section on troubleshooting memory bottlenecks in Tuning .NET Application Performance
Create a mini-dump of the w3wp process and use WinDbg to see what objects are in memory. This is what the IIS support team at Microsoft does whenever they get questions like this.
If you have access to the source code, you may want to check that any objects that implement IDisposable are being referenced inside using statements or being properly disposed of when you are done with them.
Using is a C# construct, but the basic idea is that you are freeing up resources when you are done.
Another thing to check on is large objects getting put in the "in process" session state or cache.
More details would definitely help. How many applications are running inside the application pool? Are there ASP.NET applications in the pool?
If you're running ASP.NET, take a good look at what you're storing in the session and cache variables. Use PerfMon to check how many Generation 0, 1 and 2 collections are occurring. Be wary of storing UI elements in the session state or cache since this will prevent the entire page instance and all of the page instance's children from being collected as well. Finally, check to see if you're doing lots of string concatenation. This can cause lots of object instantiations since .NET strings are immutable. Look into using StringBuilder instead.
As other people noted common cause of this problem is resource leaking, also there is a known issue with win2k3 server and IIS6 KB916984
Related
I get this message too often and think that it is not usual, how i can fix this issue?
A worker process with process id of '8052' serving application pool 'mywebsite.com' has requested a recycle because it reached its private
bytes memory limit.
Hi this is not an expire because of a timeout, it's an application pool recycle due to memory limit. Usually very bad.
if you have full control of your ASP.NET application I suggest you profile it carefully and detect where you are allocating and not releasing / disposing a lot of memory.
It might help to understand what are the common use cases users are going through more often and start investigating those.
Are you loading a lot of byte[] in memory for file downloads?
are yoou keeping too many references in the Session?
once we had a similar problem and we found out only after several weeks that we were storing a reference to pages in the Session so all visited pages were kept in memory all the times. Not easy to find but once we corrected this the problem was solved.
You probably have a memory leak. There are many possible causes, possibly session or request instances registered with static events. Operations that are trying to load too much data into memory, session timeouts set too long etc.
You need to do some profiling to identify the cause.
Of course its also possible you just don't enough memory for the number of users and need to scale, but it's highly unlikely unless you have a lot of traffic.
Maybe your application use a lot of memory? Regarding to this post you have reached allowed memory size for application and IIS restarts the application pool.
A few months ago I was interviewing for a job inside the company I am currently in, I dont have a strong web development background, but one of the questions he posed to me was how could you improve this block of code.
I dont remember the code block perfectly but to sum it up it was a web hit counter, and he used lock on the hitcounter.
lock(HitCounter)
{
// Bla...
}
However after some discussion he said, lock is good but never use it in web applications!
What is the basis behind his statement? Why shouldnt I use lock in web applications?
There is no special reason why locks should not be used in web applications. However, they should be used carefully as they are a mechanism to serialize multi-threaded access which can cause blocking if lock blocks are contended. This is not just a concern for web applications though.
What is always worth remembering is that on modern hardware an uncontended lock takes 20 nanoseconds to flip. With this in mind, the usual practice of trying to make code inside of lock blocks as minimal as possible should be followed. If you have minimal code within a block, the overhead is quite small and potential for contention low.
To say that locks should never be used is a bit of a blanket statement really. It really depends on what your requirements are e.g. a thread-safe in-memory cache to be shared between requests will potentially result in less request blocking than on-demand fetching from a database.
Finally, BCL and ASP.Net Framework types certainly use locks internally, so you're indirectly using them anyway.
The application domain might be recycled.
This might result in the old appdomain still finishing serving some requests and the new appdomain also serving new requests.
Static variables are not shared between them, so locking on a static global would not grant exclusivity in this case.
First of all, you never want to lock an object that you actually use in any application. You want to create a lock object and lock that:
private readonly object _hitCounterLock = new object();
lock(_hitCounterLock)
{
//blah
}
As for the web portion of the question, when you lock you block every thread that attempts to access the object (which for the web could be hundreds or thousands of users). They will all be waiting until each thread ahead of them unlocks.
Late :), but for future readers of this, an additional point:
If the application is run on a web farm, the ASP's running on multiple machines will not share the lock object
So this can only work if
1. No web farm has to be supported AND 2. ASP is configured (non-default) NOT to use parallel instances during recycle until old requests are served (as mentioned by Andras above)
This code will create a bottleneck for your application since all incoming request will have to wait at this point before the previous went out of the lock.
lock is only intended to be used for multithreaded applications where multiple threads require access to the same shared variable, thus a lock is exclusively acquired by the requesting thread and all pending threads will block and wait until the lock is released.
in web applications, user requests are isolated so there is no need for locking by default
Couple reasons...
If you're trying to lock a database read/write operation, there's a really high risk of a race condition happening anyway because the database isn't owned by the process doing the lock, so it could be read from/written to by another process -- perhaps even a hypothetical future version of IIS that runs multiple processes per application.
Locks are typically used in client applications for non-UI threads, i.e. background/worker threads. Web applications don't have as much of a use for multithreaded processing unless you're trying to take advantage of multiple cores (in which case locks on request-associated objects would be acceptable), because each request can be assumed to run on its own thread, and the server can't respond until it's processed the entire output (or at least a sequential chunk) anyway.
I have an asp.net app which uses legacy COM interop library. Works fine until memory reaches somewhere around 500Mb and then it is no longer able to create new COM objects (get various exceptions, e.g. Creating an instance of the COM component with CLSID {FFFF-FFFF-FFFF-FFF-FFFFFF} from the IClassFactory failed due to the following error: 80070008.). It almost looks like it is hitting some kind of memory limit, but what is it? Can it be changed?
Solved! Turns out the object was creating a Window handle and we were hitting the 10K Window handles limit (except it was happening at 2K instances for some reason when inside IIS)
Solved! Turns out the object was creating a Window handle and we were hitting the 10K Window handles limit (except it was happening at 2K instances for some reason when inside IIS)
What OS, and is it 32-bit or 64-bit? What are you using to determine memory usage?
When you say you're explicitly releasing the objects, do you mean you're using Marshal.ReleaseComObject()?
I'm assuming you have AspCompat=true in your <%# Page > tag... wouldn't expect it to run at all if you didn't.
Can you give us some details on your COM object; what does it do, and can you post some code where you're calling it, including COM object signatures? How much memory would you expect a single object to take?
My first suspect, based only on the information that I've read so far, is that 500Mb is not truly the total memory in use, and/or that you're having a memory fragmentation issue. I've seen this occur with IIS processes when less than half of the memory is in use, and the errors tend to be random, depending on what object is being created at the time. BTW, 80070008 is 'not enough storage space'.
Process limits are 2GB on a 32-bit machine, of course, but even if a process isn't using the full 2GB, if there's not a contiguous block of memory of the size needed when creating an object, you'll get an out-of-memory error when you try to allocate. Lots of concurrent users implies lots of COM objects (and other objects) being allocated and released in a short period of time... which points to fragmentation as a suspect.
Coming up with an attack plan requires more info about the COM object and how it's being used.
Use a command pattern for queueing and executing the com interop in an asynchronous thread. This can free up the number of threads being used by iis, and allow you to control the number of calls/instances of the com app.
You may think about object pooling rather than creating every time a new object.
Under windows server 2008 64bit, IIS 7.0 and .NET 4.0 if an ASP.NET application (using ASP.NET thread pool, synchronous request processing) is long running (> 30 minutes). Web application has no page and main purpose is reading huge files ( > 1 GB) in chunks (~5 MB) and transfer them to the clients. Code:
while (reading)
{
Response.OutputStream.Write(buffer, 0, buffer.Length);
Response.Flush();
}
Single producer - single consumer pattern implemented so for each request there are two threads. I don't use task library here but please let me know if it has advantage over traditional thread creation in this scenario. HTTP Handler (.ashx) is used instead of a (.aspx) page. Under stress test CPU utilization is not a problem but with a single worker process, after 210 concurrent clients, new connections encounter time-out. This is solved by web gardening since I don't use session state. I'm not sure if there's any big issue I've missed but please let me know what other considerations should be taken in your opinion ?
for example maybe IIS closes long running TCP connections due to a "connection timeout" since normal ASP.NET pages are processed in less than 5 minutes, so I should increase the value.
I appreciate your Ideas.
Personally, I would be looking at a different mechanism for this type of processing. HTTP Requests/Web Applications are NOT designed for this type of thing, and stability is going to be VERY hard, you have a number of risks that could cause you major issues as you are working with this type of model.
I would move that processing off to a backend process, so that you are OUTSIDE of the asp.net runtime, that way you have more control over start/shutdown, etc.
First, Never. NEVER. NEVER! do any processing that takes more than a few seconds in a thread pool thread. There are a limited number of them, and they're used by the system for many things. This is asking for trouble.
Second, while the handler is a good idea, you're a little vague on what you mean by "generate on the fly" Do you mean you are encrypting a file on the fly and this encryption can take 30 minutes? Or do you mean you're pulling data from a database and assembling a file? Or that the download takes 30 minutes to download?
Edit:
As I said, don't use a thread pool for anything long running. Create your own thread, or if you're using .NET 4 use a Task and specify it as long running.
Long running processes should not be implemented this way. Pass this off to a service that you set up.
IF you do want to have a page hang for a client, consider interfacing from AJAX to something that does not block on IO threads - like node.js.
Push notifications to many clients is not something ASP.NET can handle due to thread usage, hence my node.js. If your load is low, you have other options.
Use Web-Gardening for more stability of your application.
Turn-off caching since you don't have aspx pages
It's hard to advise more without performance analysis. You the VS built-in and find the bottlenecks.
The Web 1.0 way of dealing with long running processes is to spawn them off on the server and return immediately. Have the spawned off service update a database with progress and pages on the site can query for progress.
The most common usage of this technique is getting a package delivery. You can't hold the HTTP connection open until my package shows up, so it just gives you a way to query for progress. The background process deals with orchestrating all of the steps it takes for getting the item, wrapping it up, getting it onto a UPS truck, etc. All along the way, each step is recorded in the database. Conceptually, it's the same.
Edit based on Question Edit: Just return a result page immediately, and generate the binary on the server in a spawned thread or process. Use Ajax to check to see if the file is ready and when it is, provide a link to it.
Lets say that you are using a shared hosting plan and your application stores lots of objects
in the application state.
If they start taking too much memory does this mean that the server will just remove them?
If not what will happen then? What happens when the server has no memory left? Can you still store objects into the application or session state?
I am asking this because i am planning on developing a big site that will rely on the application state, and it will be crucial that the objects stored there don't get destroyed.
What i am afraid of is that at a certain point i might have too many objects in the application state and they might get removed to free up memory.
There are three different thresholds:
The total size of your app exceeds the maximum process size on your machine (really only applicable with an x86 OS). In that case, you'll start getting out of memory errors at first, generally followed very quickly by a process crash.
Your process, along with everything else running on the machine, no longer fits in physical memory. In that case, the machine will start to page, generally resulting in extremely poor performance.
Your process exceeds the memory limit imposed by IIS on itself, via IIS Manager. In that case, the process will be killed and restarted, as with a regular AppPool recycle.
With the Application object, entries are not automatically removed if you approach any of the above thresholds. With the Cache object, they can be removed, depending on the priority you assign.
As others have said, over-using the Application object isn't generally a good idea, because it's not scalable. If you were ever to add a second load-balanced server, keeping the info in sync from one server to another becomes very challenging, among other things.
What happens when any application takes up too much memory on a computer?
It causes the server to run everything really slowly. Even the other sites that share the computer.
It's not a good idea to store that much in application state. Use your config file and/or the database.
It sounds like you have a memory leak, the process keeps leaking memory until it crushes with an out-of-memory condition and is then automatically restarted by the server.
1.5GB is about the maximum amount of memory a 32 bit process can allocate before running out of address space.
Somethings to look for:
Do you do your own caching? when are
items removed from the cache?
Is there somewhere data is added to a
collection every once in a while but
never removed?
Do you call Dispose on every object
that implements IDisposable?
Do you access any non-managed code at
all (COM objects or using DllImport)
or allocate non-managed memory (using
the Marshal class for example)?
anything that is allocated there is
never freed by the garbage collector,
you have to free it yourself.
Do you use 3rd party libraries or any
code from 3rd parties? it can have
any of the problems in the list too.
If you use the Cache object instead of the Application object, you can minimize problems of running out of memory. If the memory utilization of the ASP.Net worker process approaches the point at which the process will be bounced automatically (the recycle limit), the memory in Cache will be scavenged. Items that haven't been used for a while are removed first, potentially preventing the process from recycling. If the data is stored in Application, ASP.Net can do nothing to prevent the process from recycling, and all app state will be lost.
However, you do need to have a way of repopulating the Cache object. You could do that by persisting the cached data in a database, as others have proposed.
Here's a short article with a good code example for handling Cache.
And here's a video of how to use Cache.
Anything stored in application state should be refreshable, and needs to be saved in current status in files or database. If nothing else happens, IIS restarts worker processes at least once a day, so nothing in application state will be there forever.
If you do run out of memory, you'll probably get an out of memory exception. You can also monitor memory usage, but in a shared host environment, that may not be enough information to avoid problems. And you may get the worker process recycled as an "involuntary" fix.
When you say that it's crucial that objects stored in application state don't get destroyed, it sounds like you're setting yourself up for trouble.
I think you should use session instead of the application sate and stored that session into sql server database. So once your application user end its session that will release your memory.
If you want more specific answer then please provide the more information about your application.