I have an asp.net app hosted at a (for now, unnamed) hosting company. This app uses a small MS SQL database.
During the past couple of months, my complete website (whether it's the app page or plain static html page) was not accessible twice due to error 503. They just shut down the website without notifying me. However I monitor the website every few hours and called them when I found the problem. Then they restarted the application pool and everything worked again.
They told me that my website/app reached their "Worker Process Limit" of "100MB" and that's the reason for their shutting it down.
This seems to be the only hosting company that advertises their Worker Process Limit with their hosting plan so I don't have a basis of comparison with other companies's plans.
So, my questions are:
1.Could someone explain what the Worker Process Limit is in Windows server environment? And is 100MB considered small or adequate?
2.How can I handle or avoid reaching this limit within my app? Is this possible or is it just the number of visitors is too high? I average only a few hundred visitors a day.
Thanks.
From my understanding, the WPL is based off of how much resources your application utilizes. Naturally, this could be a result of how many users visits your web application (or more so, concurrently) but it extends a bit beyond that.
I believe what this is saying, is that your application is memory hungry. Is that a bad thing? No, but when you only have 100MB at your disposal it is.
I would say your first step would be to get that web application local and see the actual resource utilization. I know this is a very open-ended answer, but you'll have to do the resource tuning to slim down your memory consumption. There's a chance that this is just from a bug (say, an array of objects that are not getting disposed of properly).
Related
We have a problem on our website, seemingly at random (every day or so, up to once every 7-10 days) the website will become unresponsive.
We have two web servers on Azure, and we use Redis.
I've managed to run DotNetMemory and caught it when it crashes, and what I observe is under Event handlers leak two items seem to increase in count into the thousands before the website stops working. Those two items are CaliEventHandlerDelegateProxy and ArglessEventHandlerProxy. Once the site crashes, we get lots of Redis exceptions that it can't connect to the Redis server. According to Azure Portal, our Redis server load never goes above 10% in peak times and we're following all best practises.
I've spent a long time going through our website ensuring that there are no obvious memory leaks, and have patched a few cases that went under the radar. Anecdotally, these seem to of improved the website stability a little. Things we've checked:
All iDisposable objects are now wrapped in using blocks (we did this strictly before but we did find a few not disposed properly)
Event handlers are unsubscribed - there are very few in our code base
We use WebUserControls pretty heavily. Each one had the current master page passed in as a parameter. We've removed the dependency on this as we thought it could cause GC to not collect the page perhaps
Our latest issue is that when the web server runs fine, but then we run DotNetMemory and attach it to the w3wp.exe process it causes the CaliEventHandlerDelegateProxy and ArglessEventHandlerProxy event leaks to increase rapidly until the site crashes! So the crash is reproducible just by running DotNetMemory. Here is a screenshot of what we saw:
I'm at a loss now, I believe I've exhausted all possibilities of memory leaks in our code base, and our "solution" is to have the app pools recycle every several hours to be on the safe side.
We've even tried upgrading Redis to the Premium tier, and even upgraded all drives on the webservers to SSDs to see if it helps things which it doesn't appear to.
Can anyone shed any light on what might be causing these issues?
All iDisposable objects are now wrapped in using blocks (we did this
strictly before but we did find a few not disposed properly)
We can't say a lot about crash without any information about it, but I have some speculations about it.
I see 10 000 (!) not disposed objects handled by finalization queue. Let start with them, find all of them and add Dispose call in your app.
Also I would recommend to check how many system handles utilized by your application. There is an OS limit on number of handles and if they are exceeded no more file handles, network sockets, etc can be created. I recommend it especially since the number of not disposed objects.
Also if you have a timeout on accessing Redis get performance profiler and look why so. I recommend to get JetBrains dotTrace and use TIMELINE mode to get a profile of your app, it will show thread sleeping, threads contention and many many more information what will help you to find a problem root. You can use command line tool to obtain profile data, in order not to install GUI application on the server side.
it causes the CaliEventHandlerDelegateProxy and
ArglessEventHandlerProxy event leaks to increase rapidly
dotMemory doesn't change your application code and doesn't allocate any managed objects in profiled process. Microsoft Profiling API injects a dll (written in c++) into the profiling process, it's a part of dotMemory, named Profilng Core, playing the role of the "server" (where standalone dotMemory written in C# is a client). Profiling Core doing some work with gathered data before sending it to the client side, it requires some memory, which allocated, of course, in the address space of the profiling process but it doesn't affect managed memory.
Memory profiling may affect performance of your application. For example, profiling API disables concurrent GC when application is under profiling or memory allocation data collecting can significantly slow down your application.
Why do you thing that CaliEventHandlerDelegateProxy and ArglessEventHandlerProxy are allocated only under dotMemory profiling? Could you please describe how you explored this?
Event handlers are unsubscribed - there are very few in our code base
dotMemory reports an event handler as a leak means there is only one reference to it - from event source at there is no possibility to unsubscribe from this event. Check all these leaks, find yours at look at the code how it is happened. Anyway, there are only 110.3 KB retained by these objects, why do you decide your site crashed because of them?
I'm at a loss now, I believe I've exhausted all possibilities of memory leaks in our code base
Take several snapshots in a period of time when memory consumption is growing, open full comparison of some of these snapshots and look at all survived objects which should not survive and find why they survived. This is the only way to prove that your app doesn't have memory leak, looking the code doesn't prove it, sorry.
Hope if you perform all the activities I recommend you to do (performance profiling, full snapshots and snapshots comparison investigation, not only inspections view, checking why there are huge amount of not disposed objects) you will find and fix the root problem.
I got website that running on IIS7 that response very slow during weekend night time, after restarting the application pool the website would response fast for certain of time. As time goes (maybe few hours) the website response become slow again.
I open the resource monitor at task manager, and look for the process for the website, the Average CPU is like 6-8, what makes me curious is the Receive B/s go up to 0.8-1MB while the Sent B/s only have 200B, I don't know if that is relevant.
So I was hoping to get an insight of what is the problem here.
Your question is a bit vague. For starters, are we talking a web app, connected to a database? If so, what type of database? There could be any number of causes for this.
But from the sounds of it, I would first start with any database queries you have running. I had a similar issue to yours with a web app, which turned out to be the result of an inefficient SQL Stored Procedure. The site would slow down and grind almost to a halt, but would spring back to life after the app pool was restarted. Rewriting the main Sproc solved the problem.
I have a number of ASP.NET (4.0) web applications that appear to leak (a small amount) of memory during each request. It is such a small amount, that for most use-cases, it will not grow to become a problem for weeks or even months at a time. I generally try to be good with closing any connections managed by the application, avoiding state-variables (or instance variables for my singleton), etc.
My question is this - is this normal behavior for ASP.NET applications? I had turned off the default (IIS 7) behavior of recycling the app pool after 20 minutes of being idle. I do this since the application takes a few minutes to build its internal cache, and I want to avoid negatively impacting the user experience (and having them wait for the application to start when they issue the request).
I am aware this could be mitigated by serializing the cache or speeding up the cache generation process but my question has more to do with the principle of it: I personally consider relying on the IIS auto-recycle feature as a bandaid approach. Am I wrong? am I just not seeing the garbage collector at work because the application's memory usage is not high enough compared to the amount of available memory? or should I dig deeper into the memory issues?
Any insight would be appreciated.
Unfortunately it is normal, though mostly dues to shoddy application writers than anything else.
IIS by default configures newly created application pools to recycle every 1740 minutes for this reason.
As you said, this is a band-aid. A well written application that clears up all of its resources (including dangling event handlers), shouldn't leak at all.
See this blog post about the subject.
We have a web application running which having around 100 users logged in, All clients are connected to server using websync. I was having requirement for keeping the session always live, so I am regenerating session when it is about to expire.
But after 3 or 4 days, I found cpu reached to 100% and application locked, then we need to restart the server to make it working.
Thanks for providing solutions in advance.
Thanks
Why don't you just extend the session duration to be extremely long instead of regenerating it?
Have you run a profiler against the server when it reaches 100% cpu? This should effectively tell you which methods/classes are being run in how many different threads. With this information you can figure out why your application is running those methods/classes across what I'm guessing is a lot of threads.
We've got plenty of clients using WebSync with tens of thousands of concurrent connections (and our On-Demand cluster sits at multiple thousands of users nonstop every day as well), so if you're seeing a CPU lockup, more than likely you've got a threading issue in your code, probably in one of your events (assuming it's related to the WebSync code at all).
Don't forget that the WebSync events are all static, so if you're using shared resources, you'll need to manage them accordingly (i.e., you have to count on the fact that they're multi-threaded). All the WebSync methods themselves are threadsafe, but if you've got stuff in your own events, you'll need to manage that yourself.
Feel free to chat with us directly though, as #Anton suggested!
As we deal with before, Windows update maybe reason for 100% CPU.
Greetings!
I have an ASP.NET app that scrapes data from a handful of external pages, parses the relevant bits and displays them in a table. Total data retrieved is 3-4MB and the resulting page is about 1MB. I am using synchronous WebRequest GetResponse for the retrieval, but the same problem existed using an asynchronous BeginGetResponse/EndGetResponse process.
There is no database access, no session storage, no caching, but an in-memory list of about 100 objects (total 1MB of data), plus a good amount of AJAX (AjaxControlToolkit). This issue appears on the very first run of the app, even if I have restarted IIS.
The issue:
When I run the app on my dev computer, the maximum commit charge is about 1.5GB. The biggest user, measured by Task Manager's VM Size, is WebDev.WebServer.exe (600MB). The app runs perfectly.
When I run it on my rent-a-server (IIS 7.5, 1GB RAM), the maximum commit charge is over 3.8GB. The biggest user is w3wp.exe at 2.7GB. IIS grinds to a halt and spits out a timed-out error page.
Given my limited server budget and the hope of having multiple simultaneous users, I'm kind of in a panic.
Is this normal? If I bump the server RAM up to 4GB, will that be enough?
Will multiple users require even more memory?
Could the culprit be AJAX or the list of objects?
Thanks for any insight you can provide.
Did you try running this in your dev environment under IIS 7.5?
Make sure debug="false", not "true" in your web.config
I think you need to dig out some debugging tools and capture a worker process dump of your production server, you won't be able to properly diagnose this issue with just PerfMon and Task Manager.
I posted this answer on Stack Overflow a while back which should get you started:
CLR out Of Memory Exceptions
Well, after some hard work, I have identified the culprit: our old nemesis the Endless Loop. Of course, if the development environment had thrown an exception, I would have caught and excised the problem - but it didn't.
I would say the lesson learned here is to understand that different versions of IIS and DevEnv respond to errors differently and that we must test the app in the same configuration in which it will be deployed.
Thanks everyone for your feedback.