Recently we experienced crashing on application (ASP.Net) with below issue since issue is resolved but a question arises in my mind; can we capture history of memory used by application? I want to add a job which will capture application memory usage after each half an hour.
Memory gates checking failed because the free memory (373817344 bytes)
is less than 5% of total memory. As a result, the service will not be
available for incoming requests. To resolve this, either reduce the
load on the machine or adjust the value of
minFreeMemoryPercentageToActivateService on the
serviceHostingEnvironment config element.
Edit: Question arise from link
Related
I'm running a simple CRUD app built with ASP.NET Core and EF Core 3.1 in a docker swarm cluster on ubuntu. I'm only using managed code.
The container has a 10GB memory limit specified. I can inspect a running container and verify that this limit is actually set, I also see that DOTNET_RUNNING_IN_CONTAINER is set to true. When the app is started the memory consumption is about 700MB and it slowly builds up. Once it reaches 7GB (I see it in container generic stats) I start getting OutOfMemoryExceptions and it stays at this level for days. So the first question is
Why doesn't it go up to 10 GB?
Anyway I expect memory leaks so I have a dotnet-gcdump tool installed in this same container so I go ahead and collect the dump for future analysis with dotnet-gcdump collect. Once I execute this command I see the memory consumption of the running container drops from 7GB to 3GB and stays at this level. The resulting .gcdump file itself size though is only ~200MB with nothing suspicious in it. So next questions are
How does the collection of a dump reduce memory consumption? I'd assume it's doing GC with LOH compaction but it doesn't mention it in the docs.
Why isn't this memory freed automatically if the tool is able to do it?
Why is a resulting dump only 200 MB in size?
As the gcdump documentations explains: "GC dumps are created by triggering a GC in the target process, turning on special events, and regenerating the graph of object roots from the event stream".
Thus, it directly answers your question 2 - it triggers full GC, which may or may not be compacting, but it collects gen2 for sure. It also answers question 4 - it is not a "memory dump" but a special kind of diagnostics data about the objects graph (depndencies and typenames), without the data itself.
And regards to the questions 1 and 3 - it is an example of the GC being "not aggressive" enough. It is kind of the "living on the edge" problem when the process almost meets the containers limits and GC sometimes is not able to interpret it. In other words, it thinks it has enough space, but it doesn't. Please, be warned that this is a super-oversimplification. In such a case full GCs may not happen or happen too late. I would confirm that by observing the process by the dotnet-trace with gc-collect profile.
As a solution, consider setting the limit manually, by using GCHeapHardLimit, to some clearly smaller value like 8GB.
Good day,
Our production servers (w2k2012 with iis8.x) got a memory upgrade recently from 4GB to 8GB. The reason was that application pools (20-30, one per mvc/.net application were claiming too much memory (reaching the 5% limit).
We used the private bytes claimed by the worker processes to find out how much memory was reserved per pool. This was about 300-400MB per pool.
After the memory upgrade the same pools now claimed 600-800GB. We verified on our development machines with even more memory and there the claim got even higher (up to 1GB).
So now our in-house hoster things adding more memory does not make sense.
My questions:
- Why are the pools taking memory relative to the physical memory (so it appears)? I could not find information about this, only that the maximum for private bytes is 60% of physical memory, not the actual claim would increase too for the same pool if more physical memory is added.
- How can we influence this? Is setting private byte limit per pool the solution?
We hoped to be able to add more customers to our server by adding more memory.
Best regards, Rob
You may be seeing an increase in Standy memory which is actually available. An interesting link with some related details is Where has all my Physical RAM Gone?
The related part is
Standby: Pages of physical ram not actively being used. These are still left in physical ram but will be repurposed first by the memory
manager (either returned to the active list or zeroed out and reused)
if something needs physical ram for active pages. Standby pages are
essentially cache – it’s better to have infrequently used data kept in
RAM “just in case” than pushing it out to disk when the memory isn’t
needed for anything else.
In general it is the operating taking as much of it as it can, just in case it needs it. This happens on Windows 7 as well in a similar manner. A computer with a couple of browser tabs and Outlook open is using 5 GB of memory, but will free up unused portions as needed.
On a 2012R2 development server, I see something similar
As you mentioned, you can also limit the memory usage of an App Pool within IIS itself. It sounds like you may know how to do this but in case it helps you or others:
You can limit memory taken by an App Pool
Open Internet Information Services (IIS) Manager
Navigate to the Application Pool section
Click once on the App Pool you'd like to limit
On the right side under the Actions pane, click Recycling...
Within the window that appears, you can choose your preferred options under Memory Based Maximums
In general I'm suspecting you'll be okay to add more customers to your server, unless you have some heavy hitting web applications currently running. You may consider setting a reasonable limit for the memory for a few App Pools and see if it has any effect on the application in terms of performance and test what works.
This will give you a better idea of how many additional customers you could host reasonably.
You may also want to set a limit to the App Pools in terms of memory to prevent customer sites from taking resources away from other customer sites. This of course completely depends on your environment and what type of loads the server handles at certain times.
I have a small WCF service which is executed on an XP box with 256 megs of RAM running in VM.
When I make a request (with a request size of approximately 5mbs) to that service I always get the following message in the event log:
aspnet_wp.exe was recycled because memory consumption exceeded the 153 MB (60 percent of available RAM).
and the call fails with error 500.
I've tried to increase memory limit to 95% but it still takes up all the available memory and fails in the same manner.
It looks like something is wrong with my app (I do not reuse byte[] buffers and maybe something else) but I cannot find root cause of such memory overuse.
Profiling showed that all CLR objects that I have in memory together do not take up that much space.
Doing a dump analysis with windbg showed same situation - nothing that big in object heap.
How can I find out what is contributing to such memory overuse?
Is there any way to make a dump right before process is recycled (during peak mem usage)?
Tess Ferrandez's blog "If broken it is, fix it you should" has lots of hints, tips and recommendations for sorting out exactly this sort of problem.
Of particular use to you would be Lab 3: Memory, where she walks you through working out what has caused all the memory on your machine to disappear.
Could be a lot of things, hard to diagnose this one. Have you watched perfmon to see if the memory usage does peak on aspnet process or on the server itself? 256MB is pretty low, but it should still be able to handle it. Do you have a SWAP file on this machine? AT what point do you take the memory dump? Have you stepped though the code, and does it work on other machines? Perhaps it is getting stuck in a loop and leaking memory until it crashes?
I'm looking for a way for the application itself to monitor the amount of memory it is using, so I can record it in a log file every hour or so and keep an eye on the applications usage.
Its all hosted so we can make changes to the system to see what is going on so the solution will have to be from within the application code.
We may in future use the memory information to affect the caching policies.
Hmm, how detailed information do you need? If you just want the memory usage you can ask the GC. It knows. ;)
long bytes = GC.GetTotalMemory(false); // use 'false' to not wait for next collect
The variable 'bytes' will contain the number of bytes currently allocated in managed memory. I'm not sure whether the managed memory entails the entire process or just the AppDomain. You'll have to test this by running several AppDomains in one process and see if managed memory allocation is measured cross AppDomains. If they don't, then you can use this to measure total memory usage in an ASP.NET application.
If you want more specific information there's a diagnostics API for the CLR which you could interface with. There's also plenty of memory profilers out there, but if they'll work within an ASP.NET application I cannot say.
As an alternative, if you want more detailed information, you can read the performance counters using the System.Diagnostics.PerformanceCounter class. Here are some of the counters that you can plug into:
Request Bytes Out Total
Request Bytes In Total
Request Wait Time
Requests Executing
Requests/Sec
Errors Total
Environment:
Windows 2003 Server (32 bit); IIS6, ASP.NET 2.0 (3.5); 4Gb Ram; 1 Worker Process
We have a situation where we have a very large System.XmlDocument is being loaded into memory, and then it heads into a complied XSL transform.
What is happening is when a web request comes in the server is sitting in an idle state with 2500Mb of available system memory.
As the XML DOM is populated, the available memory drops approx 500Mb at which point we get a System.OutOfMemoryException event. At this point the system should theoretically still have 2000Mb of available memory available to service the request (according to Perfmon).
The related questions I have are:
1) At what level in the stack is this out of memory limitation being met? OS? IIS? ASP.NET? worker process? Is this a per individual web request limit?
2) Is this limit configurable somewhere?
3) Why can’t this web request access the full available system memory?
1) I would guess at the worker process but this should be configurable within IIS to the limit of memory that a worker process can use. Another factor is what level of bits does your software use, e.g. 32 bit has a physical limit of 4 GB since this is the total address space.
2) Probably but don't forget that memory fragmentation may play a role in getting to out of memory faster than you think, e.g. if there is a memory request for a contiguous 1000 Mb piece of memory then this may not necessarily be found in the current memory.
3) Have you examined dump data to see what is in the memory when the exception gets thrown? If not, there are ways to get a snapshot of the memory to see what it looks like as this may give you more clues about what is going on.
You are running in a process. A process can only access 2 gigs of memory. This task is sharing memory with everything else running in this process, so this bit of code does not get the full 2 gig -- even if it is available.
There is a 3 gig switch on the os as well. I believe it is a registry setting. But you will have to search MSDN to find that info.
But realistically, you need to do this another way. Possibly by switching to a SAX style xml parser.
I'm sure there are some bright heads here that can answer your specific questions, but have you asked yourself if there is another way to do what you want? I specifically mean that you probably do not want to process a very large XML document, but you probably more specifically want to return something back to the client. Could you rewrite the code to avoid this XML document altogether, or perhaps not load it all into memory at the same time, and still produce the same end-result?
1) Dunno. Check your logs.
2) IIS limits memory divvied out to websites/application pools. Check your settings.
3) Servers are all about uptime; if an single app hogs all the resources everybody else suffers. Thats why enterprise apps like IIS limit memory to prevent runaways from taking down the entire server.