My development team is using ASP.NET Core 3.1.20 with Angular 10.2. We are seeing a large amount of unmanaged memory being consumed and never released. We are simulating load via a simple locust script with 200 concurrent users. I initially speculated that this was logic error in our code that was never releasing memory but I am not starting to wonder if it is something to do with how Microsoft manages memory. I have tried setting the false setting as well. This does help with overall memory but I can still expose the same problem regardless over time. I have verified that my code is purely managed code via ILDASM and PEVerify.
Any help is greatly appreciated
Application Memory
Related
I have successfully deployed a series of solutions as .NET Core console apps. These run successfully under normal circumstances but often it appears the data load (using EntityFramework Core query with .ToList() to retain in memory for multiple accesses) seems to be blowing the process with no error but never at the same place. Does anyone know any established memory or process limits on .NET Core C# console apps to be aware of? I never exceed the CPU or Memory on the hosting machine. It tends to be that a list once accessed for the first time after .ToList() executed earlier in the code with a hefty query that fueled it will do an null object exception on the list.
This is .NET Core 2.0.0 SDK at this point.
If you know of published limits that would be excellent. I am going to try using MiniProfiler to do forensics as well. TIA.
The answer to this question was two-fold. First...changing to x64 versus AnyCPU increased some ceiling but secondly the .Include() method in EntityFramework Core should be used sparingly and profiled. This exploded the in-memory graph as well. So yes a simple question which two people downgraded my question for may have been a simple answer yet it was exacerbated with .Include() that is a helper method yet can hurt you if not properly profiled and used.
I am trying to profile a web application running on IIS in a Production environment (.Net framework 4.0, WebForms, SQLServer, Windows 2008 server) .
Several pages are repeatedly slow in Production, but we are unable to reproduce in Development.
We cannot install any IDE or similar tool in the Production environment.
Does anyone know of a DLL or a stand-alone exe that we could easily drop into the server, run for about an hour, and then quickly and easily remove?.... we are seeking one important aspect to profiling:
--> what is the amount of time spent for each CSharp method to run. <--
Thank you in advance.
Try this https://github.com/jitbit/cpu-analyzer command-line tool for profiling cpu load in production.
We forked this from the original Sam Saffron's cpu profiler, which is kinda abandoned now.
Disclaimer: I'm the maintainer of this project.
Perfview is the one of the best tool for this purpose, have used to find numerous production issues.
You can narrow down the issue using multiple approach, it can tell from network level to IIS to asp.net to your C# methods.It uses ETW events emitted by IiS,Asp.net and CLR to do this.
https://channel9.msdn.com/Series/PerfView-Tutorial/PerfView-Tutorial-7-Using-the-Event-Viewer-in-ASPNET-Scenarios
https://channel9.msdn.com/Series/PerfView-Tutorial
using the ThreadTime view you can narrow down to a particular method
Perfview always collect system wide data and you can also set a circular buffer .But best way to collect data is when the issue starts happening, You go to the server in question and start perfview trace and reproduce the issue.Then you can analyze this data later to find the performance bottleneck.
I am using Neo load to test the performance of my ASP.NET 2.0 application. The application works fine until the memory of the w3wp.exe process reaches about 800000K and then it starts throwing System.OutOfMemoryException. The ASP.NET application is hosted in Windows server 2003 SP2 machine and the machine has 4GB of RAM. How can i get to know the reason for this error.
The 800000K threshold is also mentioned (as 800MB) in the following article:
Troubleshooting System.OutOfMemoryExceptions in ASP.NET
When your worker process approaches 800MB in private bytes, your
chances of seeing an OOM condition begin to increase simply because
the chances of finding a large, contiguous piece of memory within the
2GB address space begin to decrease significantly.
The article then goes on to cover common scenarios that lead to this situation, such as the usage of large data sets, runaway session or application data storage, usage of debuggable assemblies, lots of exceptions thrown and even certain uses of regular expressions.
It ends with some pointers to resources to help troubleshooting Out of Memory exceptions, like the Debug Diagnostic Tool and information about garbage collection in .NET. But the most valuable resource may be this list of 21 blog posts from an ' escalation engineer' at Microsoft's ASP.NET developer support team:
MSDN Blogs > If broken it is, fix it you should > The 21 most popular blog posts
So right off the bat, not sure if this question is better suited for another StackExchange site.
I've got an ASP.NET MVC 3 web application running on Windows Server 2008 and IIS 7.5
Site runs fine initially, but i can see the memory usage gradually growing. After about 12 hours it's nearly out of memory and the site chokes.
I'm using a lot of caching, so i'm thinking this combined with some possibly memory leaks is the cause of the issue.
So my question - what's the best way (tools, for example) to monitor memory usage on a web server running ASP.NET MVC?
In the past i've used good old' perfmon and put the IIS counters on to measure these things.
It this still the best way, and if so, can someone recommend a good perfmon counter template for my scenario?
Perfmon's counters are still a good technique (and free!).
PAL (Performance Analysis of Logs), a free tool, has an ASP.NET perfmon counter template for general health (in addition to generating reports of counter log files based on thresholds).
Check out:
.NET Debugging Demos Lab 7: Memory Leak
.NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon
Tracking down managed memory leaks (how to find a GC leak)
Determine if your .NET Application has a Memory Leak
Commercial tools like MemProfiler, RedGate's memory profiling tool and JetBrains Profiler are all very good (and all have free trials).
There is a great presentation by Dan Farino, Chief Systems Architect at MySpace.com, showcasing a web-based stack dump tool that catalogues all threads running in a given process (what they're doing, how long they've been executing, etc.)
Their techniques are also summarized on highscalability.com:
PerfCollector.
Centralized
collection of performance data via
UDP. More reliable than Windows and
allows any client to connect and see
stats.
Web Based Stack Dump Tool.
Can right-click on a problem server
and get stack dump of the .Net
managed threads. Used to have to RDC
into system and attach a debugger and
1/2 later get an answer. Slow,
nonscalable, and tedious. Not just a
stack dump, gives a lot of context
about what the thread is doing.
Troubleshooting is easier because you
can see 90 threads are blocked on a
database so the database may be down.
Web Base Heap Dump Tool.
Dumps all
memory allocations. Very useful for
developers. Save hours of doing it by
hand. • Profiler. Traces a request
from start to finish and produces a
report. See URL, methods, status,
everything that will help you
identify a slow request. Looks at
lock contentions, are a lot of
exceptions being thrown, anything
that might be interesting. Very light
weight. It's running on one box in
every VIP (group of 100 servers) in
production. Samples 1 thread every 10
seconds. Always tracing in
background.
The question is: what tools are required to build a web-based stack dump tool for ASP.NET? For convenience, let's assume that an *.aspx hosted in the target AppDomain, able to output all managed call stacks in that process, is sufficient.
There are a few posts that cover the use of Mdbg (debugger for managed code written entirely in C#/IL that started shipping with CLR 2 SDK) and the mdbgcore assembly typically found in C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin:
http://dotnetdebug.net/2005/11/09/exceptiondbg-v01-debug-your-exceptions/
http://blogs.msdn.com/jmstall/archive/tags/MDbg/default.aspx
http://blogs.msdn.com/vijaysk/archive/2009/11/04/asp-net-debugger-extension-for-iis-7.aspx
Would a solution simply reference this assembly to produce the desired output? What impact would a "list all managed call-stacks" operation have on the running process that's servicing production traffic?
I believe the profiling API of .Net are the way to go.
Look at the SlimTune project on Google Code to have a live sample, with sources, that you can check how to adapt and improve to work in a Asp.NET scenario.
Regards
Massimo
With the profiling API of .Net you have to stop the server and it takes a lot CPU (but it gives you full control over all called methods).
I think the most “light way” solution is to doing this with MDbg, I put together a very small but useful little app called StackDump that does the following:
1) The debugger stops the application and generates a list of all CLR stacks running for the process.
2) The application is started again.
This operation is a quick operation and can (maybe) be executed on a running production server with unchanged production code.
It just 80 lines of .Net code to manage this. I have published the source code on Codeplex.