There is a great presentation by Dan Farino, Chief Systems Architect at MySpace.com, showcasing a web-based stack dump tool that catalogues all threads running in a given process (what they're doing, how long they've been executing, etc.)
Their techniques are also summarized on highscalability.com:
PerfCollector.
Centralized
collection of performance data via
UDP. More reliable than Windows and
allows any client to connect and see
stats.
Web Based Stack Dump Tool.
Can right-click on a problem server
and get stack dump of the .Net
managed threads. Used to have to RDC
into system and attach a debugger and
1/2 later get an answer. Slow,
nonscalable, and tedious. Not just a
stack dump, gives a lot of context
about what the thread is doing.
Troubleshooting is easier because you
can see 90 threads are blocked on a
database so the database may be down.
Web Base Heap Dump Tool.
Dumps all
memory allocations. Very useful for
developers. Save hours of doing it by
hand. • Profiler. Traces a request
from start to finish and produces a
report. See URL, methods, status,
everything that will help you
identify a slow request. Looks at
lock contentions, are a lot of
exceptions being thrown, anything
that might be interesting. Very light
weight. It's running on one box in
every VIP (group of 100 servers) in
production. Samples 1 thread every 10
seconds. Always tracing in
background.
The question is: what tools are required to build a web-based stack dump tool for ASP.NET? For convenience, let's assume that an *.aspx hosted in the target AppDomain, able to output all managed call stacks in that process, is sufficient.
There are a few posts that cover the use of Mdbg (debugger for managed code written entirely in C#/IL that started shipping with CLR 2 SDK) and the mdbgcore assembly typically found in C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin:
http://dotnetdebug.net/2005/11/09/exceptiondbg-v01-debug-your-exceptions/
http://blogs.msdn.com/jmstall/archive/tags/MDbg/default.aspx
http://blogs.msdn.com/vijaysk/archive/2009/11/04/asp-net-debugger-extension-for-iis-7.aspx
Would a solution simply reference this assembly to produce the desired output? What impact would a "list all managed call-stacks" operation have on the running process that's servicing production traffic?
I believe the profiling API of .Net are the way to go.
Look at the SlimTune project on Google Code to have a live sample, with sources, that you can check how to adapt and improve to work in a Asp.NET scenario.
Regards
Massimo
With the profiling API of .Net you have to stop the server and it takes a lot CPU (but it gives you full control over all called methods).
I think the most “light way” solution is to doing this with MDbg, I put together a very small but useful little app called StackDump that does the following:
1) The debugger stops the application and generates a list of all CLR stacks running for the process.
2) The application is started again.
This operation is a quick operation and can (maybe) be executed on a running production server with unchanged production code.
It just 80 lines of .Net code to manage this. I have published the source code on Codeplex.
Related
I am trying to profile a web application running on IIS in a Production environment (.Net framework 4.0, WebForms, SQLServer, Windows 2008 server) .
Several pages are repeatedly slow in Production, but we are unable to reproduce in Development.
We cannot install any IDE or similar tool in the Production environment.
Does anyone know of a DLL or a stand-alone exe that we could easily drop into the server, run for about an hour, and then quickly and easily remove?.... we are seeking one important aspect to profiling:
--> what is the amount of time spent for each CSharp method to run. <--
Thank you in advance.
Try this https://github.com/jitbit/cpu-analyzer command-line tool for profiling cpu load in production.
We forked this from the original Sam Saffron's cpu profiler, which is kinda abandoned now.
Disclaimer: I'm the maintainer of this project.
Perfview is the one of the best tool for this purpose, have used to find numerous production issues.
You can narrow down the issue using multiple approach, it can tell from network level to IIS to asp.net to your C# methods.It uses ETW events emitted by IiS,Asp.net and CLR to do this.
https://channel9.msdn.com/Series/PerfView-Tutorial/PerfView-Tutorial-7-Using-the-Event-Viewer-in-ASPNET-Scenarios
https://channel9.msdn.com/Series/PerfView-Tutorial
using the ThreadTime view you can narrow down to a particular method
Perfview always collect system wide data and you can also set a circular buffer .But best way to collect data is when the issue starts happening, You go to the server in question and start perfview trace and reproduce the issue.Then you can analyze this data later to find the performance bottleneck.
I have Visual Studio 2010 and a pretty large web application project running on IIS 7. Startup for the web application is over a minute (75 seconds). I've attached ANTS to it and very little of the 75 seconds is my code. Most of it seems to be something like CreateAppDomainWithHostingEnvironment and BuildManager stuff. Now I know that ASP .NET will compile dynamically the first time but I certainly don't expect it to compile for that long. Why could I be experiencing this problem and what are some ways I can try to fix it or try to better understand what is taking so much time. Aksi the CPU utilization doesn't seem to be that high. I have an awesome machine.
The problem with the 75 second startup is that for developers working on this, everytime they make a change they have to wait this 75 seconds.
I am using .NET 4.0
EDIT
I ran Microsoft Network Monitor on my machine to see if there was anything suspicious going on the network. There wasn't as far as I can tell though i wasn't sure what to look for (I am familiar with network monitor though so I did have an idea of what I was doing). I tried to run it in release build and though it may have improved the performance a little bit its not really significant
EDIT
I have SQL Session state. As far as i can tell, the connection string is pointing local. For some reasons though, when examinning ANTS, i'm getting a lot PollLockedSessionCallback on many threads. The function seems to be called over 70 times. Does this help at all?
Try building the application in release mode. You can set this in the Build tab of the properties window. You might also consider pre-compiling when publishing the application before deployment.
Are you trying to access anything via a network share at startup? If so, bring those resources local for startup comparison.
I have a large (as in many pages, objects...) asp.net web app that pegs the server at 100% at times for no reason that i can tell. is there a way to detect what page is at fault?
I know its asp.net and not sql because task manager shows w3wp.exe as the culprit.
are there tools for doing this? or profiling .net web apps?
Check out this Red Gate product: http://www.red-gate.com/products/dotnet-development/ants-performance-profiler/
It's not trivial to master, though.
ASP.Net Tracing is the answer.
http://www.asp101.com/articles/robert/tracing/default.asp
http://www.asp.net/general/videos/how-do-i-implement-tracing-in-an-aspnet-web-site
(source: asp101.com)
If you can repeat the issue on a test server or you own machine you can use a profiler (the one included in Visual Studio 2010 or any other) to figure out what's happening.
If the problem only occurs in production you probably are best off performing a dump of the process when the problem is occuring and analyze it using DebugDiag.
Tess Ferrandez has lots of information on her blog on how to perform low level analysis of application behaviour on her blog as well, including how to use DebugDiag.
You wouldn't by chance happening to be using GC.Collect() in your web app would you? Excessive calls to GC.Collect() for large applications will cause it to spike to 100% as it scans memory to look for disposable objects. If you call GC.Collect() in rapid succession, you would likely get a continuous 100% cpu usage when memory usage is high.
I work for a hosting company, providing ASP.Net 3.5 hosting. Honestly, we usually provide quite good uptime and velocity. However, we are having problems with one of our shared pools. As usual, we try to maximize the number of webs that can run into one pool.
Lately we are suffering continuous hangs. The process doesn't crash, but starts to show OutOfMemoryExceptions or stops processing requests. We think this is responsability of one of the applications (it would be great to know which one).
I have some memory dumps that I have processed with WinDbg. I've run f.e:
!dumpheap -stat
This method provide global memory usage of objects. Nothing remarkable... Also I've checked:
~*e!clrstack
I see various non managed threads. In those who are managed appears stacks like:
[HelperMethodFrame_1OBJ: 0f30e320]
System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle...
0f30e3ec 7928b3ff System.Threading.WaitHandle.WaitAny(System.Threading...
0f30e40c 7a55fc89 System.Net.TimerThread.ThreadProc()...
0f30e45c 792d6e46 System.Threading.ThreadHelper.ThreadStart_Context(System...
0f30e468 792f5781 System.Threading.ExecutionContext.runTryCode(System...
At least, I haven't seen exception throwing or similar (in that moment). I've also had access to two scripts written by Tess Ferrandez for calculating the number of sessions and size. Also here not promising results. Anything peculiar or remarkable (24000 bytes as average).
I would like to know what kind of strategies are you usually using facing this kind of problems. Have you ever used Microsoft Support?
Thanks a lot!
Very nice question, well a bad asp.net can hang all shared web apps on the same pool...
Ok let see... if the problem is on memory, get the VMMap from Sysinternals, and also the Process Explorer
Run them both, and from Process explorer find the PID number of pool that you wish to investigate, its under the inetinfo.exe, and have probably the name aspnet_wp.exe.
Now on the VMMap add for monitoring this Pool using for help the PID, and voila, you see the memory and the open images (aspx files) that probably are a lot and make the problems... The files that you going to see are located on temporary of asp.net Framework, but you can connect them and see from witch site they come from.
Well if the problem is not on memory, but the programmer have create bad loops, or even create thread sleeps, then I think process explorer is a way to investigate the pools and search for whats eating the power.
Additional
Maybe a pool recycle every 15minute can solve this issue ?
More about
In those videos there are a lot of informations about VMMap and memory manager.
Mysteries of Windows Memory Management, Part 1, and , Part 2
There are many tools, but it sounds like your main goal is to determine what's causing the problem. This can be done very simply with a binary search.
Break the pool in half, and see which one crashes. Repeat until you have a crashed pool with only one application in it.
This is already O(log2n), but you can speed the process up arbitrarily by dividing into more than two sub-pools.
We have a fairly high volume ASP.Net site written in c# using MS commerce server, running in a 32-bit environment. I see the worker process up to 980 megabytes quite often. I would like to profile this process and determine where any gains could be made in code to reduce the memory foot print of this site. My question what tools have worked well for you doing this sort of thing on ASP.Net web applications?
I am looking for tools that will give me very specific feedback, that will really help to clearly see what needs to change in the code. It would be best if this tool could profile our production environment worker process for a more concrete set of data to compare.
[edit]
So far it seems the consensus is that it's a toss up between Ants and JetBrains. Has anyone used both? If so which one was superior, or what are the pros and cons of each?
There's a free way.
launch the task manager
right-click the w3wp process
select "create dump" (I'm amazed how few people know about this feature - including myself at some point!)
copy the dump file to your local machine (so we don't bother the production server)
open the file in Visual Studio
enjoy
select "Debug Managed memory" for advanced view which class uses memory etc.
AFAIK, the above requires Visual Studio "Ultimate" edition (I guess its called "Enterprise" now?). If you don't have one, then follow these steps (very simple too)
launch WinDbg (free tool, part of Windows SDK, there are tons of answers here on StackOverflow on how to download WinDbg without all the SDK bloatware)
Press Ctrl + D and load the dump file into WinDbg
type .loadby sos clr (this will load SOS.dll that allows WinDbg to analyze .NET processes, SOS.dll is a part of NET Framework so you probably already have it)
type !dumpheap -stat (this will output the class names, sorted by memory usage, ascending order. Skip system.string and system.byte[] classes cause these are side-effects, not the cause...)
UPDATE FROM 2019: WinDbg is now available via MS Store, just search for "WinDbg", then couple of clicks and its there.
ANTS Profiler is very good at profiling ASP.NET applications.
http://www.jetbrains.com/profiler/
Something like ANTS memory profiler might be useful to you.
Also consider AQTime from Automated QA.
dotTrace from JetBrains saved me several times.
It is not free (trial is available), but it is really powerfull!
We use AviCode, and it works pretty good for us.