I am trying to profile a web application running on IIS in a Production environment (.Net framework 4.0, WebForms, SQLServer, Windows 2008 server) .
Several pages are repeatedly slow in Production, but we are unable to reproduce in Development.
We cannot install any IDE or similar tool in the Production environment.
Does anyone know of a DLL or a stand-alone exe that we could easily drop into the server, run for about an hour, and then quickly and easily remove?.... we are seeking one important aspect to profiling:
--> what is the amount of time spent for each CSharp method to run. <--
Thank you in advance.
Try this https://github.com/jitbit/cpu-analyzer command-line tool for profiling cpu load in production.
We forked this from the original Sam Saffron's cpu profiler, which is kinda abandoned now.
Disclaimer: I'm the maintainer of this project.
Perfview is the one of the best tool for this purpose, have used to find numerous production issues.
You can narrow down the issue using multiple approach, it can tell from network level to IIS to asp.net to your C# methods.It uses ETW events emitted by IiS,Asp.net and CLR to do this.
https://channel9.msdn.com/Series/PerfView-Tutorial/PerfView-Tutorial-7-Using-the-Event-Viewer-in-ASPNET-Scenarios
https://channel9.msdn.com/Series/PerfView-Tutorial
using the ThreadTime view you can narrow down to a particular method
Perfview always collect system wide data and you can also set a circular buffer .But best way to collect data is when the issue starts happening, You go to the server in question and start perfview trace and reproduce the issue.Then you can analyze this data later to find the performance bottleneck.
Related
I have Visual Studio 2010 and a pretty large web application project running on IIS 7. Startup for the web application is over a minute (75 seconds). I've attached ANTS to it and very little of the 75 seconds is my code. Most of it seems to be something like CreateAppDomainWithHostingEnvironment and BuildManager stuff. Now I know that ASP .NET will compile dynamically the first time but I certainly don't expect it to compile for that long. Why could I be experiencing this problem and what are some ways I can try to fix it or try to better understand what is taking so much time. Aksi the CPU utilization doesn't seem to be that high. I have an awesome machine.
The problem with the 75 second startup is that for developers working on this, everytime they make a change they have to wait this 75 seconds.
I am using .NET 4.0
EDIT
I ran Microsoft Network Monitor on my machine to see if there was anything suspicious going on the network. There wasn't as far as I can tell though i wasn't sure what to look for (I am familiar with network monitor though so I did have an idea of what I was doing). I tried to run it in release build and though it may have improved the performance a little bit its not really significant
EDIT
I have SQL Session state. As far as i can tell, the connection string is pointing local. For some reasons though, when examinning ANTS, i'm getting a lot PollLockedSessionCallback on many threads. The function seems to be called over 70 times. Does this help at all?
Try building the application in release mode. You can set this in the Build tab of the properties window. You might also consider pre-compiling when publishing the application before deployment.
Are you trying to access anything via a network share at startup? If so, bring those resources local for startup comparison.
I need to set up a web server on WinCE (7.0) and the following are some of the features that I need to implement through this web site:
be able to update the device software (run an exe) and display output
start/end process
display the run-time of device
Should I use ASP or ISAPI on my web server on WinCE 7(WEC 7) platform? If you include some reasons for your response, it would be great.
I was planning on using ASP because of its simplicity compared to ISAPI; however, when I tested a simple Hello World(hello.asp) page on the device, it took around 30 seconds to load the page. For testing purposes, I am using Windows Virtual PC. So I am not sure what is causing it to be so slow. If you could please enlighten me on this topic too, it would be great.
I looked up ASP .NET Compact framework but I am concerned about performance again.
I am beginner on WinCE development and web development so any answers would be greatly appreciated.
If you need any other information to better answer the question, please let me know and I will provide it.
Classic ASP on CE is really, really limited. You're not going to be able to hit your targets without a custom COM control, which is not very fun to write or debug and it will have to be installed on the service device. Out of the two you've proposed, ISAPI is the only one that will reasonably do what you want.
It's a side note, but from experience I'd not use either due to the absolute pain of debugging either one. I'd use a separate commercial ASP.NET web server for Windows CE. Of course I created it, so I'm biased, but really, even creating your own port 80 server is going to be less painful than the piss-poor out-of-box offerings in CE.
I found out the problem with the slow loading ASP. It is an error with the WinCE 7 web server. Microsoft has released an update that fixes that problem.
Windows Embedded Compact 7 Monthly Update July 2011
So I think I will go ahead and use ASP. I'll see if I can add the features that I need to implement.
There is a great presentation by Dan Farino, Chief Systems Architect at MySpace.com, showcasing a web-based stack dump tool that catalogues all threads running in a given process (what they're doing, how long they've been executing, etc.)
Their techniques are also summarized on highscalability.com:
PerfCollector.
Centralized
collection of performance data via
UDP. More reliable than Windows and
allows any client to connect and see
stats.
Web Based Stack Dump Tool.
Can right-click on a problem server
and get stack dump of the .Net
managed threads. Used to have to RDC
into system and attach a debugger and
1/2 later get an answer. Slow,
nonscalable, and tedious. Not just a
stack dump, gives a lot of context
about what the thread is doing.
Troubleshooting is easier because you
can see 90 threads are blocked on a
database so the database may be down.
Web Base Heap Dump Tool.
Dumps all
memory allocations. Very useful for
developers. Save hours of doing it by
hand. • Profiler. Traces a request
from start to finish and produces a
report. See URL, methods, status,
everything that will help you
identify a slow request. Looks at
lock contentions, are a lot of
exceptions being thrown, anything
that might be interesting. Very light
weight. It's running on one box in
every VIP (group of 100 servers) in
production. Samples 1 thread every 10
seconds. Always tracing in
background.
The question is: what tools are required to build a web-based stack dump tool for ASP.NET? For convenience, let's assume that an *.aspx hosted in the target AppDomain, able to output all managed call stacks in that process, is sufficient.
There are a few posts that cover the use of Mdbg (debugger for managed code written entirely in C#/IL that started shipping with CLR 2 SDK) and the mdbgcore assembly typically found in C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin:
http://dotnetdebug.net/2005/11/09/exceptiondbg-v01-debug-your-exceptions/
http://blogs.msdn.com/jmstall/archive/tags/MDbg/default.aspx
http://blogs.msdn.com/vijaysk/archive/2009/11/04/asp-net-debugger-extension-for-iis-7.aspx
Would a solution simply reference this assembly to produce the desired output? What impact would a "list all managed call-stacks" operation have on the running process that's servicing production traffic?
I believe the profiling API of .Net are the way to go.
Look at the SlimTune project on Google Code to have a live sample, with sources, that you can check how to adapt and improve to work in a Asp.NET scenario.
Regards
Massimo
With the profiling API of .Net you have to stop the server and it takes a lot CPU (but it gives you full control over all called methods).
I think the most “light way” solution is to doing this with MDbg, I put together a very small but useful little app called StackDump that does the following:
1) The debugger stops the application and generates a list of all CLR stacks running for the process.
2) The application is started again.
This operation is a quick operation and can (maybe) be executed on a running production server with unchanged production code.
It just 80 lines of .Net code to manage this. I have published the source code on Codeplex.
We have a fairly high volume ASP.Net site written in c# using MS commerce server, running in a 32-bit environment. I see the worker process up to 980 megabytes quite often. I would like to profile this process and determine where any gains could be made in code to reduce the memory foot print of this site. My question what tools have worked well for you doing this sort of thing on ASP.Net web applications?
I am looking for tools that will give me very specific feedback, that will really help to clearly see what needs to change in the code. It would be best if this tool could profile our production environment worker process for a more concrete set of data to compare.
[edit]
So far it seems the consensus is that it's a toss up between Ants and JetBrains. Has anyone used both? If so which one was superior, or what are the pros and cons of each?
There's a free way.
launch the task manager
right-click the w3wp process
select "create dump" (I'm amazed how few people know about this feature - including myself at some point!)
copy the dump file to your local machine (so we don't bother the production server)
open the file in Visual Studio
enjoy
select "Debug Managed memory" for advanced view which class uses memory etc.
AFAIK, the above requires Visual Studio "Ultimate" edition (I guess its called "Enterprise" now?). If you don't have one, then follow these steps (very simple too)
launch WinDbg (free tool, part of Windows SDK, there are tons of answers here on StackOverflow on how to download WinDbg without all the SDK bloatware)
Press Ctrl + D and load the dump file into WinDbg
type .loadby sos clr (this will load SOS.dll that allows WinDbg to analyze .NET processes, SOS.dll is a part of NET Framework so you probably already have it)
type !dumpheap -stat (this will output the class names, sorted by memory usage, ascending order. Skip system.string and system.byte[] classes cause these are side-effects, not the cause...)
UPDATE FROM 2019: WinDbg is now available via MS Store, just search for "WinDbg", then couple of clicks and its there.
ANTS Profiler is very good at profiling ASP.NET applications.
http://www.jetbrains.com/profiler/
Something like ANTS memory profiler might be useful to you.
Also consider AQTime from Automated QA.
dotTrace from JetBrains saved me several times.
It is not free (trial is available), but it is really powerfull!
We use AviCode, and it works pretty good for us.
Is it better to use NGEN an ASP.NET application when we know it is not going to change much? Or is the JIT good enough?
The only reason I asked was because this article by Jeffrey Richter in 2002 says :
And, of course, Microsoft is working quite hard at improving the CLR
and its JIT compiler so that it runs faster, produces more optimized
code, and uses memory more efficiently. These improvements will take
time. For developers that can't wait, the .NET Framework
redistributable includes a utility called NGen.exe.
NGen will only help startup time - it doesn't make the code execute any faster than it would after JITting. Indeed, I believe there are some optimizations which NGen doesn't do but the JIT does.
So, the main question is: do you have an issue with startup time? I don't know how much of an ASP.NET application's start-up time will be JITting vs other costs, btw... you should probably look at the Performance Manager graphs for the JIT to tell you how much time it's really costing you.
(In terms of availability, having multiple servers so you can do rolling restarts is going to give you much more benefit than a single server with an NGENed web app.)
NGen isn't the way to go for ASP.NET -- the creation of the .dlls in the bin folder isn't the final step -- they are compiled again with the web/maching.config settings applied into your C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files folder. Instead of NGen, to decrease initial load-time, use the Publish Website Tool or aspnet_compiler.exe
I'm not sure that NGEN's primary benefit is to start-up time alone - if the application's suffering from a high '% Time in JIT', this is listed as a potential remedy:
http://msdn.microsoft.com/en-us/library/dd264972(VS.100).aspx.
The discussion is closely related to a different question on how JIT'd machine code is cached and re-used?
I was looking into this tonight and came across the following:
The common language runtime cannot load images that you create with NGEN into the shared application domain. Because ASP.NET standard assemblies are shared and are then loaded into a shared application domain, you cannot use Ngen.exe to install them into the native image cache on the local computer.
http://support.microsoft.com/kb/331979
Not sure if this just refers to assemblies referenced from ASP.net app or the app itself?
NGen helps startup time. For example, the Entity Framework documentation says this about the benefit of running NGen:
Empirical observations show that native images of the EF runtime assemblies can cut between 1 and 3 seconds of application startup time.
The context is just the Entity Framework assemblies. NGenning other assemblies would provide additional speedup.