We have a fairly high volume ASP.Net site written in c# using MS commerce server, running in a 32-bit environment. I see the worker process up to 980 megabytes quite often. I would like to profile this process and determine where any gains could be made in code to reduce the memory foot print of this site. My question what tools have worked well for you doing this sort of thing on ASP.Net web applications?
I am looking for tools that will give me very specific feedback, that will really help to clearly see what needs to change in the code. It would be best if this tool could profile our production environment worker process for a more concrete set of data to compare.
[edit]
So far it seems the consensus is that it's a toss up between Ants and JetBrains. Has anyone used both? If so which one was superior, or what are the pros and cons of each?
There's a free way.
launch the task manager
right-click the w3wp process
select "create dump" (I'm amazed how few people know about this feature - including myself at some point!)
copy the dump file to your local machine (so we don't bother the production server)
open the file in Visual Studio
enjoy
select "Debug Managed memory" for advanced view which class uses memory etc.
AFAIK, the above requires Visual Studio "Ultimate" edition (I guess its called "Enterprise" now?). If you don't have one, then follow these steps (very simple too)
launch WinDbg (free tool, part of Windows SDK, there are tons of answers here on StackOverflow on how to download WinDbg without all the SDK bloatware)
Press Ctrl + D and load the dump file into WinDbg
type .loadby sos clr (this will load SOS.dll that allows WinDbg to analyze .NET processes, SOS.dll is a part of NET Framework so you probably already have it)
type !dumpheap -stat (this will output the class names, sorted by memory usage, ascending order. Skip system.string and system.byte[] classes cause these are side-effects, not the cause...)
UPDATE FROM 2019: WinDbg is now available via MS Store, just search for "WinDbg", then couple of clicks and its there.
ANTS Profiler is very good at profiling ASP.NET applications.
http://www.jetbrains.com/profiler/
Something like ANTS memory profiler might be useful to you.
Also consider AQTime from Automated QA.
dotTrace from JetBrains saved me several times.
It is not free (trial is available), but it is really powerfull!
We use AviCode, and it works pretty good for us.
Related
I am trying to profile a web application running on IIS in a Production environment (.Net framework 4.0, WebForms, SQLServer, Windows 2008 server) .
Several pages are repeatedly slow in Production, but we are unable to reproduce in Development.
We cannot install any IDE or similar tool in the Production environment.
Does anyone know of a DLL or a stand-alone exe that we could easily drop into the server, run for about an hour, and then quickly and easily remove?.... we are seeking one important aspect to profiling:
--> what is the amount of time spent for each CSharp method to run. <--
Thank you in advance.
Try this https://github.com/jitbit/cpu-analyzer command-line tool for profiling cpu load in production.
We forked this from the original Sam Saffron's cpu profiler, which is kinda abandoned now.
Disclaimer: I'm the maintainer of this project.
Perfview is the one of the best tool for this purpose, have used to find numerous production issues.
You can narrow down the issue using multiple approach, it can tell from network level to IIS to asp.net to your C# methods.It uses ETW events emitted by IiS,Asp.net and CLR to do this.
https://channel9.msdn.com/Series/PerfView-Tutorial/PerfView-Tutorial-7-Using-the-Event-Viewer-in-ASPNET-Scenarios
https://channel9.msdn.com/Series/PerfView-Tutorial
using the ThreadTime view you can narrow down to a particular method
Perfview always collect system wide data and you can also set a circular buffer .But best way to collect data is when the issue starts happening, You go to the server in question and start perfview trace and reproduce the issue.Then you can analyze this data later to find the performance bottleneck.
You know, the next "big" and "enterprisey" thing from Microsoft.
Is it just me, or is it really hardly for humans? Main highlights are (IMO):
Absolutely cryptic syntax (-skip:objectName=filePath,absolutePath=App_Offline.* just for skipping App_Offline.html)
Manifest as an after thought
Lack of thorough documentation
Not a word about extensibility (except for several blog posts out there). Moreover, all these extensions developed in great pains have to be registered in GAC and registry
Waaay too low-level (metadata/metakey; all this IIS jazz)
No integration with MSBuild
Granted, MSDeploy and MSDeployAgent are quite powerful, but do they really need to be that complex for relatively simple tasks?
I too share your frustrations over the lack of documentation and the apparent low-level nature of this tool.
However what MS has done is finally create a free tool with which you can actually script whole server deployments, including parameterising addresses, configurations etc. This is unfortunately a very complicated thing to do - given how many bits of configuration actually go into a web server - and this is probably the best way to do it all.
What we need now is a really good GUI that can help build up these packages, and scripts etc. The GUI that is embedded within IIS is good - but again, short on explanation - so hopefully soon that'll be addressed.
On the functional side, I'm using at the moment to deploy a site from dev -> staging -> live with parameters to change bound IP addresses etc. I was deeply frustrated that it took me a few days to get it all working - however now I have it, I can remove a lot of the possibly of human error at the IT Support side - who are responsible for our deployments. I now only have the configuration of my master staging server to worry about - and can be sure that all the servers in the web farm will be kept in sync whenever I deploy.
As Sayed mentions, as well, there are MSBuild tasks in 2010 (the Website Deployment feature is now implemented using msdeploy) to work with this - which also brings the possibility of a true Continuous Integration environment to VSTeamSystem - having a team build that can actually perform a full web deployment as its last step is very exciting (and scary, granted!).
Actually there are MSBuild tasks for MSDeploy. They will be shipped with .NET 4/Visual Studio 2010.
Although a bit rough around the edges, I've come to like MSDeploy quite a bit. Using it to sync web servers in a farm is very useful as it is efficient (only copies changes) and takes care of actual IIS settings in addition to content files. It seems like MSDeploy is a building block for various scenarios and uses. Also, as previously mentioned, there is a MSBuild task for MSDeploy in .NET 4. I've taken advantage of this MSBuild task to make deployment of my web applications from TeamCity trivially easy. I've blogged here it here:
Web Deploy (MS Deploy) from TeamCity - http://www.geekytidbits.com/web-deploy-ms-deploy-from-teamcity/
I have recently started implementing a deployment pipeline and I found below links quite useful:
MSBuild commands I used for Continuous Integration:
http://www.troyhunt.com/2010/11/you-deploying-it-wrong-teamcity_24.html
WebDeploy sync commands, I used for deployment packages to production server:
http://sedodream.com/2012/08/20/WebDeployMSDeployHowToSyncAFolder.aspx
Also I used these references:
Video about MSBuild on dnrtv.com
Microsoft Press book called "Inside the Microsoft® Build Engine: Using MSBuild and Team Foundation Build" which you can buy PDF version from Oreilly
Finally, "Continuous Delivery" book, gave me good ideas about deployment pipe line, although the book is not focusing on MSDeploy, but it is really worth reading.
The statement of documentation is typical of a MSFT 1.0 product, unfortunately MSDN no longer have dedicated Developer Technology Engineers to fill the gaps --- instead, there is a blind faith that the web will provide it.
I am actually considering dusting off my writing skills and write a short ebook on it since there is likely a market for it....
Msdeploy definitely has a touch of the PowerShell to it: power over simplicity rather than worse is better.
There is no Windows alternative to it, however you can hybridize some of its powers to make automated deployments. For example:
Compile your solution with Team City and msbuild
Use msdeploy to transform your site and web.configs on the build server
Manually FTP a ZIP file of your site (it doesn't support FTP)
Alternatively, use its remote deploy capabilities. This requires port 8172 open, lots of security changes and as far as I'm aware no concessions for load balancing
Use msdeploy on the live site to sync changes
As a tool it's clearly aimed at service providers as it's an enormous Swiss army knife. You can do all kinds of things to IIS with it, which for the most part are over kill for small businesses. I've no experience of large scale IIS setups so maybe that's where it shines.
There is a great presentation by Dan Farino, Chief Systems Architect at MySpace.com, showcasing a web-based stack dump tool that catalogues all threads running in a given process (what they're doing, how long they've been executing, etc.)
Their techniques are also summarized on highscalability.com:
PerfCollector.
Centralized
collection of performance data via
UDP. More reliable than Windows and
allows any client to connect and see
stats.
Web Based Stack Dump Tool.
Can right-click on a problem server
and get stack dump of the .Net
managed threads. Used to have to RDC
into system and attach a debugger and
1/2 later get an answer. Slow,
nonscalable, and tedious. Not just a
stack dump, gives a lot of context
about what the thread is doing.
Troubleshooting is easier because you
can see 90 threads are blocked on a
database so the database may be down.
Web Base Heap Dump Tool.
Dumps all
memory allocations. Very useful for
developers. Save hours of doing it by
hand. • Profiler. Traces a request
from start to finish and produces a
report. See URL, methods, status,
everything that will help you
identify a slow request. Looks at
lock contentions, are a lot of
exceptions being thrown, anything
that might be interesting. Very light
weight. It's running on one box in
every VIP (group of 100 servers) in
production. Samples 1 thread every 10
seconds. Always tracing in
background.
The question is: what tools are required to build a web-based stack dump tool for ASP.NET? For convenience, let's assume that an *.aspx hosted in the target AppDomain, able to output all managed call stacks in that process, is sufficient.
There are a few posts that cover the use of Mdbg (debugger for managed code written entirely in C#/IL that started shipping with CLR 2 SDK) and the mdbgcore assembly typically found in C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin:
http://dotnetdebug.net/2005/11/09/exceptiondbg-v01-debug-your-exceptions/
http://blogs.msdn.com/jmstall/archive/tags/MDbg/default.aspx
http://blogs.msdn.com/vijaysk/archive/2009/11/04/asp-net-debugger-extension-for-iis-7.aspx
Would a solution simply reference this assembly to produce the desired output? What impact would a "list all managed call-stacks" operation have on the running process that's servicing production traffic?
I believe the profiling API of .Net are the way to go.
Look at the SlimTune project on Google Code to have a live sample, with sources, that you can check how to adapt and improve to work in a Asp.NET scenario.
Regards
Massimo
With the profiling API of .Net you have to stop the server and it takes a lot CPU (but it gives you full control over all called methods).
I think the most “light way” solution is to doing this with MDbg, I put together a very small but useful little app called StackDump that does the following:
1) The debugger stops the application and generates a list of all CLR stacks running for the process.
2) The application is started again.
This operation is a quick operation and can (maybe) be executed on a running production server with unchanged production code.
It just 80 lines of .Net code to manage this. I have published the source code on Codeplex.
I am going to develop my first application (4 members team).I am not aware of source control tool ( Visual
source safe,Tortoise SVN).
My .NET version is 3.5. What is the best source control tool ?
Is CruiseControl a source control tool ?
Definitely avoid Visual Source Safe.
Subversion is probably a safe choice, but you will have to elaborate about your situation (E.g. how big is your team?) to get more specific advice.
Another no vote for Visual Source Safe here.
Might be an idea to get familiar with subversion as - just as others mentioned - it's widely adopted, so might be useful in the future and simply works..oh and it's free too!
TortoiseSVN integrates into the windows explorer and is both easy to use and well documented so I would give it a try.
If it's a small project of yours and you don't want to worry about hosting, I would recommend signing up for a free account at something like beanstalk, to keep it safe, without any hassle.
But if it's your first ever project, it might just give you too many additional things to learn about, so you might want to put it aside for a while.
Try GIT, it's much lighter weight than svn
http://sourceforge.net/projects/gitextensions/
you can use a local repository for just checking in your own work (to keep a history, or to keep a working version before changing everything) Or you can connect to a central repository for enterprise source control.
The company where I work used to use the SVN, Tortoise, Ankh stack but we're using Git now. Plus I use it at home on my on play projects.
(quated part is no more important since you edited your post about team size)
If you are doing some small project on
your own, maybe you shouldn't use code
versioning at all because you probably
don't need it. Code versioning helps
you with central code storage,
multi-person development code merging
(several people working on the same
file and then commit changes) and code
branching to name the most important
ones.
If I were you, and I would be the only
person developing something simple, I
wouldn't use it, because it would also
be a bit of a time-overkill.
But otherwise source control doesn't address technology like .Net framework version. Code control only stores files (with history) and is able to merge text files. Written in whatever language possible.
If you are part of a team I'd suggest using Subversion + TortoiseSVN + AnkhSVN
And No, CruiseControl is not a code versioning system. It's a continuous integration system.
If you have a pure Windows environment then VisualSVN server is a Windows version of SVN server and installs easily and cleanly. You can setup security using Windows usernames and passwords, or SVN usernames and passwords.
You can then use Tortoise to provide integration in Explorer and VisualSVN to provide Visual Studio integration (there's a small cost for VisualSVN) or you could try AnkhSVN if all you want is VS integration and don't want to spend any money. Personally I prefer VisualSVN's integration.
Running a GIT server under Windows is, umm, frankly a pain in the ass, and not a route I'd recommend for beginners. VS integration is also fraught with pain.
As with everyone else I'd say avoid SourceSafe like the plague.
CruiseControl is not a source control system, it's an automated build server. It monitors source control servers looking for changes, then checks everything out, compiles it and runs it through your unit tests, so people know if they've broken the build. Once you have source control up and running it's the next big step towards a better build environment.
Before we start talking about source control, I would like you to consider your actual need for it, if this is to be your first .NET application. Is this your actual first attempt of writing some .NET 3.5 code? If so, I doubt that you need to worry about source control just yet. I would say it's not just a little bit overkill for the first learning projects.
On the other hand, if this is to be your first application that you'll actually sell to someone, it's a completely different matter.
I currently use VisualSVN, an add-in to Visual Studio that lets you do check-in, check-out etc from within the environment. Along with it, I use the free SVN repository service at XP-Dev. It works well for me =)
Your choice of a versioning system does not have to do anything with the .NET version you are using.
I'd vote for Subversion / Tortoise SVN. It's rock-solid, has lots of handy features, widely adopted and free.
Is it better to use NGEN an ASP.NET application when we know it is not going to change much? Or is the JIT good enough?
The only reason I asked was because this article by Jeffrey Richter in 2002 says :
And, of course, Microsoft is working quite hard at improving the CLR
and its JIT compiler so that it runs faster, produces more optimized
code, and uses memory more efficiently. These improvements will take
time. For developers that can't wait, the .NET Framework
redistributable includes a utility called NGen.exe.
NGen will only help startup time - it doesn't make the code execute any faster than it would after JITting. Indeed, I believe there are some optimizations which NGen doesn't do but the JIT does.
So, the main question is: do you have an issue with startup time? I don't know how much of an ASP.NET application's start-up time will be JITting vs other costs, btw... you should probably look at the Performance Manager graphs for the JIT to tell you how much time it's really costing you.
(In terms of availability, having multiple servers so you can do rolling restarts is going to give you much more benefit than a single server with an NGENed web app.)
NGen isn't the way to go for ASP.NET -- the creation of the .dlls in the bin folder isn't the final step -- they are compiled again with the web/maching.config settings applied into your C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files folder. Instead of NGen, to decrease initial load-time, use the Publish Website Tool or aspnet_compiler.exe
I'm not sure that NGEN's primary benefit is to start-up time alone - if the application's suffering from a high '% Time in JIT', this is listed as a potential remedy:
http://msdn.microsoft.com/en-us/library/dd264972(VS.100).aspx.
The discussion is closely related to a different question on how JIT'd machine code is cached and re-used?
I was looking into this tonight and came across the following:
The common language runtime cannot load images that you create with NGEN into the shared application domain. Because ASP.NET standard assemblies are shared and are then loaded into a shared application domain, you cannot use Ngen.exe to install them into the native image cache on the local computer.
http://support.microsoft.com/kb/331979
Not sure if this just refers to assemblies referenced from ASP.net app or the app itself?
NGen helps startup time. For example, the Entity Framework documentation says this about the benefit of running NGen:
Empirical observations show that native images of the EF runtime assemblies can cut between 1 and 3 seconds of application startup time.
The context is just the Entity Framework assemblies. NGenning other assemblies would provide additional speedup.