As we trying to make more frequent builds (and as our site traffic has increased), we've noticed that during higher traffic, the initial post-build page loads spike to 20 or 30 seconds. Our builds are published by copying DLL's + PDB's to a single server which is sync'd to one other server. That file-copy generally takes a few seconds.
What are some contributing factors to this sort of initial latency spike? Are there any commonly taken steps to avoid this problem? (I can't imagine high-traffic site that perform multiple builds/day, if not multiple builds/hour, tolerate this sort of thing.)
The main cause of this delay is ASP.Net doing compilation on the pages during first load, transforming the aspx markup into code.
You can solve this (and is actually listed as the first advantage on this link) by doing a pre-compile during your build. Of course, the trade off to this is longer build times. More information is here: http://msdn.microsoft.com/en-us/library/bb398860(v=vs.100).aspx
If you're using MSBuild to handle your CI builds by using the AspNetCompiler task in MSBuild: http://msdn.microsoft.com/en-us/library/ms164291.aspx
Another advantage this has (And why I tend to use this even in development builds), is if you integrate this into your build process, and you end up with syntax errors on a page, the build will fail, instead of your users being the first one to catch it.
In response to your comment (my response was getting too long for a comment):
Actually, I wasn't even aware of the batch settings myself until now. It looks like setting batch to false makes sense during development to reduce initial load times there. But, it seems that doing so would make asp.net function in assembly-per-page mode, which can slow things down on larger apps. So, probably the best compromise would be in development environments, set batch to false to speed up development time, then use the web.config transforms to set it back to true for production, and use the pre-compiler during the production build. Then you'll only pay the pre-compilation costs once for both servers, and in a way that's not visible to users.
Related
I am working on a .NET WebForms application and I have observed that whenever I build, after the build, the very first page load takes longer to load than usually. This happens even if I wait after building before I load a page. Is there a way to increase human workforce performance by changing IIS/.NET to initialize things on postbuild instead of first page load?
Yes you can, like this.
Quoting:
You can use Application Initialization Module which comes in-box with IIS 8.0, like this:
<applicationInitialization
doAppInitAfterRestart="true" >
<add initializationPage="/" />
</applicationInitialization>
This will send a request to the root of your app (initializationPage="/") every time your app starts automatically.
You can also configure the Start Mode for your application pool to Always Running which means every time IIS restarts, it will make sure to start your application pool immediately (this if from right click on your application pool then Advanced Settings).
Professional servers have hardly any latency, though it requires quite a bit of tweaking. Also, by default, applications recycle regularly on IIS (as well as when some kinds of exceptions occur, when some files are changed, or when some thresholds are reached). Professional web application hosting is anything but simple :) You might get help with that on Server Fault, perhaps.
Another option is to avoid mixing pre-compilation and JIT-compilation - if you only pre-compile, you don't need to do any compilation when the application is deployed, resulting in faster startup times. If you only deploy sources, the application domain doesn't need to be torn down when you make a change, which means that only the change needs to be recompiled, which is much faster.
And of course, ASP.NET Core is much, much faster in both scenarios - it can do the whole compilation in-memory, unlike the legacy system which uses csc to build multiple assemblies, save them to disk, load them from disk, merge them together, save that again, just to load it again and initialize.
My main goal: make development faster by not having to wait as long each time I change the code of my site.
I'm developing an ASP.NET web application in Visual Studio 2010. While developing, I normally run the app with "Control+F5" (ie. start w/o debugging). This starts up the built-in ASP.NET Development Server. However, when I modify code and do this, I get the following:
-Press Control+F5
-The modified project(s) in my solution build
-The web-server is started (if it's not already)
-My browser opens & waits
At this point there's a 15-30 second delay before I see the first page. Reloading a page is instant, as well as modifying an .aspx page and reloading. But changing any code & recompiling causes another 15-30 second delay.
First, I'm trying to see where that time is spent (which is the point of this question). Looking at the log file, there's about 5 seconds until my global.asax.cs file runs. Then a wait of 10-15 seconds, then my Site.master.cs file runs. What is running in that time in between? What files does ASP.NET run and in what order?
Second, I can see that some of this time is spent with "csc.exe", which leads me to believe that the pages are being compiled-on-request. Can I precompile this code (again, using the built-in web server, not IIS) and will this be faster?
I'm open to other suggestions on how to make this faster. I want to minimize the time between modifying code & seeing changes on the site. There are multiple projects in this solution. One project uses the other as a reference. I'm on XP. I can use XP's IIS if that will make things faster. Any ideas?
Thnaks!
If you are curious about the internal processes and architecture of asp.net and IIS work then follow this link.. may be you will get your problem point on which you want to work.
low-level Look at the ASP.NET Architecture
and have look on this question somewhat similar to yours as Aristos comment on your question.
Slow Performance -- ASP .NET ASPNET_WP.EXE and CSC.EXE Running After Clicking Redirect Link
I am getting pretty frustrated by debugging, but maybe I am just doing it wrong.
When I am actively developing it is extremely cumbersome to write some code, fire up the debugger to test said code, wait a minute for the debugger to start, look at the page in the browser, stop the debugger, edit the code, rinse, lather, repeat.
I can get around that by using CTRL-F5 and CTRL-SHIFT-B during development but I lose all the benefits of the debugger.
Is there a better way to use the debugger, or something else I can do to get quick rebuilds and use of the debugger?
Thanks,
Kyle
P.S. I/we do write unit tests, but you also need to test your app in the browser so please no "you shouldn't have this problem if your unit tests were written properly." comments ;)
Update
Thanks for the "get a better machine" suggestions. Not much I can do there. Loads of RAM and an Intel SSD. I shouldn't need more than a $2500 machine to code a web app.
Debug fewer times: If you are stopping the debugger to change values or test different scenarios then don't. While debugging you can change the values of the variables using QuickWatch or the Immediate Window.
Make debugging less costly: Turning off batch will make your page load faster on the first time since it will no longer precompile all of your pages and user controls. This is good for development if you are making changes quite often.
<compilation ... batch="false"> ...</compilation>
You should take a look at this post (tweeted by Scott Guthrie):
Slash your ASP.NET compile/load time without any hard work
http://blog.lavablast.com/post/2010/12/01/Slash-your-ASPNET-compileload-time.aspx
Summary
Get better hardware (Big impact)
Store your temporary IIS files on your fastest disk or a RAM disk e.g. <compilation ... tempDirectory="q:\temp\iistemp\"> ... </compilation>
Restructure your projects
Selectively build the necessary projects
Review a few magical settings (Most impact)
<compilation ... batch="false"> ...</compilation>
<compilation ... optimizeCompilations="true"> ... </compilation>
Get an SSD and boat-loads of RAM.
Maybe what you need is not to debug faster, but to reduce the amount of times you need to debug. Perhaps a more liberal Debug.* or trace logging approach would help.
#Kyle West - Well... there are a bunch of different ways you can go about it. The approach that works best for me is to use the MS Enterprise Library Logging app block (main site) to log events to a rolling daily file. The log level can be ratcheted up (verbose detail) or down (exceptions only), just by editing the .config file.
There is a lot in the app block, so we created a wrapper around the logging calls so that we can more easily make the calls that matter. For example,
DebugEvent.Log(String.Format("the value of _myVariable is {0}", _myVariable))
InfoEvent.Log("Reached the entry to the gatesOfHell method")
ExceptionEvent.Log(ex)
The nice thing about EL is you can change the config without having to change code. So if you want to log to the event log or even email, its just a few lines of configuration.
You can also substitute any other logger (log4Net, etc), or use the built in Debug or Trace in a way that is useful for you.
The statement "Really I just want to see any exceptions that don't buddle up the UI." is a bit worrisome, and implies that exception swallowing or some similar poor practice is happening. (Thats a whole other bucket of roosters, and may be the reason why you have to debug so much).
Well, there are tools like Watin, which allow you to script browser interaction, but I don't think that's really what you want.
I guess the answer here is "Get a faster machine"...
During our build process we run aspnet_compiler.exe against our websites to make sure that all the late-bound stuff in ASP.NET/MVC actually builds (I know nothing about ASP.NET but am assured this is necessary to prevent finding the failures at runtime).
Our sites are fairly large in size, with a few hundred pages/views/controls/etc. however the time taken seems excessive in the 10-15 minute range (for reference, this is longer than it takes the entire solution with approx 40 projects to compile, and we're only pre-compiling two website projects).
I doubt that hardware is the issue as I'm running on the latest Quad core Intel chip, with 4GB RAM and a WD Velociraptor 10,000rpm hard disk. And part of what's odd is that the EXE doesn't seem to be using much CPU (1-5%) and doesn't seem to be doing an awful lot of I/O either.
So... is this a known issue? Why is it so slow? And is there any way to speed it up?
Note: To clarify a couple of things people have answered about, I am not talking about the compilation of code within Visual Studio. We're using web application projects already, and the speed of compilation of those is not the issue. The problem is the pre-compilation of the site after these projects have already been compiled (see this MSDN page for more details) as part of the dev build script. We are performing in-place pre-compilation, not copying the files to a target directory.
Switching to Roslyn compiler most likely will significantly improve precompile time. Here is a good article about it: https://devblogs.microsoft.com/aspnet/enabling-the-net-compiler-platform-roslyn-in-asp-net-applications/.
In addition to this, make sure that batch compilation is enabled by setting batch attribute to true on the compilation element.
Simply, the aspnet_compiler uses what is effectively a "global compiler lock" whenever it starts pre-compiling any individual aspx page; it is basically only allowed to compile each page sequentially.
There are reasons for this (although I personally disagree with them) - primarily, in order to detect and prevent circular references causing an infinite loop of sorts, as well as ensuring that all dependencies are properly built before the requiring page is compiled, they avoid a lot of "nasty CS issues".
I once started writing a massively-forked version of aspnet_compiler.exe last time I worked at a web company, but got tied up with "real work" and never finished it. Biggest problem is the ASPX pages: the MVC/Razor stuff you can parallelize the HELL out of, but the ASPX parse/compile engine is about 20 levels deep of internal and private classes/methods.
Compiler should generate second code-behind file for every .aspx page, check
During compilation, aspnet_compiler.exe will copy ALL of the web site files to the output directory, including css, js and images.
You'll get better compilation times using Web application project instead of Web site model.
I don't have any specific hot tips for this compiler, but when I have this sort of problem, I run ProcMon to see what the process is doing on the machine, and I run Wireshark to check that it isn't spending ages timing-out some network access to a long-forgotten machine which is referenced in some registry key or environment variable.
Just my 2 cents.
One of the things slowing down ASP.NET views precompilation significantly is the -fixednames command line option for aspnet_compiler.exe. Do not use it especially if you're on Razor/MVC.
When publishing the wep app from Visual Studio make sure you select "Do not merge", and do not select "create separate assembly" cause this is what causes the global lock and slows things down.
More info here https://msdn.microsoft.com/en-us/library/hh475319(v=vs.110).aspx
Basically, what I'm wondering is if I need to set debug="false" before hitting the "Publish Web Site" button or if I can switch it after all the files have been published.
You do not have to turn that setting off, however, you will want to set debug="false" before running the website as a production application. It will have a profound impact on your site's performance.
As to what Ryan wrote - see debug code in production.
Another option you may want to use is retail="true".
You can keep it set to true when you publish/precompile, but once its at a production status, its strongly recommended that you set the value to false, the reasons are here outlined by Scott Guthrie (he manages the ASP.NET team) himself.
Highlights from Scott's post:
Doing so causes a number of non-optimal things to happen including:
1) The compilation of ASP.NET pages takes longer (since some batch optimizations are disabled)
2) Code can execute slower (since some additional debug paths are enabled)
3) Much more memory is used within the application at runtime
4) Scripts and images downloaded from the WebResources.axd handler are not cached
On the production server you could put deployment retail=”true” in machine.config. This will ensure that debug is always false for that server. Details here.
Unfortunately, no. You can publish it with the option set to true, although you certainly should not (if the page is going into production). I publish apps to a test environment initially with it set to 'true', the set it to false in the test environment, and absolutely to false when into production.
But your site will build and publish fine to whatever environment you send it to with the debug set to true. As I understand it, there are a quite a few sites out there that in production with this set that way. Dror posted a great link to the issues that come with leaving this option on.
I hope this adds some value. I have been learning a lot about msbuild, and what a incredible tool it is!
If you are publishing sites to production, there is probably some of the steps that you would like to automate. If you want to use a batch file to do your publish instead of the visual studio interface, or included this in your build scripts, after you have compiled sucessfully the following will publish your website.
msbuild <yourProjectFile>.csproj
/target:"ResolveReferences;_CopyWebApplication"
/properties:"debug=false;
retail=true;
WebProjectOutputDir = <YourCorrectOutputDir>;
OutDir = <YourCorrectOutputDir>\bin\"
The time we have spent in our time, investing in fairly extensive build scripts, that even run a few basic http get reguests against the site after we are done building and publishing it, has reduced a huge amount of frustrations.
I am a firm beleiver of automating as much as possible, machines dont seem to forget to do things as often as I do :D
Hope it helps
Cheers
Rihan