I have an asp.net mvc4 application that we deploy to about 1400 clients. Our current deployment process goes like this:
:: Publish the project files to a local directory
%msbuild% RPortal.csproj "/p:Platform=AnyCPU;Configuration=Release;ConfigurationName=Release;SolutionDir=%solutionDir%;PublishDestination=.\Deploy\Release" /t:PublishToFileSystem
And then we have some powershell scripts that push the new build to each of our clients, using other tools to help with the sync.
We are noticing some sever slowness with application boot up times (sometimes upwards of 5 minutes), and one of the approaches we've investigated to solve this would be the idea of ASP.Net Precompiling (http://msdn.microsoft.com/en-us/library/vstudio/bb398860%28v=vs.100%29.aspx)
In some experiments with the idea, it appears that calling aspnet_compile.exe on our published system does, in fact, create some new files. (A couple of dlls, and a few *.compiled files).
My questions are as follows:
How does this compilation differ from the primary compilation from the .cs source
files?
Does this compilation happen regardless, at first run, or only when run manually?
(related to 2), Does this compilation survive app pool restarts and server reboots?
With our current scenario, it seems that we are killing our servers trying to get 1400 applications to start up (yes, they all live on 1 webserver... that's not a situation I am in control of). The server will be humming along, with no particular problems, resonable resource consumption, and then, all of a sudden, our CPU will peg to 100% and stay there. The only factor we can tie it back to is that it happens when more than 5 (or so) of the 1400 apps are all booting up.
Our hope is that precompiling will front load most of the app start burden, but I clearly don't understand whats really going on here.
Any light you might shed would be most appreciated.
Related
I am working on a .NET WebForms application and I have observed that whenever I build, after the build, the very first page load takes longer to load than usually. This happens even if I wait after building before I load a page. Is there a way to increase human workforce performance by changing IIS/.NET to initialize things on postbuild instead of first page load?
Yes you can, like this.
Quoting:
You can use Application Initialization Module which comes in-box with IIS 8.0, like this:
<applicationInitialization
doAppInitAfterRestart="true" >
<add initializationPage="/" />
</applicationInitialization>
This will send a request to the root of your app (initializationPage="/") every time your app starts automatically.
You can also configure the Start Mode for your application pool to Always Running which means every time IIS restarts, it will make sure to start your application pool immediately (this if from right click on your application pool then Advanced Settings).
Professional servers have hardly any latency, though it requires quite a bit of tweaking. Also, by default, applications recycle regularly on IIS (as well as when some kinds of exceptions occur, when some files are changed, or when some thresholds are reached). Professional web application hosting is anything but simple :) You might get help with that on Server Fault, perhaps.
Another option is to avoid mixing pre-compilation and JIT-compilation - if you only pre-compile, you don't need to do any compilation when the application is deployed, resulting in faster startup times. If you only deploy sources, the application domain doesn't need to be torn down when you make a change, which means that only the change needs to be recompiled, which is much faster.
And of course, ASP.NET Core is much, much faster in both scenarios - it can do the whole compilation in-memory, unlike the legacy system which uses csc to build multiple assemblies, save them to disk, load them from disk, merge them together, save that again, just to load it again and initialize.
I am a C# winforms programmer, not used to ASP.Net.
As a winforms programmer I build regularly to detect syntax errors.
Recently I opened a Kentico website in Visual Studio and to my surprise found that there were build errors.
Does this matter?
My instinct is to go about correcting the site until it builds. This is a side track from what I set out to do.
If you are attempting to build any kind of quality into your project/software, then yes, it does matter if it builds.
Regarding Kentico and build times, if you're using a website vs. a web project, yes the build times are typically longer and range anywhere from a few minutes to I've seen upwards of an hour. The build times depend greatly on the machine building it as well. So if your machine has a Celeron processor, with 1GB of RAM and a 5400 RPM drive, you're going to take longer to build than a machine with an i7 processor, 16GB of RAM and a solid state hard drive that can read/write 500+ MB/s. Also keep in mind Kentico out of the box has over 9000 system files in it so as a website, it will take some time to build.
One of the first things I check when a site doesn't build is to ensure all the referenced DLLs are in the website/project. If not, this will cause several errors and is usually a very simple fix. If you have any kind of errors from code which resides in the /App_Code directory, your site will NOT run at all when you publish it. If you have errors within any other directory, the site will run BUT wherever those code files are referenced on the website, will display errors. So in your instance if you have webpart files in the /CMSWebparts/OurCompany folder, if those webparts are placed on pages within the website, those pages will error out even though the rest of the site is running.
In my opinion, just fix the errors and be done with them. Then check the code into a version control system to keep track of the changes.
Does this matter?
It depends on what you are trying to achieve with your website. If you want to make it available to the public then building is definitely something you should consider as top priority. If on the other hand you want to have the source code open in Visual Studio on your local machine, just for reading purposes, then building is not necessary.
We have a Sitecore 6.6 instance which is used to host multiple sites. It is hosted in IIS 7.5. We developed custom Sitecore sublayouts and pipelines which are used across websites.
When any dll is deployed in bin folder, the Sitecore site takes long time to startup (8-10 mins). But when IIS is reset, startup time is less (30-40 seconds).
What could be the reason for application startup time to be more for DLL deployment than IIS Reset ?
Any suggestions to improve the application startup time for DLL deployment ?
Update 1: The startup time after DLL deployment impacts our build process as it increases the overall build deployment time in all environments (DEV,STG,LIVE).
Profiling snapshot of w3wp process revealed two major hotspots:
Sitecore.Threading.Semaphore.P
Sitecore.IO.FileWatcher.Worker
Update 2: After following the deployment suggested by Vicent, profiling snapshot of w3wp process revealed major hotspot at
Sitecore.Web.UI.WebControls.Sublayout.GetUserControl(Page)
Further analysis of memory dump showed that thread was waiting for JIT compilation of newly deployed DLL.
For me, it's like your problem is not the start up of sitecore, but the shutdown.
When you copy your dll, the filewatcher detects the change in the bin folder (write it to the logs) and tries to shutdown sitecore (log this too), but if sitecore has tasks running on different threads, (indexing, publishing, scheduled tasks, etc), the semaphore will wait until the other threads finish normally.
That's why when you "kill" the process without waiting for threads to finish sitecore starts up quickly.
I've this behaviour in my environments too, so when i need a quick restart, i copy the dll, wait for a few seconds so at least sitecore tries to shutdown and then and i kill the w3p.exe related to my pool. I won't advice nobody to do this, but i don't have any way to "kindly" kill the threads... Maybe somebody knows how to "force shutdown nicely..."
This blog post by Alex Shyba has some interesting pointers to improve the startup time for Sitecore (but might not be applicable if you're talking about a live environment rather than a DEV environment).
It might also be worth checking your prefetch caches and running through the Performance Tuning Guide if you haven't done that yet.
I've seen this problem before. It happened on version 6.5, but I've not seen a fix for it in the release notes since then.
Sitecore Support has a hotfix for this - it was indeed related to something with their Filesystem watcher tasks. You would need to raise a ticket with them, to get the hotfix or additional information.
My support ticket reference for this issue is 370593. The hotfix has issue 323775. If you mention this in your support ticket, it should speed up the process a bit - if it is indeed the same issue you're experiencing.
I have an ASP.NET 3.5 website running under IIS7 on Windows 2008.
When I restart IIS (iisreset), then hit a page, the initial startup is really slow.
I see the following activity in Process Explorer:
w3wp.exe spawns, but shows 0% CPU
activity for about 60 seconds
Finally, w3wp.exe goes to 50% CPU for
about 5 seconds and then the page
loads.
I don't see any other processes using CPU during this time either. It basically just hangs.
What's going on during all that time? How can I track down what is taking all this time?
We had a similar problem and it turned out to be Windows timing out checking for the revocation of signing certificates. Check to see if your server is trying to call out somewhere (e.g. crl.microsoft.com). Perhaps you have a proxy setting incorrect? Or a firewall in the way? We ultimately determined we had enough control over the server and did not want to 'call home', so we simply disabled the check. You can do this with .NET 2.0 SP1 and later by adding the following to the machine.config.
<runtime> <generatePublisherEvidence enabled="false"/> </runtime>
I am not sure if you can just put this in your app.config/web.config.
IL is being converted into machine native code (Assembly) by the Just-In-Time compiler and you get to wait while all the magic happens.
When compiling the source code to
managed code, the compiler translates
the source into Microsoft intermediate
language (MSIL). This is a
CPU-independent set of instructions
that can efficiently be converted to
native code. Microsoft intermediate
language (MSIL) is a translation used
as the output of a number of
compilers. It is the input to a
just-in-time (JIT) compiler. The
Common Language Runtime includes a JIT
compiler for the conversion of MSIL to
native code.
Before Microsoft Intermediate Language
(MSIL) can be executed it, must be
converted by the .NET Framework
just-in-time (JIT) compiler to native
code. This is CPU-specific code that
runs on the same computer architecture
as the JIT compiler. Rather than using
time and memory to convert all of the
MSIL in a portable executable (PE)
file to native code. It converts the
MSIL as needed whilst executing, then
caches the resulting native code so
its accessible for any subsequent
calls.
source
Thats the compilation of asp.Net pages into intermediate language + JIT compilation - it only happens the first time the page is loaded. (See http://msdn.microsoft.com/en-us/library/ms366723.aspx)
If it really bothers you then you can stop it from happening by pre-compiling your site.
EDIT: Just re-read the question - 60 seconds is very long, and you would expect to see some processor activity during that time. Check the EventLog for errors / messages in the System and Application destinations. Also try creating a crash dump of the w3wp process during this 60 seconds - there is an chance you might recognise what its doing by looking at some of the call stacks.
If it takes exactly 60 seconds each time then its likely that its waiting for something to time out - 60 seconds is a nice round number. Make sure that it has proper connections to the domain controllers etc...
(If there are some IIS diagnostic tools that would do a better job then I'm afraid I'm not aware of them, this question might be more suited to ServerFault, the above is a much more developer-ish approach to troubleshooting :-p)
I found that there was a network delay making an initial connection from the front end web server to the database server.
The issue was peculiar to Windows 2008 and our specific network hardware.
The resolution was to disable the following on the web servers:
Chimney offload state
Receive window auto-tuning level
Greater than 60 seconds sounds fishy. Try running a test.html page to see how long that takes. That will isolate IIS7's role.
Then temporarily rename your web.config, global.asax and application folders and try a test.aspx page (very simple page). That will isolate ASP.NET.
If both of those are fast (i.e. about 10 seconds), then it's your application. But, if either are slow then not the application and something with the server itself.
This hat nothing to do with JIT compiling. The normal C# compiler compiles your code behind files (.aspx.cs) into intermediate language into an assembly at startup if this assembly dont exist or code files have changed. Your web site assembly is located in the "bin" folder of your web site.
In fact the JIT compiling occures after that, but this is very fast and won't take several minutes. JIT Compiling happens on every startup of an .net application and that won't take more than a view seconds.
You can avoid the copmpiling of your web site if you deploy the already compiled website assembly (YourWebsite.dll) into the bin folder. It is also possible to deploy only the aspx files and leave the code behind files (aspx.cs) files away.
I've just been battling a similar issue. For me it turned out to be that I had enabled internal logging for NLog. It added about 3 minutes to the startup time!
Original config
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true"
throwExceptions="false" throwConfigExceptions="false"
internalLogLevel="Debug"
internalLogFile="C:\Temp\NLog.Internal.txt">
Fixed Config
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true"
throwExceptions="false" throwConfigExceptions="false">
For Info I discovered this by using SysInternals' ProcMon.exe, filtering on the Process Name "w3wp.exe"
During our build process we run aspnet_compiler.exe against our websites to make sure that all the late-bound stuff in ASP.NET/MVC actually builds (I know nothing about ASP.NET but am assured this is necessary to prevent finding the failures at runtime).
Our sites are fairly large in size, with a few hundred pages/views/controls/etc. however the time taken seems excessive in the 10-15 minute range (for reference, this is longer than it takes the entire solution with approx 40 projects to compile, and we're only pre-compiling two website projects).
I doubt that hardware is the issue as I'm running on the latest Quad core Intel chip, with 4GB RAM and a WD Velociraptor 10,000rpm hard disk. And part of what's odd is that the EXE doesn't seem to be using much CPU (1-5%) and doesn't seem to be doing an awful lot of I/O either.
So... is this a known issue? Why is it so slow? And is there any way to speed it up?
Note: To clarify a couple of things people have answered about, I am not talking about the compilation of code within Visual Studio. We're using web application projects already, and the speed of compilation of those is not the issue. The problem is the pre-compilation of the site after these projects have already been compiled (see this MSDN page for more details) as part of the dev build script. We are performing in-place pre-compilation, not copying the files to a target directory.
Switching to Roslyn compiler most likely will significantly improve precompile time. Here is a good article about it: https://devblogs.microsoft.com/aspnet/enabling-the-net-compiler-platform-roslyn-in-asp-net-applications/.
In addition to this, make sure that batch compilation is enabled by setting batch attribute to true on the compilation element.
Simply, the aspnet_compiler uses what is effectively a "global compiler lock" whenever it starts pre-compiling any individual aspx page; it is basically only allowed to compile each page sequentially.
There are reasons for this (although I personally disagree with them) - primarily, in order to detect and prevent circular references causing an infinite loop of sorts, as well as ensuring that all dependencies are properly built before the requiring page is compiled, they avoid a lot of "nasty CS issues".
I once started writing a massively-forked version of aspnet_compiler.exe last time I worked at a web company, but got tied up with "real work" and never finished it. Biggest problem is the ASPX pages: the MVC/Razor stuff you can parallelize the HELL out of, but the ASPX parse/compile engine is about 20 levels deep of internal and private classes/methods.
Compiler should generate second code-behind file for every .aspx page, check
During compilation, aspnet_compiler.exe will copy ALL of the web site files to the output directory, including css, js and images.
You'll get better compilation times using Web application project instead of Web site model.
I don't have any specific hot tips for this compiler, but when I have this sort of problem, I run ProcMon to see what the process is doing on the machine, and I run Wireshark to check that it isn't spending ages timing-out some network access to a long-forgotten machine which is referenced in some registry key or environment variable.
Just my 2 cents.
One of the things slowing down ASP.NET views precompilation significantly is the -fixednames command line option for aspnet_compiler.exe. Do not use it especially if you're on Razor/MVC.
When publishing the wep app from Visual Studio make sure you select "Do not merge", and do not select "create separate assembly" cause this is what causes the global lock and slows things down.
More info here https://msdn.microsoft.com/en-us/library/hh475319(v=vs.110).aspx