We have a Sitecore 6.6 instance which is used to host multiple sites. It is hosted in IIS 7.5. We developed custom Sitecore sublayouts and pipelines which are used across websites.
When any dll is deployed in bin folder, the Sitecore site takes long time to startup (8-10 mins). But when IIS is reset, startup time is less (30-40 seconds).
What could be the reason for application startup time to be more for DLL deployment than IIS Reset ?
Any suggestions to improve the application startup time for DLL deployment ?
Update 1: The startup time after DLL deployment impacts our build process as it increases the overall build deployment time in all environments (DEV,STG,LIVE).
Profiling snapshot of w3wp process revealed two major hotspots:
Sitecore.Threading.Semaphore.P
Sitecore.IO.FileWatcher.Worker
Update 2: After following the deployment suggested by Vicent, profiling snapshot of w3wp process revealed major hotspot at
Sitecore.Web.UI.WebControls.Sublayout.GetUserControl(Page)
Further analysis of memory dump showed that thread was waiting for JIT compilation of newly deployed DLL.
For me, it's like your problem is not the start up of sitecore, but the shutdown.
When you copy your dll, the filewatcher detects the change in the bin folder (write it to the logs) and tries to shutdown sitecore (log this too), but if sitecore has tasks running on different threads, (indexing, publishing, scheduled tasks, etc), the semaphore will wait until the other threads finish normally.
That's why when you "kill" the process without waiting for threads to finish sitecore starts up quickly.
I've this behaviour in my environments too, so when i need a quick restart, i copy the dll, wait for a few seconds so at least sitecore tries to shutdown and then and i kill the w3p.exe related to my pool. I won't advice nobody to do this, but i don't have any way to "kindly" kill the threads... Maybe somebody knows how to "force shutdown nicely..."
This blog post by Alex Shyba has some interesting pointers to improve the startup time for Sitecore (but might not be applicable if you're talking about a live environment rather than a DEV environment).
It might also be worth checking your prefetch caches and running through the Performance Tuning Guide if you haven't done that yet.
I've seen this problem before. It happened on version 6.5, but I've not seen a fix for it in the release notes since then.
Sitecore Support has a hotfix for this - it was indeed related to something with their Filesystem watcher tasks. You would need to raise a ticket with them, to get the hotfix or additional information.
My support ticket reference for this issue is 370593. The hotfix has issue 323775. If you mention this in your support ticket, it should speed up the process a bit - if it is indeed the same issue you're experiencing.
Related
I am working on a .NET WebForms application and I have observed that whenever I build, after the build, the very first page load takes longer to load than usually. This happens even if I wait after building before I load a page. Is there a way to increase human workforce performance by changing IIS/.NET to initialize things on postbuild instead of first page load?
Yes you can, like this.
Quoting:
You can use Application Initialization Module which comes in-box with IIS 8.0, like this:
<applicationInitialization
doAppInitAfterRestart="true" >
<add initializationPage="/" />
</applicationInitialization>
This will send a request to the root of your app (initializationPage="/") every time your app starts automatically.
You can also configure the Start Mode for your application pool to Always Running which means every time IIS restarts, it will make sure to start your application pool immediately (this if from right click on your application pool then Advanced Settings).
Professional servers have hardly any latency, though it requires quite a bit of tweaking. Also, by default, applications recycle regularly on IIS (as well as when some kinds of exceptions occur, when some files are changed, or when some thresholds are reached). Professional web application hosting is anything but simple :) You might get help with that on Server Fault, perhaps.
Another option is to avoid mixing pre-compilation and JIT-compilation - if you only pre-compile, you don't need to do any compilation when the application is deployed, resulting in faster startup times. If you only deploy sources, the application domain doesn't need to be torn down when you make a change, which means that only the change needs to be recompiled, which is much faster.
And of course, ASP.NET Core is much, much faster in both scenarios - it can do the whole compilation in-memory, unlike the legacy system which uses csc to build multiple assemblies, save them to disk, load them from disk, merge them together, save that again, just to load it again and initialize.
I have an asp.net mvc4 application that we deploy to about 1400 clients. Our current deployment process goes like this:
:: Publish the project files to a local directory
%msbuild% RPortal.csproj "/p:Platform=AnyCPU;Configuration=Release;ConfigurationName=Release;SolutionDir=%solutionDir%;PublishDestination=.\Deploy\Release" /t:PublishToFileSystem
And then we have some powershell scripts that push the new build to each of our clients, using other tools to help with the sync.
We are noticing some sever slowness with application boot up times (sometimes upwards of 5 minutes), and one of the approaches we've investigated to solve this would be the idea of ASP.Net Precompiling (http://msdn.microsoft.com/en-us/library/vstudio/bb398860%28v=vs.100%29.aspx)
In some experiments with the idea, it appears that calling aspnet_compile.exe on our published system does, in fact, create some new files. (A couple of dlls, and a few *.compiled files).
My questions are as follows:
How does this compilation differ from the primary compilation from the .cs source
files?
Does this compilation happen regardless, at first run, or only when run manually?
(related to 2), Does this compilation survive app pool restarts and server reboots?
With our current scenario, it seems that we are killing our servers trying to get 1400 applications to start up (yes, they all live on 1 webserver... that's not a situation I am in control of). The server will be humming along, with no particular problems, resonable resource consumption, and then, all of a sudden, our CPU will peg to 100% and stay there. The only factor we can tie it back to is that it happens when more than 5 (or so) of the 1400 apps are all booting up.
Our hope is that precompiling will front load most of the app start burden, but I clearly don't understand whats really going on here.
Any light you might shed would be most appreciated.
We have an application which, on startup, populates it's cache from an csv. This takes about 1 minute. We thought putting an application warm up on it would eliviate the problem, but it seems like the requests specified are not being executed. Hitting the application locally will take a minute and then be instant thereafter. I tried several requests, relative (to localhost and application root) and absolute ones, to no avail. It does not work after stop/starting the corresponding app pool, nor recyclying (if there is a difference to application warm up), nor rebuilding the application.
Anyone have suggestions I could try?
Edit: The event viewer does not have anything related to the application warm up (as far as I can tell) and Application_start() is not called
if you're not seeing warmup requests in your logs perhaps you don't have them enabled in your application settings? What language are you using? (Warmup requests are enabled by default for all Java applications.)
I see entries for /_ah/warmup in my logs.
I deployed an ASP.NET web application last night and I when I woke up this morning it was very slow and would occasionally just throw a 'Service Unavailable' error.
I checked the Event Viewer and it was filled up with these errors:
An unhandled exception occurred and the process was terminated.
Exception: System.Runtime.Serialization.SerializationException
Message: Unable to find assembly 'MonoTorrent, Version=0.80.0.0, Culture=neutral, PublicKeyToken=null'
I'm puzzled as it was working perfectly when I deployed it (MonoTorrent is required to retrieve the number of seeders/leechers for a certain torrent off the tracker - this was working fine), but it's no longer working and whenever code that uses MonoTorrent gets involved, the worker process just crashes.
MonoTorrent.dll is in the /bin/ directory.
UPDATE 6/4/10: I compiled the MonoTorrent source code in with the rest of my web application, but it still crashes whenever it uses MonoTorrent. However, it now says that it is Unable to find assembly 'OpenPeer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null. Here, OpenPeer is the name of the web application's assembly.
This can happen in these circumstances:
The ASP.NET app creates a background thread, which throws an uncaught exception. It looks like ASP.NET catches the exception and wants to log it to the Event Log. To do this, it sends this exception from the Web app's app domain to its own app domain (the default one of the w3wp process). This needs a serialization/deserialization of the exception.
If the exception is a custom one (i.e. defined by the Web app), it cannot be deserialized in the main app domain of ASP.NET because the assembly defining the exception is typically in the Web app's bin directory, not where w3wp.exe is (c:\windows\system32\inetsrv). This causes a serialization exception and w3wp crashes.
There are possible ways to fix the issue (in a - very subjective - order of preference):
Copy the missing DLL in c:\windows\system32\inetsrv
Install the missing DLL in the GAC
Remove the cause of the exception (harder to do than to say, as we say in French)
Catch all exceptions from the background thread yourself and do the logging yourself.
Notes:
If WCF is used and the uncaught exception is FaultException, WCF swallows it and there is no crash
If the uncaught exception is in the thread of the Web request, there is a yellow screen of death, not this serialization exception
It really seems like a bug in ASP.NET
The above is actually a summary of my investigations of this issue yesterday and are only a theory. I tested fixes 1 and 4, as well as using FaultException.
Here are some things you can try..
1.) Flush ASP.Net Temp directory. Restart IIS and recycle Application pool.
2.) Make sure your web-application is running in FULL-TRUST if it really needs FULL-TRUST.
3.) Take the Assembly, try to use it in other asp.net application and run the test application on a seperate server. This might help you diagnose the problem. Also try to run the test asp.net app on the same server but in seperate application pool.
4.) Make sure the IIS website of your application is running under the user account with necessary security priviliges. Try running the application under Administratotr as user.
EDIT-1
5.) Also check if the assembly version is the same as mentioned in web.config. If there's a version mismatch then you can do AssemblyBinding Redirection in web.config.
6.) Also try registaering the Assembly in GAC and see if it loads properly.
EDIT-2
7.) Try reconfigring ASP.NET support on the server or maybe framework runtime re-setup may help. This may not be a sure-shot solution but looking at the problem condition we may want to try various solutions.
8.) Make sure you're not missing any critical update of your windows server platform.
I try to give you some ideas - what I do if I was on your position.
First of all I take a long look of the MonoTorrent.dll before some days that you make your question, and I look it again today. I found and the function that load the dll. My first opinion is that something have to do with the permissions.
I hope that you have access to the server - right ?
My first steps is that:
Ensure that your monotorrent.dll actuall have the right permissions to the bin directory, for Read, and execute by your asp.net app. Some times the copy of one dll, did not get the directory permissions buts carriage out his own permissions. To check if your dll have different permissions from the rest, just right click and see Properties | Security, then go to bin directory and do the same, and compare the Security permissions. If they are different then apply again the Directory permissions and make sure that the dll inherited by the directory.
My second step
Download the ProcessMonitor from sysinternals
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
Run ProcessMonitor and try to recreate the error, stop it and analyse to see where and why the dll gets the denied permissions to run.
With the ProcessMonitor you can even see if there is any dll that can not found !
I have check the MonoTorrent dlls and I do not found anything unusual. He have kerner32.dll calls, and use unsafe code to run, ok nothing so special about.
So if you do that 2 steps and give me some feedback, maybe I can go further. (if not solve by you and what you find)
I would advice to setup Regular maintanence probably once in a week at sunday night etc for following,
Delete all temporary files
Delete all ASP.NET IIS temporary files
Restart Server
Problem is, ASP.NET web apps cause lot of temp files to be left in the disk, because of dynamic compilation of regex, seriliazation assemblies etc, such temp stuff never gets deleted, and more and more junk starts getting collected in temp locations, ASP.NET goes slower and slower, and a point comes in where disk as well as memory defragmentation reaches very high point, things start to fail.
No body likes to restart server once a week, but I remember we had no choice, in ASP.NET 1.1 we had stable system after restarting every day, in ASP.NET 2.0 onwards, we are good to have restarting scheduled at once a week.
I have found this problem and I have do all of thing as I can, such as clear temp file, restart server, delete and add reference and I also rebuild the solution. However I can't solving this problem. Finally I move my entity class (almost of them need to serialize) to new folder that I have added to the project and then this problem solved.
This method is work for me.
Try clearing the ASP.NET temp files. It's solved some odd issues before for me.
Otherwise, Fusion-logging may shed some light.
UPDATE: #Charlie - I'm not sure what to make of those logs...it looks like the failed log is from a different AppDomain. Notice the AppBase is set to "file:///c:/windows/system32/inetsrv/" and AppName is w3wp.exe.
I'm pretty sure the Event Viewer should show Application Id: LM/W3SVC/#/ROOT if it was the default AppDomain, too. At this point, all I've got is random guesses.
I notice you're running x64...does MonoTorrent perhaps require x86?
Have you double checked that the directory is an IIS application, and is configured for the correct version of ASP.NET?
Is there some other application that uses MonoTorrent on this server? Maybe a WCF service or something? I'm not sure where the Serialization is happening....
Try hooking the AssemblyResolve event and loading it manually.
Can you repro on a development machine? If not, maybe it's a borked FX install. Uninstall and reinstall.
Does restarting, recycling or stopping/starting the AppPool fix the issue temporarily, or cause the issue to appear?
You may want to type out your screenshot text too so you'll get some Google love....
Is the server timezone different than your timezone? I've had this issue when deploying resource files, the compile time was in the future so they would fail to load.
My guess that you have plenty of open but not closed connections. I mean the connections are not returned to the pool. It looks okay, when you start the application, but after some time there are only several sockets available in the pool and it goes slow. Another thing - non-closed connection may keep DLL in memory, not allowing to release the handler. Try to debug object destruction.
I know it's simple but I had this problem once and itwas because I had a Web Application project which contains
References
Folder and I just copied my files into a
Bin
folder, in any .net web application in the Project Properties windows, a Reference Path tab is available which by default should nothing be include on it. check this option and also Build tab in Project Properties window which Output path be as the same as bin\
I have an ASP.NET 3.5 website running under IIS7 on Windows 2008.
When I restart IIS (iisreset), then hit a page, the initial startup is really slow.
I see the following activity in Process Explorer:
w3wp.exe spawns, but shows 0% CPU
activity for about 60 seconds
Finally, w3wp.exe goes to 50% CPU for
about 5 seconds and then the page
loads.
I don't see any other processes using CPU during this time either. It basically just hangs.
What's going on during all that time? How can I track down what is taking all this time?
We had a similar problem and it turned out to be Windows timing out checking for the revocation of signing certificates. Check to see if your server is trying to call out somewhere (e.g. crl.microsoft.com). Perhaps you have a proxy setting incorrect? Or a firewall in the way? We ultimately determined we had enough control over the server and did not want to 'call home', so we simply disabled the check. You can do this with .NET 2.0 SP1 and later by adding the following to the machine.config.
<runtime> <generatePublisherEvidence enabled="false"/> </runtime>
I am not sure if you can just put this in your app.config/web.config.
IL is being converted into machine native code (Assembly) by the Just-In-Time compiler and you get to wait while all the magic happens.
When compiling the source code to
managed code, the compiler translates
the source into Microsoft intermediate
language (MSIL). This is a
CPU-independent set of instructions
that can efficiently be converted to
native code. Microsoft intermediate
language (MSIL) is a translation used
as the output of a number of
compilers. It is the input to a
just-in-time (JIT) compiler. The
Common Language Runtime includes a JIT
compiler for the conversion of MSIL to
native code.
Before Microsoft Intermediate Language
(MSIL) can be executed it, must be
converted by the .NET Framework
just-in-time (JIT) compiler to native
code. This is CPU-specific code that
runs on the same computer architecture
as the JIT compiler. Rather than using
time and memory to convert all of the
MSIL in a portable executable (PE)
file to native code. It converts the
MSIL as needed whilst executing, then
caches the resulting native code so
its accessible for any subsequent
calls.
source
Thats the compilation of asp.Net pages into intermediate language + JIT compilation - it only happens the first time the page is loaded. (See http://msdn.microsoft.com/en-us/library/ms366723.aspx)
If it really bothers you then you can stop it from happening by pre-compiling your site.
EDIT: Just re-read the question - 60 seconds is very long, and you would expect to see some processor activity during that time. Check the EventLog for errors / messages in the System and Application destinations. Also try creating a crash dump of the w3wp process during this 60 seconds - there is an chance you might recognise what its doing by looking at some of the call stacks.
If it takes exactly 60 seconds each time then its likely that its waiting for something to time out - 60 seconds is a nice round number. Make sure that it has proper connections to the domain controllers etc...
(If there are some IIS diagnostic tools that would do a better job then I'm afraid I'm not aware of them, this question might be more suited to ServerFault, the above is a much more developer-ish approach to troubleshooting :-p)
I found that there was a network delay making an initial connection from the front end web server to the database server.
The issue was peculiar to Windows 2008 and our specific network hardware.
The resolution was to disable the following on the web servers:
Chimney offload state
Receive window auto-tuning level
Greater than 60 seconds sounds fishy. Try running a test.html page to see how long that takes. That will isolate IIS7's role.
Then temporarily rename your web.config, global.asax and application folders and try a test.aspx page (very simple page). That will isolate ASP.NET.
If both of those are fast (i.e. about 10 seconds), then it's your application. But, if either are slow then not the application and something with the server itself.
This hat nothing to do with JIT compiling. The normal C# compiler compiles your code behind files (.aspx.cs) into intermediate language into an assembly at startup if this assembly dont exist or code files have changed. Your web site assembly is located in the "bin" folder of your web site.
In fact the JIT compiling occures after that, but this is very fast and won't take several minutes. JIT Compiling happens on every startup of an .net application and that won't take more than a view seconds.
You can avoid the copmpiling of your web site if you deploy the already compiled website assembly (YourWebsite.dll) into the bin folder. It is also possible to deploy only the aspx files and leave the code behind files (aspx.cs) files away.
I've just been battling a similar issue. For me it turned out to be that I had enabled internal logging for NLog. It added about 3 minutes to the startup time!
Original config
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true"
throwExceptions="false" throwConfigExceptions="false"
internalLogLevel="Debug"
internalLogFile="C:\Temp\NLog.Internal.txt">
Fixed Config
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true"
throwExceptions="false" throwConfigExceptions="false">
For Info I discovered this by using SysInternals' ProcMon.exe, filtering on the Process Name "w3wp.exe"