We have 4 servers load balanced:
4 cores # 2.6Ghz (E5-2650 v2)
14GB RAM
Windows 2012 R2
High Performance power setting
IIS 8.5
ASP 5.3
EF 6.1
They each have a single application pool with one worker process and a single website. Each server has its own copy of the site (DLLs & views), running on a local disk. We are using IIS virtual directories to point to shares on a clustered file server for log files and common images etc (content only). The application pools are set to not shut down when idle (interval of 0) and we have also disabled the every-1740 minute recycle interval too.
We have New Relic's .NET agent installed on all servers, and looking through our slow transaction log, I can see that many requests are taking 15 seconds or so to complete. Looking into the trace, I can see a common call to System.Web.Compilation.AssemblyBuilder.Compile() and System.Web.Compilation.BuildManager.CompileWebFile().
As far as I know or understand, ASP would compile these views upon first request to them, and cache it (to the temporary ASP files in C:\Windows\Microsoft.Net) and then load from there for subsequent requests.
I'm confused how this is happening so often - when I visit these URLs, the TTFB is about 400ms, and due to constant load I can't see the websites "losing" their cache and needing to compile the views again. These pages are frequently hit - it's an e-commerce store and I can see that it happens often, and on our most popular pages: catalogue (category/brand/gender etc) listings and product details.
I've set the settings against each application pool to log an event when recycling, and there have been no events logged when I'm checking the WAS service in the event viewer. We also have New Relic server installed, and looking over the past 6 hours' worth of data, I can't see any dip in RAM usage on any of the servers - which would indicate the application pool recycling. This has really baffled me!
I'm thinking of moving towards pre-compiling our views as part of our release process - it makes sense really. But it feels like that is working around, or masking an issue which as far as I can see should not be happening. We build our site in Release mode and have <compilation debug="false" /> on all web.config files.
Can anyone think of any causes for this?
It is because of how JIT (Just-In-Time) compilation works.
When you build your application, it is converted into .NET Microsoft Intermediate Language (MSIL) or Intermediate Language (IL).
As your applications is accessed, Common Language Runtime (CLR) converts only executed IL parts of your code into native instructions.
Just-In-Time compilation process converts IL to native machine instructions and it is a part of CLR.
In a simplified terms when you run a .NET application and your program calls a method. JIT Compiler reads IL from metadata and compiles it into native instructions and run it. Next when your program calls the same method, CLR executes native CPU instructions directly. This process adds some overhead for the first method call. You can go with the other option of pre-compiling your application using NGEN, which is usually not recommended because you will loose some optimizations that only JIT can perform due to its awareness of underlying hardware platform. These two articles has more details
http://geekswithblogs.net/ilich/archive/2013/07/09/.net-compilation-part-1.-just-in-time-compiler.aspx and https://msdn.microsoft.com/en-us/library/ms366723.aspx
There are also other things you can try that might help you speed up your application. You can use IIS Application warm up module How to warm up an ASP.NET MVC application on IIS 7.5?, implement distributed caching etc to alleviate some of your application bottlenecks.
Related
After building my asp.net solutions, and then run it for the first time in the browser, the time it takes to load increases. Then running it thereafter the time has decreased until the next build.
Can someone explain why this happens?
When websites are deployed to IIS, the assemblies that make up the site still need to be natively compiled. On first access, the CLR 'interprets' the MSIL and converts it into a binary version specific to the CPU - e.g. x86, x64, Itanium etc.
This also occurs with .EXE's and any outputs from the .NET compiler.
To address this issue with websites taking a while to start up, you can configure pre-compilation to be turned on. See this Microsoft article for more information on the details.
From your comment on
And, from the URL being entered in the browser, how does the .NET compiler know that it needs to convert the MSIL to binary?
IIS stores a list of websites it maintains, and knows what applications service the requests received at those endpoints. For applications written in .NET, IIS will pass the request information to an ASP.NET pipeline, which will take the information provided by IIS and place the information into various .NET objects like HttpContext, Session etc.
From there, the ASP.NET pipeline works out which part of the application responds to that request and executes the code for it.
We are using IIS7 to host an asp.net web-based application.
In this environment administrators and developers can deploy code to the application on a regular basis.
The new code or app goes as a DLL to the ASP.NET bin folder. Upon deployment of the new DLL, IIS restarts the process, impacting (slowing down) all online users.
Is there a way to configure IIS to run the process in the background and once ready make the switch from old state into new without impacting the users?!
Thanks in advance for your feedback!
IIS already does this, that's what recycling is all about. IT's loading the DLL's while the old version of the application is still running. only after this is completed the recycling is complete.
However loading the DLL's is only part of getting web applications ready, there might also be initial loads like loading/caching the user db etc.
These actions are not part of the recycle process, they happen after all DLL's reloaded and the recycling is already completed.
A while back I ran into this issue with an application that had a huge startup time due to heavy db activity/caching during startup. So I was interested if there is some functionality that allows us to execute code before the recycle is marked as completed, so that the application is first considered recycled when everything is ready to run. Basically what I wanted is some kind of staging functionality.
I was in contact with the IIS team regarding this issue, sadly they told me that no such functionality exists, nor is it planned.
To solve this you could try do the following:
Use alternating deploys:
You setup 2 Websites with separate application pools. One of them is the LIVE website the other one is the STAGED website. If you want to deploy changed you simply deploy to the STAGED website. After everything is loaded/cached etc. you switch the URL settings of the web applications to reroute incoming requests from the LIVE to the STAGED one. So the LIVE one becomes the new STAGED and the other way around. The next deploy would then go to the new STAGED again and so on.
UPDATE
Apparently they have created a IIS Module that provides this functionality by now:
IIS Application Warm-Up Module for IIS 7.5
The IIS team has released the first beta test version of the
Application Warm-Up Module for IIS 7.5. This makes warming up your
applications even easier than previously described. Instead of writing
custom code, you specify the URLs of resources to execute before the
Web application accepts requests from the network. This warm-up occurs
during startup of the IIS service (if you configured the IIS
application pool as AlwaysRunning) and when an IIS worker process
recycles. During recycle, the old IIS worker process continues to
execute requests until the newly spawned worker process is fully
warmed up, so that applications experience no interruptions or other
issues due to unprimed caches. Note that this module works with any
version of ASP.NET, starting with version 2.0.
For more information, see Application Warm-Up on the IIS.net Web site.
For a walkthrough that illustrates how to use the warm-up feature, see
Getting Started with the IIS 7.5 Application Warm-Up Module on the
IIS.net Web site.
See:
http://www.asp.net/whitepapers/aspnet4
If you use ASP.NET 4 Auto Start feature:
You can still choose to auto-recycle the worker processes from time to
time. When you do that, though, the app will immediately restart and
your warm up code will execute (unlike today - where you have to wait
for the next request to-do that).
The main difference between Warm Up and Auto Start feature is that the Warm Up Module is part of the recycling process. Rather than blocking the application for requests, while running the init code.
Only thing you get by using the Auto Start feature is that you don't have to wait for a user to hit the page, which does not help your case.
See the Gu's blog post:
http://weblogs.asp.net/scottgu/archive/2009/09/15/auto-start-asp-net-applications-vs-2010-and-net-4-0-series.aspx
UPDATE 2:
Sadly the Warmup Module has been discontinued for IIS 7/7.5:
http://forums.iis.net/t/1176740.aspx
It will be part of IIS8 though (It's now called Application Initialization Module):
http://weblogs.asp.net/owscott/archive/2012/03/01/what-s-new-in-iis-8.aspx
UPDATE 3:
As pointed out in the comments the Warmup Module resurfaced for IIS 7.5 as Application Initialization Module for IIS 7.5 after IIS 8 was released:
http://www.iis.net/downloads/microsoft/application-initialization
The first part of ntziolis answer is a wee bit inaccurate. The worker process isn't being recycled or restarted, it just keeps running. If this were the case, then in shared pool environments you would have sites knocked out every time a new one was deployed.
When you deploy a new ASP.NET application it's the site's "Application Domain" within the worker process is torn down, not the pool process.
In addition pool recycling is a completely separate concept to deployment
At this point in time in the commercial life of ASP.NET, during a deployment, a site will be in an inconsistent state until all of the site is deployed. There is still no good story about this from Microsoft at this time for single site on a single server deployments.
This is why ASP.NET has the special App_Offline.htm page. It's there so you can enable that page, deploy and then turn it off.
The second part of ntziolis answer is nearly correct but you don't need two sites or two application pools. You just need two file system folders that switch between being the physical folders for the site...if you're on a single server and not behind a load balancer or ARR.
If your sites were on a web server behind a load-balancer or ARR then having two different sites would make sense, you could route requests from one site to the other and round-robin on each deploy.
Obviously if there is a large amount of user generated content (uploaded files and the like) then you'd map a virtual directory in your site to a common location for this data.
In larger scale deployments where your app is running across (for example) a load-balanced environment you can do more sophisticated deployments.
For related questions please see:
How Do I deploy an application to IIS while that web application is running
Publishing/uploading new DLL to IIS: website goes down whilst uploading
Is smooth deployment possible with componentized ASP.NET MVC apps?
We would like to warm up an ASP.NET MVC application hosted on IIS 7.5 server. The warm up module that used to be available at http://forums.iis.net/t/1176740.aspx has been removed since sometime.
The application should be warmed up everytime IIS or ASP.NET worker-process restarts for any reason. During the warm up period, IIS should return some HTTP status code signifying its warm up state or its inability to serve any clients.
Would creating a executable that navigates through necessary pages in the site via HttpRequests be a good idea? The executable can be triggered from IProcessHostPreloadClient implementation. Is it possible to configure IIS so that it would only accept requests from localhost and once the executable is done, it can switch over to all clients - but that switch should not trigger an IIS restart (obviously).
Is it possible to use an Visual Studio 2010 - Web Performance Test to warm-up an application instead of creating an manual executable? Any other alternatives?
PS: The application uses Forms Authentication and uses sessions - so maintaining state cookie and other cookies is important.
UPDATE 1 - We are using .NET Framework 4.0 and Entity Framework (database first) in our application. The first time hits to EF queries are slow. The reason behind the warm up is to get these first time hits out of the way. We are already using compiled queries at most places and we have implemented pre-compiled views for EF. The size of the model and application is very large and complex. Warm up needs to walk through many pages to ensure that compiled and non-compiled EF queries get executed at-least once before any end user gets access to the application.
Microsoft has released a module that does exactly what you ask for. The Application Initialization Module for IIS 7.5 improves the responsiveness of Web sites by loading the Web applications before the first request arrives.
You can specify a series of Urls that IIS will preload before accepting requests from real users. I don't think you can get a true user login expereince, but maybe you can set up simulated pages that does not require login that fulfills the same warmup you ask for?
The feature I think is most compelling is that this module also enables overlapped process recycling. The following tutorial from IIS 8.0 include a step-by-step approach on how to enable overlapped process recycling.
When IIS detects that an active worker process is being recycled, IIS does not switch active traffic over to the new recycled worker process until the new worker process finishes running all application initialization Urls in the new process. This ensures that customers browsing your website don't see application initialization pages once an application is live and running.
This IIS Application Initialization module is built into IIS 8.0, but is available for download for IIS 7.5.
You may take a look at the following post for the Auto-Start feature built into IIS 7.5 and ASP.NET 4.0.
Any application that generates a server request for the hosted resources can be used to warm up an IIS process. Exactly how many requests you need depends on what parts need warming up. Typically, warm-up is used for:
Starting up a worker process. For this, you only need to ask for one resource to warm up a process for the entire application.
Perform any static initialization, database startup, or pre-caching. Anything you do in your Global.asax file will happen when you do your first request, so if you can make all of your initialization happen then, you'll still only need to make one page request.
Force pre-compilation of ASP.NET pages. For this to happen you would need to hit every page. Fortunately, this is typically not much of a time cost, so you likely don't need to worry about it. If you do have individual pages that load slowly, you can warm them up separately.
The "warm-up" process here isn't anything magical. You just need force IIS to serve the URL in question. Everything you mentioned would take care of that: using a stress-test tool to query the URL, writing a custom utility to post HTTP requests, even just scripting out a tool like 'wget' or a PowerShell script to download the URLs would do it.
As far as restricting access to localhost, as far as I know, within IIS, the only way to change that requires you to restart IIS. You could always build a pre-request hook into your application and maintain the state there, and have your warm-up process query some specific URL that toggles that state to "open". But I'm not sure what you would accomplish. If, somehow, a user did try to query your site before your warm-up finished, all that would happen is your site would take a long time to respond, then they would eventually get the page they asked for. If you locked them out of the site during warm-up, they would instead get a browser network error that claimed the site was offline, which (to me) sounds much worse.
Backstory
Last month our development team created a new asp.net 3.5 application to place out on our production website. Once we had the work completed, we requested from the group that manages are server to copy the app out to our production site, and configure the virtual directory as a new application.
On 12/27/2010, two public 'Gineau Pigs' were selected to use the app, and it worked great.
On 12/30/2010, We received notification by internal staff, that when that staff member tried to access the application (this was the Business Process Owner) they recieved the 'Server Application Unavailable' message.
When I called the group that does our server support, I was told that it probably failed, because I didn't close the connections in my code. However, the same group went in and then created a separate app pool for this Extension Request application. It has had no issues since.
I did a little googling, since I do not like being blamed for things. I found that the 'Server Application Unavailable' message will also appear when you have multiple applications using different frameworks and you do not put them in different application pools.
Technical Details - Tree of our website structure
Main Website <-- ASP Classic
+-Virtual Directory(ExtensionRequest) <-- ASP 3.5
From our server support group:
'Reviewed server logs and website setup in IIS. Had to reset the application pool as it was not working properly. This corrected the website and it is now back online. We went ahead and created a application pool for the extension web so it is isolated from the main site pool. In the past we have seen other application do this when there is a connection being left open and the pool fills up. Would recommend reviewing site code to make sure no connections are being left open.'
The Real Question:
What really caused the failure? Isn't the connection being left open issue an ASP Classic issue? Wouldn't the ExtensionRequest application have to be used (more than twice) in the first place to have the connections left open? Is it more likely the failure is caused by them not bothering to setup the new Application in it's own App Pool in the first place?
Sorry for the long windedness
You'd really need to obtain and review the server's Application & System event and HTTPERR logs for the period the server was reporting these errors.
Without these it'd be hard speculate what was the root cause of the problem.
Update:
OP incorrectly tagged his question so this next section no longer applies. However I'll leave in place because I think the information is useful for those encountering these issues and perhaps thinking about migrating to IIS7.x.
You are correct that running two different .NET Framework's in the same application pool can cause these errors but that's something you'd tend to see on Windows 2003/IIS6, not Windows 2008/IIS7.
IIS7 uses a slightly different approach to specifying which .NET Framework version is loaded and it's determined by the Application pool's managedRunTimeVersion property. When requests are processed by IIS/ASP.NET the site's Handler Mapping's use a preCondition attribute to determine when to load the requisite handler (which is kind of like a script mapping in previous versions of IIS).
This mechanism prevents the incorrect runtime version being loaded into the application pool's worker process.
So if an application pool is configured to run .NET Framework version v4.0 only that version will load, even if your application is built against v2.0.
There's a great article on how this works here:
Achtung! IIS7 Preconditions
The section on Handlers about half way through explains why the dangers of accidentally loading the wrong .NET version into a pool are mitigated by the preCondition feature.
A Server Application Unavailable error usually means something catastrophic has happened (like loading the wrong ASP.NET version's ISAPI filter into an already running worker process).
Not closing SQL connections is unlikely to cause this type of serious error. You'd more than likely be seeing a yellow screen of death runtime errors if that were the case. Running out of SQL connections usually doesn't bend ASP.NET so out of shape that the whole service tops itself.
My prime suspect would be a permissions problem where the application pool identity was unable to correctly access the application folders. But it's just a hunch.
Again, what you need to do is get the Application & System event logs and the HTTPERR logs (they reside in %systemroot%\System32\LogFiles\HTTPERR. That will contain clues and facts about what went wrong.
Update 2:
On Windows 2003/IIS6, if you have two applications running different ASP.NET versions that reside in the same pool you will get this error. In my experience (I work for a web hoster) it is the primary cause of this infamous error page:
There's also a tell-tale event logged to the Application Event log:
Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1062
Date: 12/01/2011
Time: 12:31:43
User: N/A
Computer: KK-DEBUG
Description:
It is not possible to run two different versions of ASP.NET in the same
IIS process. Please use the IIS Administration Tool to reconfigure your
server to run the application in a separate process.
Whilst your root application may not be written in ASP.NET it's likely that something has triggered loading of a different version of the framework into your site's application pool.
there's a rogue web.config in the root...this will trigger ASP.NET to load
there's a wildcard mapping to ASP.NET 1.1 in the site script maps (less likely, but possible)
I'm inclined to think that your new application most certainly ended up in a pool where other sites or applications were running a different framework version. The only way to really find out is to obtain the Application event logs and look for the event shown above.
It's hard to tell; there could be many causes (too many resources used, calling outside of .NET caused something to crash, etc). I would look in the Event log and see if you can find something there.
If you're running different versions of .NET you definitely want separate pools. If you have the option, I would recommend separate pools for each application (even if in the same .NET version).
As far as "closing the connection" (I assume you mean the connection to the database). If you're creating "low level" connections (i.e. SqlConnection, SqlCommand) then make sure you're wrapping them in a "using" statement, otherwise your connection pool can fill up. In my experience though, you should receive regular .NET errors in this case. If you're using an ORM this shouldn't be an issue.
Edit:
If you can't find anything useful in the Event Log, you could try this: http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis-7/
Working with one of our partners, we have developed now two separate sets of web services for their use. The first one was a simple "post to an https URL" style web service, which we facilitated by building a web page in ASP.NET that inspected the arguments in the URL, and then acted accordingly. This "web service" (if you can call it that) has been very stable.
At some point, the partner asked us to begin using SOAP based web services. At their request, we built them a new set of web services largely based on the previous objects, reimplemented as an actual "Web Service". This web service has not been very stable: around once a week, Nagios will alert us that our web service is not responding - and a quick iisreset does the trick.
Analyzing the log output and working in a debugger has not led us to anything concrete. The volume on this new web service is actually much lower than the HTTP web service. I think this could be a code problem or a platform problem, or of course something in between.
We've tried, with little improvement:
To duplicate the behavior in the lab
Debugging in the Visual Studio debugger
Tinkering with IIS options to give it its own application pool
My question, what are the next steps for troubleshooting?
Environment:
Windows Server 2003 Standard Edition R2 Service Pack 2 32 bit, Visual Studio 2005, MS SQL 2005, .NET Framework 2.0.50727
You may get some answers by profiling your webservices and understanding how they are using their resources. perfmon and procmon are both very useful tools in this regard.
EDIT: Since you say errors happen after about a week, the only thing I can think of is resource usage. Ensure your DB connections are being cleaned up, and any opened files (system call to the exe) are being closed.
Also, if your webservices can tolerate it, IIS has a setting that triggers a periodic recycle of an App Pool to handle cases where performance degrades over time. Its dirty, but it may work well for your case.
Since there isn't much to go on - here's another odd issue we came up against regarding our web services.
When the web service stops responding how is memory utilization? We have experienced issues with memory and memory fragmentation relating to busy web services on a system (there was also other things running causing additional fragmentation). When we re-factored the web services to load from smaller dll's and depend on other libraries (instead of one large library) we were able to resolve the memory fragmentation.
To identify what was occurring we would take a dump from the offending iis worker process where the app pool resided and then reviewed that using WinDbg.
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
Additionally we used DebugDiag to take the postmortem dumps.
http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1286
Hope this provides another direction to look at.