We are just starting to deploy our hosted web application in an Azure environment. We have the database running as an Azure SQL Database and the web application running as an Azure App Service Web App (including a continuous WebJob). We are having some significant performance issues with the Web App and I am hoping to get some advice on how to resolve these performance issues.
Here are some key points:
The Web App is deployed as a compiled web application (binaries, markup, *.resx files), not source code
The Web App is deployed via FTP
The Web App has "Always On" turned On
The App Service is running in the South Central US Azure datacenter
The Azure SQL Database is also running in the South Central US Azure datacenter (so database latency should be minimal)
We have an exact replica of the Web App (same binaries, etc.) running on a non-Azure server in a datacenter in Los Angeles (if database latency were the issue, this site would have worse database latency)
Both instances of the Web App (Azure and non-Azure) are connecting to the very same Azure SQL Database (running in the South Central US Azure datacenter)
I have been testing from a workstation in Utah – not particularly close to either web server
On any “cold” request (site has been idle for a period of time), the Azure site seems to be significantly slower. Pages that load quickly from the non-Azure server take a long time (e.g. 20 seconds or longer) from the Azure server. The content is the same, and both apps are accessing the same database – so I don’t believe it is a database issue. I have narrowed it down to two .NET functions that seem to be the problem:
System.Web.UI.TemplateControl.GetLocalResourceObject(string)
System.Web.UI.TemplateControl.LoadControl(string)
The application includes *.resx files for a few different languages. When a request comes in, it uses the culture in the request header to load the correct resources. The GetLocalResourceObject function seems to be compiling the *.resx files on the fly, which seems to take several seconds on the Azure server. I may need to try something like this to prevent Azure from compiling *.resx files on the fly.
Some of our pages load controls dynamically, hence the call to the LoadControl method. These controls are deployed as *.ascx markup, but the code-behind is already compiled into a *.dll in the .\bin folder. Even though the controls are already compiled, it appears that the LoadControl function is trying to compile each control, which takes several seconds on the Azure server.
A “warm” page access seems to go more quickly on the Azure server (almost as quickly as the non-Azure server), probably because the resources and controls are already compiled and still in memory. I have yet to be able to determine at what point the resources/controls have to be recompiled.
I am trying to figure out the best way to make the site respond well, even on a “cold” request. I have tried running aspnet_compiler.exe on the Azure App Service server (via Kudu CMD). That seems to work sometimes, but it has not been very reliable (sometimes it doesn’t seem to work, or the process doesn’t seem to complete). It also seems that, even after compiling the site, the problem returns after a period of idleness.
Our primary objective is to get the Azure App Service Web App to perform acceptably. It seems that the only way to do that is to force the Web App to precompile the resources and user controls so they are not compiled on demand. I have yet to find a reliable way to accomplish this objective. Any help or insight would be appreciated.
Related
Is it possible to publish an ASP.NET (core) application to a running site on IIS 8 without having to stop and start the website manually?
Visual Studio 2015 keeps giving the error that a file is in use. I'm using publish to the file system, because my Web Deploy to a server in our network fails at the end with an error that it cannot authenticate on the server with port 443.
I don't mind IIS having to recycle the application pool, but when I constantly have to put the app down, publish (which takes about a minute) and restart it, it's not really good for the users.
Another option would be something like 2 websites running the same application, but only if this is automatable. Then it would put down 1 of the 2, update this, put it on, put down the second, update this and start it.
A third option is something like a hot update, where I could just update the application while it being on.
Can anyone point me in the right direction (perhaps some blog posts), because my Google searches didn't give me any good information?
If you copy a file called app_offline.htm to the application folder IIS will gracefully stop your application and start serving the contents of the app_offline.htm file. When application is stopped you can copy your files. After files are copied remove the app_offline.htm file and IIS will start your app. VS does that for you when you deploy to Azure but not when deploying to file system.
You mention "two websites running the same application". Do you mean two web servers hosting the same app?
If you already have multiple web servers (a.k.a. cluster of servers, or web farm), you can simply take some servers out of the cluster and update them. That is how we push out our updates. We use software from Citrix for managing the server farm. It also handles load balancing. This type of software allows one to monitor the servers, so you can determine when all the users have "moved" off the web servers (that were recently taken out of cluster). Then you can iisreset, deploy the new build, and move on to the next server (or set of servers, depending on your configuration). We have more than 20 virtual web servers. Typically we take down half of the servers, update them, and take the other half down as we put the first half back in the cluster/farm. This should allow for uninterrupted service. I understand you want to automate the process. I'll assume you're using Windows NLB (network load balancing). You could write a PowerShell script to automate taking down the servers. Here's a reference:
https://technet.microsoft.com/en-us/library/ee817138
I understand this may be viewed as a workaround. I'm not sure if recycling the app pool is always necessary with ASP.NET Core. I wasn't able to find a definitive answer. Most production applications should be on more than one web server anyway. Even if you don't have a lot of users, you should have multiple web servers for failover purposes.
We have DLLs that contain hundreds of custom client processes that are kicked off from an ASP.NET application. Our clients run these processes while performing data entry, and typically there's only 1 process per client. On any given day, we might update 2 or 3 of these processes.
Currently these are all housed in a series of DLLs, which means that we are publishing our application a couple times per day. As a result, any logged-in clients get booted out of the system since the publish causes an app restart.
Is there a way that we can update these DLLs without requiring a full publish each time?
If your client processes have a common API then you could host, the DLLs separately in a WCF (or similar) service, and call the client processes remotely. So basically, consider moving to a service oriented architecture.
Check out the Managed Extensibility Framework (MEF) from Microsoft. It provides not only dependency management but also plug-in like library loading. Most likely it's exactly what you're looking for.
You can switch to SQL server or state server sessions in order to perserve session and logged users after app restart. Or store these dlls in APP_DATA and load it dinamically. Then of course you have to think of some refreshing system and refresh loaded dlls with newly uploaded ones.
There is no sensible way to avoid an application restart. Please note the emphasis on the word sensible.
We are using IIS7 to host an asp.net web-based application.
In this environment administrators and developers can deploy code to the application on a regular basis.
The new code or app goes as a DLL to the ASP.NET bin folder. Upon deployment of the new DLL, IIS restarts the process, impacting (slowing down) all online users.
Is there a way to configure IIS to run the process in the background and once ready make the switch from old state into new without impacting the users?!
Thanks in advance for your feedback!
IIS already does this, that's what recycling is all about. IT's loading the DLL's while the old version of the application is still running. only after this is completed the recycling is complete.
However loading the DLL's is only part of getting web applications ready, there might also be initial loads like loading/caching the user db etc.
These actions are not part of the recycle process, they happen after all DLL's reloaded and the recycling is already completed.
A while back I ran into this issue with an application that had a huge startup time due to heavy db activity/caching during startup. So I was interested if there is some functionality that allows us to execute code before the recycle is marked as completed, so that the application is first considered recycled when everything is ready to run. Basically what I wanted is some kind of staging functionality.
I was in contact with the IIS team regarding this issue, sadly they told me that no such functionality exists, nor is it planned.
To solve this you could try do the following:
Use alternating deploys:
You setup 2 Websites with separate application pools. One of them is the LIVE website the other one is the STAGED website. If you want to deploy changed you simply deploy to the STAGED website. After everything is loaded/cached etc. you switch the URL settings of the web applications to reroute incoming requests from the LIVE to the STAGED one. So the LIVE one becomes the new STAGED and the other way around. The next deploy would then go to the new STAGED again and so on.
UPDATE
Apparently they have created a IIS Module that provides this functionality by now:
IIS Application Warm-Up Module for IIS 7.5
The IIS team has released the first beta test version of the
Application Warm-Up Module for IIS 7.5. This makes warming up your
applications even easier than previously described. Instead of writing
custom code, you specify the URLs of resources to execute before the
Web application accepts requests from the network. This warm-up occurs
during startup of the IIS service (if you configured the IIS
application pool as AlwaysRunning) and when an IIS worker process
recycles. During recycle, the old IIS worker process continues to
execute requests until the newly spawned worker process is fully
warmed up, so that applications experience no interruptions or other
issues due to unprimed caches. Note that this module works with any
version of ASP.NET, starting with version 2.0.
For more information, see Application Warm-Up on the IIS.net Web site.
For a walkthrough that illustrates how to use the warm-up feature, see
Getting Started with the IIS 7.5 Application Warm-Up Module on the
IIS.net Web site.
See:
http://www.asp.net/whitepapers/aspnet4
If you use ASP.NET 4 Auto Start feature:
You can still choose to auto-recycle the worker processes from time to
time. When you do that, though, the app will immediately restart and
your warm up code will execute (unlike today - where you have to wait
for the next request to-do that).
The main difference between Warm Up and Auto Start feature is that the Warm Up Module is part of the recycling process. Rather than blocking the application for requests, while running the init code.
Only thing you get by using the Auto Start feature is that you don't have to wait for a user to hit the page, which does not help your case.
See the Gu's blog post:
http://weblogs.asp.net/scottgu/archive/2009/09/15/auto-start-asp-net-applications-vs-2010-and-net-4-0-series.aspx
UPDATE 2:
Sadly the Warmup Module has been discontinued for IIS 7/7.5:
http://forums.iis.net/t/1176740.aspx
It will be part of IIS8 though (It's now called Application Initialization Module):
http://weblogs.asp.net/owscott/archive/2012/03/01/what-s-new-in-iis-8.aspx
UPDATE 3:
As pointed out in the comments the Warmup Module resurfaced for IIS 7.5 as Application Initialization Module for IIS 7.5 after IIS 8 was released:
http://www.iis.net/downloads/microsoft/application-initialization
The first part of ntziolis answer is a wee bit inaccurate. The worker process isn't being recycled or restarted, it just keeps running. If this were the case, then in shared pool environments you would have sites knocked out every time a new one was deployed.
When you deploy a new ASP.NET application it's the site's "Application Domain" within the worker process is torn down, not the pool process.
In addition pool recycling is a completely separate concept to deployment
At this point in time in the commercial life of ASP.NET, during a deployment, a site will be in an inconsistent state until all of the site is deployed. There is still no good story about this from Microsoft at this time for single site on a single server deployments.
This is why ASP.NET has the special App_Offline.htm page. It's there so you can enable that page, deploy and then turn it off.
The second part of ntziolis answer is nearly correct but you don't need two sites or two application pools. You just need two file system folders that switch between being the physical folders for the site...if you're on a single server and not behind a load balancer or ARR.
If your sites were on a web server behind a load-balancer or ARR then having two different sites would make sense, you could route requests from one site to the other and round-robin on each deploy.
Obviously if there is a large amount of user generated content (uploaded files and the like) then you'd map a virtual directory in your site to a common location for this data.
In larger scale deployments where your app is running across (for example) a load-balanced environment you can do more sophisticated deployments.
For related questions please see:
How Do I deploy an application to IIS while that web application is running
Publishing/uploading new DLL to IIS: website goes down whilst uploading
Is smooth deployment possible with componentized ASP.NET MVC apps?
I'm a developer now developing my startup. I really don't know much about IIS setup. I will host my startup on Amazon EC2. And I want to know how can I scale my application if my traffic increase. I been reading about MS Deploy and Web Farm Framework here: https://serverfault.com/questions/127409/iis-configuration-synchronization-for-web-server-farm . And I want a simple architecture, with not to much configuration. So I been looking an experience with an IIS web farm and Amazon ELBs. And I did not find any one.
So the question is:
It is possible to make a IIS web farm with Amazon ELBs?
Any experience on Ec2? IIS web deploy or WFF and/or without ELBs?
What you recommend for an easy web farm setup?
You can do almost anything you want with IIS on EC2. They are full servers (well window 2k8 datacenter edition) and you can open any ports you need to communicate between servers. Here is an explicit tutorial on how to set up WFF, for example, on EC2.
The question is, are you sure you need to build a web farm? If you simply want to have multiple servers running your code then you can accomplish this without anything more than IIS and the tools that EC2 provides.
You build your app so it uses shared resources (like a session state server, central location for storing user uploaded content), configure a server the way you like it, and capture a server image (AMI). You use this image when you configure AutoScaling to launch new instances based on server metrics (like CPU usage), and they would be automatically added to the load balancer when launched.
The last challenge is ensuring servers launched automatically are running your latest code. You can write a custom program to get the latest code from somewhere (like SVN) on server startup, or you can use something much simpler like Dropbox to handle the synchronization.
I currently work with a legacy asp.net web application and one of the requirements going forward is that it be deployable to windows azure.
I would like to know how difficult it will be to manage deployment to both Azure and a traditional IIS web server.
Azure seems to require a specific customized version of a web applicaiton project is it possible to deploy the customized web application to a standard IIS instance once it has been converted.
EDIT:
It is a ASP.NET Web Application rather than a Web Site (compiles everything into one dll)
UPDATE:
In the end due to the amount of work involved in converting the application to work in Azure and the cost of Azure compared with other cloud solutions it was decided to go with a traditional Cloud hosted virtual server.
And thank you for the really good answers.
Whether or not you can deploy your application to Azure almost as is depends a lot on how your application works. Azure pretty much requires your application be stateless. If it's a plain vanilla web application that stores data in the session or application cache only and saves data to a database only, then you can deploy it to Azure.
If you have stateful services running like background threads (which is bad anyways), or if you save data to the file system (besides temporary caching), then you may have issues. Really, the issues moving to Azure are really the same as moving to any multi-server load balanced solution. One caveat is permanent storage.
If you need to store data in a place other than the database, then you're best off working with Azure's storage solution which has an API and client library for storing binary data, key/value data (they call it tables, but really, it's not tables), and queues. They also do have a transparent blob-as-file-system option for compatibility. If you want to use these in your app that also is used outside of Azure then you need to write an extra layer between your code and the Azure client library that supports both Azure services and standard local service. Azure SDK does include emulators for Azure services, but they're definitely not meant for production use.
As far as the mechanics of Azure-specific projects, that is actually not that difficult. Yes, you need to create an Azure-specific project in your solution that defines the Web Role and what gets deployed, but it will reference your existing Web Application, not the other way around. You can deploy the Azure Web Role to Azure or you can continue to deploy the existing application to IIS normally and concurrently.
Web Site, Web Application, MVC, really doesn't make much of a difference. Actually doesn't have to be .NET either. Can be PHP or Java or whatever you want to put on your VM. It'll all work the same as far as Azure is concerned.
MS likes to push Azure as a Platform-as-a-Service (Paas) solution where they have a ton of services they offer and you run apps on their standard platform, and contrasts that with Amazon AWS which they call Infrastructure-as-a-Service (Iaas) which is "just" a Virtual Machine. However, MS is really just as much a IaaS solution as AWS, perhaps even more so. The only difference between AWS and Azure is AWS allows you to choose what to install on your VM and with Azure you have to use Windows Server 2008 R2 as the basis for your VM (but you can customize the VM image to install custom software on top of windows). With both Azure and AWS, the hosts offer additional PaaS services you can take advantage of for data storage and message routing. AWS also offers tons of extra services like video streaming.
Also note that with Azure (and AWS I think) you can use the services they offer even in a non-hosted application. If you want to use Azure's data storage from a non-Azure application, you can do that, it's just HTTP REST calls to get/put data. The only differences you pay for data in/out between datacenter and your non-datacenter-hosted application which would be free if the app was also inside the datacenter (just the data in/out is free in-datacenter, you still have storage and transaction fees).
A few things:
Samuel Neff's answer mentioned mounting a file system in a blob (a Cloud Drive). Only one instance may lock this cloud drive for writing, so it does not behave like a network file share. You'll need to plan for this.
You'll need to integrate with the Windows Azure diagnostics subsystem, to gain visibility into your app's run state (e.g. performance counters, trace logs, etc.).
If there are 3rd-party apps that your web app depends on, you'll need to install these. These actually get installed as part of the role instance's boot process, either via your OnStart() event handler or as a startup task. The latter allows for admin-level installs (including registry changes, COM component installations, etc.). You'll need to carefully manage these installations, as they impact the boot time of the instance.
For an asp.net app, you'll need to think about session state. In-proc session state won't work, because each instance will have its own state store in memory. The SQL Azure session state provider doesn't have background cleanup agents, so you'll need to build this into your web or worker role instance (see this blog post by the SQL Azure team for the implementation). The best option is to use the AppFabric Cache, a new service that just went into production. This cache-as-a-service provides an custom session state provider for asp.net as well. Note: As of today, the AppFabric Cache service is only accessible via a .NET interface; there's no REST interface for it (all other storage services - tables, blobs, queues - have a REST interface). .NET, Java, and PHP all have storage client libraries. Ruby has one from the open source community.
You'll have to manage scaling out to more than one instance, when the need arises. This is not a built-in service today, but there are 3rd-party services such as ParaLeap's AzureWatch. There's also Microsoft's System Center Operations Manager, which now has Windows Azure monitoring support. You'll also need to handle scale-back situations, where you reduce the number of server instances.
I have some additional details in an answer for a similar StackOverflow question, here.
I have not tried Windows Azure Migration Scanner personally, but if it works as advertised, this would really come in handy.