I have an asp.net web application running on an IIS 7.5/.NET Framework 4.0 server. Whenever I navigate to any webpage running on this server, it takes about 12 seconds to initially load. After that, navigation is quick (about 0.5 seconds, even on heavy hitting pages). If you leave it idle for two minutes, it slows again for the next request. I can tell that this is because the connection time-out is set to 120 seconds...I am guessing that after that limit, the site must reload everything when the next page is requesting. This site does call two different databases, however, default page should not do so and suffers the long initial load time. I have tried setting up Application Initialization for IIS 7.5, but noticed very marginal change at best after this was done. From what I've been reading, there seems to be very mixed success with this module in IIS 7.5. Is there any other means of circumventing this load time without having to rely solely on a high connection timeout value, since that would not resolve initial load time anyway?
When a WebApp is idle for along time IIS will close the application to save resources. This might have happened in your case.
Its also said that the application would turn off if the last user session timed out. I hope this article will guide you properly.
Look at what happens when the request gets to the runtime.
When ASP.NET receives the first request for any resource in an
application, a class named ApplicationManager creates an application
domain. (Application domains provide isolation between applications
for global variables, and allow each application to be unloaded
separately.)
Within an application domain, an instance of the class named Hosting
Environment is created, which provides access to information about
the application such as the name of the folder where the application
is stored.
After the application domain has been created and the Hosting
Environment object instantiated, ASP.NET creates and initializes
core objects such as HttpContext, HttpRequest, and HttpResponse.
After all core application objects have been initialized, the
application is started by creating an instance of the
HttpApplication class.
If the application has a Global.asax file, ASP.NET instead creates
an instance of the Global.asax class that is derived from the
HttpApplication class and uses the derived class to represent the
application.
See How it happens
Was not related the connection timeout as I thought it was, but rather another timeout and necessary files missing. Per Zerkey's question in the comments above, I got a little curious and looked around for ways to see what was loading, as debugging it from my PC was still slow, but considerably faster (about 4-6 seconds). In IIS on the server this is published to, I went to Worker Processes, selected the process and clicked current requests on the right. This showed me that it gets hung up on a 3rd party mobile redirection service I am using called 51degrees.mobi. It was taking about 10 of those 12 seconds for those file to load. What was happening is that the logging capabilities were set to log in an App_Data folder, and that directory was missing. It evidently wasn't giving me a visible error, it was just trying it and failing. Once I added this directory and log file, and reactivated Application Initialization, everything is quick.
Related
I am having some performance issues with my iis webserver. It is hanging randomly and I am trying to figure out how to speed up the server. I enabled Failed request tracing on the server and set it to generate a log when the request is over 3 seconds.
The resulting logs(xml) dont show much but there is a point in the compact performance log that indicates what part of the log the server is hanging on. Below is the part of the log where the large time loss is occurring.
65. i GENERAL_GET_URL_METADATA PhysicalPath="", AccessPerms="513" 17:46:32.577
66. i HANDLER_CHANGED OldHandlerName="", NewHandlerName="ExtensionlessUrlHandler-Integrated-4.0", NewHandlerModules="ManagedPipelineHandler", NewHandlerScriptProcessor="", NewHandlerType="System.Web.Handlers.TransferRequestHandler" 17:46:32.577
67. i VIRTUAL_MODULE_UNRESOLVED Name="FormsAuthentication", Type="System.Web.Security.FormsAuthenticationModule" 17:46:47.771
I am not sure what Handler changed is but it is taking a long time, any tips would be great on where to start looking.
It is hard to come up with a solution without having any piece of code in sight. Here are some general hints/tips you can follow in order to have great performances with an ASP.NET application.
The fastest way to do a request is to not do it in the first place. Try caching everything that can be cached. There are server-side caches and client-side caches. Each have their own uses, but you are not limited to only one type.
Make sure you do not cache and/or keep references of any request-related objects into memory. ASP.NET have a limited number of concurrent requests and keeping a request reference in memory will hang your server if it runs out of threads
Close the request as soon as you are done with it
Everything that is not needed by the client at the time of the request should be done in the background
Make sure you have no memory leak in your application. Garbage Collections are often the cause of hangs in ASP.NET application. When garbage collecting, all running threads are paused. This is especially true for Gen 2 garbage collections. You can enable background generation 2 garbage collections.
Isolate the problematic code. Use a profiler and see which type of request is CPU-intensive. Then dig deeper and see what inside that request makes it slow.
In any well-balanced application, objects should either be short-lived and live forever. In the case of an ASP.NET application, the objects created during the course of a request should ideally die within that request or during the next GC gen 0.
Consider object pooling for large objects and objects that are long to initialize
Make sure your app pool doesn't totally crash and restarts (look the IIS logs and/or the Windows Events)
Some useful debugging tools you can use:
LeanSentry. Great for diagnosing ASP.NET server hangs
windbg. High learning curve but by far the most powerful debugging tool you can use
PerfView. Useful for analyzing ETW events like I/O or CPU usage
There are many ways to improve server performance. But before that you should start with checking CPU usage during the "hang". An infinite loop in the application code may cause this behavior. Unless there is I/O, locking, or sleeps in the loop, you will be able to see it from the CPU usage as you will get exactly one full core's worth of CPU usage for each infinite loop.
Help link to improve server performance
More Info:
I can see entry related to VIRTUAL MODULE UNRESOLVED: which is related to bad use of Response.Redirect(url); Also make sure you have deployed your app on integrated mode on IIS.
here's a simple checklist you might want to reconsider:
Always pre-compiling your site, as opposed to copying it! you might gain a significant performance boost compiling your website before deployment: ASP.NET Precompilation Overview
Do not run the production application with debug="true" enabled, when debug flag is true in your web.config, Much more memory is used within the application at runtime, and since some additional debug paths are enabled, codes can execute much slower
Check your Web.config file to ensure trace is disabled in the section
IIS 7.5 comes with the Auto-Start Feature. WAS (Windows Process Activation Service) starts all the application pools that are configured to start automatically, ensure that your application pool is configured to AlwaysRunning in the IIS 7.5 applicationHost.config, check out here for more detail.
Every asp.net server can be well configured by aspnet.config file located in the root of the framework folder. Ensure that Publisher Evidence for Code Access Security (CAS) is set to false in your aspnet.config file, This might increase the initial page load when you restart the ASP.NET app pool. you can read more about it here.
Also you might want to try Application Initialization Module for IIS 7.5, this module also available on IIS 8.0 can decrease the response time for first requests by pre-loading worker processes
IIS has an annoying feature for low traffic websites where it recycles unused worker processes, causing the first user to the site after some time to get an extremely long delay (30+ seconds).
I've been looking for a solution to the problem and I've found these potential solutions.
A. Use the Application Initialization plugin
B. Use Auto-Start with .NET 4
C. Disable the idle-timeout (under IIS Reset)
D. Precompile the site
I'm wondering which of these is preferred, and more importantly, why are there so many solutions to the same problem? (My guess is they aren't, and I'm just not understanding something correctly).
Edit
Performing C seems to be enough to keep my site warmed up, but I've discovered that the real root of my site's slowness has to do with Entity Framework, which I can't seem to figure out why it's going cold. See this question, which unfortunately hasn't been answered yet has been answered!
I eventually just had to make a warm up script to hit my site occasionally to make sure it stayed speedy.
Options A, B and D seem to be in the same category since they only influence the initial start time, they do warmup of the website like compilation and loading of libraries in memory.
Using C, setting the idle timeout, should be enough so that subsequent requests to the server are served fast (restarting the app pool takes quite some time - in the order of seconds).
As far as I know, the timeout exists to save memory that other websites running in parallel on that machine might need. The price being that one time slow load time.
Besides the fact that the app pool gets shutdown in case of user inactivity, the app pool will also recycle by default every 1740 minutes (29 hours).
From technet:
Internet Information Services (IIS) application pools can be
periodically recycled to avoid unstable states that can lead to
application crashes, hangs, or memory leaks.
As long as app pool recycling is left on, it should be enough.
But if you really want top notch performance for most components, you should also use something like the Application Initialization Module you mentioned.
Web Hosting Challenge
You have to remember that none of the machine configuration options are available if you are hosted on a shared server as many of us (smaller companies and individuals) are.
ASP.NET MVC Overhead
My site takes at least 30 seconds when it hasn't been hit in over 20 minutes (and the web app has been stopped). It is terrible.
Another Way to Test Performance
There's another way to test if it is your ASP.NET MVC start up or something else. Drop a normal HTML page on your site where you can hit it directly.
If the problem is related to ASP.NET MVC start up then the HTML page will render almost immediately even when the web app hasn't been started.
That's how I first recognized that the problem was in the ASP.NET MVC startup.
I loaded an HTML page at any time and it would load blazing fast. Then, after hitting that HTML page I'd hit one of my ASP.NET MVC URLs and I'd get the Chrome message "Waiting for raddev.us..."
Another Test With Helpful Script
After that I wrote a LINQPad (check out http://linqpad.net for more) script that would hit my web site every 8 minutes (less than the time for the app to unload -- which should be 20 minutes) and I let it run for hours.
While the script was running I hit my web site and every time my site came up blazingly fast. This gives me a good idea that most likely the slowness I was experiencing was because of ASP.NET MVC startup times.
Get LinqPad and you can run the following script -- just change the URL to your own and let it run and you can test this easily.
Good luck.
NOTE: In LinqPad you'll need to press F4 and add a reference to System.Net to add the library which will retrieve your page.
ALSO : make sure you change the String URL variable to point at a URL that will load a route from your ASP.NET MVC site so the engine will run.
System.Timers.Timer webKeepAlive = new System.Timers.Timer();
Int64 counter = 0;
void Main()
{
webKeepAlive.Interval = 5000;
webKeepAlive.Elapsed += WebKeepAlive_Elapsed;
webKeepAlive.Start();
}
private void WebKeepAlive_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
webKeepAlive.Stop();
try
{
// ONLY the first time it retrieves the content it will print the string
String finalHtml = GetWebContent();
if (counter < 1)
{
Console.WriteLine(finalHtml);
}
counter++;
}
finally
{
webKeepAlive.Interval = 480000; // every 8 minutes
webKeepAlive.Start();
}
}
public String GetWebContent()
{
try
{
String URL = "http://YOURURL.COM";
WebRequest request = WebRequest.Create(URL);
WebResponse response = request.GetResponse();
Stream data = response.GetResponseStream();
string html = String.Empty;
using (StreamReader sr = new StreamReader(data))
{
html = sr.ReadToEnd();
}
Console.WriteLine (String.Format("{0} : success",DateTime.Now));
return html;
}
catch (Exception ex)
{
Console.WriteLine (String.Format("{0} -- GetWebContent() : {1}",DateTime.Now,ex.Message));
return "fail";
}
}
Writing a ping service/script to hit your idle website is rather a best way to go because you will have a complete control. Other options that you have mentioned would be available if you have leased a dedicated hosting box.
In a shared hosting space, warmup scripts are the best first level defense (self help is the best help). Here is an article which shares an idea on how to do it from your own web application.
I'd use B because that in conjunction with worker process recycling means there'd only be a delay while it's recycling. This avoids the delay normally associated with initialization in response to the first request after idle. You also get to keep the benefits of recycling.
A good option to ping the site on a schedule is to use Microsoft Flow, which is free for up to 750 "runs" per month. It is very easy to create a Flow that hits your site every hour to keep it warm. You can even work around their limit of 750 by creating a single flow with delays separating multiple hits of your site.
https://flow.microsoft.com
See this article for tips on how to help performance issues. This includes both performance issues related to starting up, under the "cold start" section. Most of this will matter no matter what type of server you are using, locally or in production.
https://blogs.msdn.microsoft.com/b/mcsuksoldev/2011/01/19/common-performance-issues-on-asp-net-web-sites/
If the application deserializes anything from XML (and that includes web services…) make sure SGEN is run against all binaries involved in deseriaization and place the resulting DLLs in the Global Assembly Cache (GAC). This precompiles all the serialization objects used by the assemblies SGEN was run against and caches them in the resulting DLL. This can give huge time savings on the first deserialization (loading) of config files from disk and initial calls to web services.
http://msdn.microsoft.com/en-us/library/bk3w6240(VS.80).aspx
If any IIS servers do not have outgoing access to the internet, turn off Certificate Revocation List (CRL) checking for Authenticode binaries by adding generatePublisherEvidence=”false” into machine.config. Otherwise every worker processes can hang for over 20 seconds during start-up while it times out trying to connect to the internet to obtain a CRL list.
http://blogs.msdn.com/amolravande/archive/2008/07/20/startup-performance-disable-the-generatepublisherevidence-property.aspx
http://msdn.microsoft.com/en-us/library/bb629393.aspx
Consider using NGEN on all assemblies. However without careful use this doesn’t give much of a performance gain. This is because the base load addresses of all the binaries that are loaded by each process must be carefully set at build time to not overlap. If the binaries have to be rebased when they are loaded because of address clashes, almost all the performance gains of using NGEN will be lost.
http://msdn.microsoft.com/en-us/magazine/cc163610.aspx
I was getting a consistent 15 second delay on the first request after 4 minutes of inactivity. My problem was that my app was using Windows Integrated Authentication to SQL Server and the service profile was in a different domain than the server. This caused a cross-domain authentication from IIS to SQL upon app initialization - and this was the real source of my delay. I changed to using a SQL login instead of windows authentication. The delay was immediately gone. I still have all the app initialization settings in place to help improve performance but they may have not been needed at all in my case.
Environment: Windows Server 2003; IIS 6, ASP.NET 2.0.50727
I'm going crazy with a brand new web server that we set up (note that this problem doesn't happen on our other web servers which have the same configuration). When loading and asp.net app the first time, the page hangs for over a full minute before showing the page in the browser. After it loads the first page, everything runs very quickly.
Note 1: You will probably say that the application is being compiled for the first time. But I've ruled that out. I put trace messages EVERYWHERE in the app and all the trace messages run within a second of requesting the page. Thus, the app compiles and runs immediately. But when the app is finished rendering the page and my last trace message is printed, nothing happens. IIS is doing something behind the scenes for a full minute before transferring the finished page along http to the user's browser.
Note 2: We found that after hitting the app the first time and things run fine, if we wait an hour then we get the delay again. Thus, IIS has something in its cache that it clears out after an hour and causes our site to stall again.
Note 3: Between each test we stop/start IIS to force it to hang upon loading the app.
Note 4: We watched the Task Manager to see if IIS was spiking and taking up a lot of resources processing something. But that wasn't it. We did see a very quick spike to 50% immediately before the browser showed the page, but for the previous 60 seconds there was only 1% usage on the server.
Note 5: On another test I created a HelloWorld.html page and this does not cause IIS to hang. Thus, it has something to do with calling the ASP.NET library the very first time it sends a rendered page across http. Also, since the app has already been compiled and runs instantly, it's just the part of asp.net that sends the rendered page to the user's browser that causes the delay.
Any ideas? We are a a loss here. All of our other web servers are setup the same way and work fine, but this is a new install. So there must be a configuration setting that was missed or maybe something needs to be installed?
Thanks,
Brian
If you have access to the servers, then make sure that app pool recycling is actually logged to the event logs
cscript adsutil.vbs get w3svc/AppPools/DefaultAppPool/LogEventOnRecycle
you can set it to log everything with
cscript adsutil.vbs Set w3svc/AppPools/DefaultAppPool/LogEventOnRecycle 255
See more here
Then check if there were any recycles.
App initialization, creation the worker process, threads, load the app domain and all the references dll's can take some time, that's normal, but that 1 minute delay is something else probably.
Try to precompile the app on the server and see if that helps
aspnet_compiler -m /LM/W3SVC/[site id ]/Root/[your appname]
If you want to dig deeper, you can check the event trace ETW.
logman query providers
Save the IIS /ASP.NET related Guids to a file like iisproviders.txt
logman start ExampleTrace -pf iisproviders.txt -ets -rt
reproduce
LogParser "SELECT * FROM ExampleTrace" -i:ETW
logman stop ExampleTrace -ets
You can find more hereTroubleshooting appdomain restarts and other issues with ETW tracing
I would also check the w3wp.exe with procexp if it has a TCP connection time out or with Procmon for other clues.
If you have experience with windbg, then you can make a request to the app then quickly attach the debugger to the process
windbg -p [process id of the app pool]
.loadby sos mscorwks
g
and take it from there. If there are exceptions, process crash, etc you should be able to catch it...
Once we had a weird server issue like this and a .NET reinstall solved the problem, still not sure what was the culprit.
Could be some aspnet.config settings on this box that are different from others. Have you tried copying over their config files to this server? There appears to be certificate options along with registry modifications that you can do to remove some lag time during the initial load of a page (precompiling aside)
See here and here
One thing you might want to check on is if there are any database access going on on your page load. That might be blocking the creation of the page during initial page load. Then when the query is cached (either by the db engine or another cache mechanism like memcached), subsequent page loads work as normal.
As per your last comment,
I could stop/start IIS multiple times and the app always ran instantly. I thought it was fixed for good. But now I just tried again (it has been sitting idle for the past couple of hours) and now it is back to hanging on the first request.
This could mean that the cache has expired and thus needs to hit the database once again, causing the delay in page load.
I have an ASP.NET Web Service (SOAP style) that is running in our production environment.
Our server guys have set things up such that things like starting and stopping of Windows services, etc., are sent via email to the appropriate parties.
Lately my boss has been getting emails about my ASP.NET web service:
The (My Web Service's Name) Application_Start method was called
Now I figure that what's happening here is that the service has gone so long since being called last that the server has unloaded it from memory and now it's being re-loaded again (the product that consumes this web service has declined in popularity, so this isn't too far fetched a theory).
However my boss tells me he's been getting this email "dozens" of times per day.
I suppose it's still possible that my theory above is accurate, especially given how it's spread out over 3-4 servers in our web tier, but is there any other explanation for why this might be happening so frequently?
At this point in time I don't know whether or not Application_End calls are being similarly emailed or not, or what the ratio is.
The application could be downloaded when some of the settings in <processModel> are exceeded. idleTimeout could be the one in your case, but also requestLimit and memoryLimit.
Also, and this is based on a true story, if you start any thread, run anything in a separate threadpool thread, or use the TPL, make sure that you catch any exception that might be thrown. Uncaught exceptions from those threads will kill the worker process. Check the application logs in the Windows event log. If this is the case, you should see the red icon application error signs around the same time that the emails go out.
I'm trying to implement some fail safes on a client's web server which is running two of their most important sites (ASP.NET on IIS7). I'm going to set up application pool limiting so that if any w3wp process uses 90%+ CPU for longer than a minute then it gets killed (producing a temporary 503 Service Unavailable message to any visitors), and based on my local testing will be restarted within a minute - a much better solution than having one CPU-hogging process taking down the whole server for any length of time.
This seems to work, however during my fiddling on my local IIS7 instance I've noticed that if a request calls my "Kill.aspx", even when the site comes back up IIS will not serve the session that caused it to hang. I can only restart the test site from a different session - but as soon as I clear my cookies on the "killer" browser I can get to the site again.
So, whatever malicious behaviour IIS is trying to curb with this would not work against an even slightly determined opponent. In most cases, if excrement does hit fan it will be coding/configuration error and not the fault of the user who happened to request a page at that time.
Therefore, I'd like to turn this feature off as the theoretical user would have no idea that they need to clear their cookies before they can access the site again. I would really appreciate any ideas on how this might be possible.
Yous should be using ASP.Net Session StateServer instead of In-Proc (see msdn for details). That way, you session will run in different process and won't be affected by IIS crash.
Turn what "feature" off? If the worker process is reset (and your using in-proc session) then the session is blown away on a reset.
You might want to investigate moving your session storage to a state server or some other out of process scenario.
Also, you might want to set the application pool to use several worker processes (aka: web garden) this way if one process is killed the others continue serving content.
Next, as another option you might want to set up multiple web servers and load balance them.
Finally, you might want to profile the app to see exactly how they are causing it to spin into nothingness. My guess is that there are a number of code issues you are simply covering up with this idea.