I have a WCF service that requires a certain response time (under 1 minute).
My problem is that every so often, most often in the mornings the service takes a long time to respond (sometimes over 2 minutes).
I'm thinking this is because the app has recycled and the first run must recompile.
Are there other reasons this might happen?
Is it possible to turn off app recycling? And if it is, will that cause any side effects or instability? I'm assuming there must be a reason why asp.net apps are set to recycle.
Is there anything else that can be done to improve that first run performance?
Yes you can prevent the AppPool from recycling. Another option would be to create a keep-alive job to continually ping the service to keep the worker process from sleeping.
Basically the following rules dictate when an application is recycled or unloaded:
After the App Pool Recycle time has been reached - by default this is every 29 hours I think.
A set time after the last request to application.
Using a keep-alive to ping the service would solve 2, and then you'd just have to deal with 1.
Depending on your version of IIS, there are slightly different ways to configure this.
For IIS 6
For IIS 7
The idle time out I think would normally default to "infinte", but can be configured through the processModel element (idleTimeout attribute) of your configuration files.
As to first run performance - without looking at your app it's hard to say, have you run something like DotTrace or another profiler over it?
Are you doing a lot of intensive lookups and caching data in that first load? Can these be deferred?
Performance problems can be caused by anything you haven't ruled out first. Since you haven't ruled anything out, it could be cauased by anything at all.
Maybe a silly idea : could you schedule a console app to hit your service at e.g. 5:30am in the morning, so that this request would take a long time to run, and your regular users coming in after that won't have that problem?
Sure - it's not dealing with the root cause, but for the time being, it might be a useful workaround - no?
Marc
Related
I am having some performance issues with my iis webserver. It is hanging randomly and I am trying to figure out how to speed up the server. I enabled Failed request tracing on the server and set it to generate a log when the request is over 3 seconds.
The resulting logs(xml) dont show much but there is a point in the compact performance log that indicates what part of the log the server is hanging on. Below is the part of the log where the large time loss is occurring.
65. i GENERAL_GET_URL_METADATA PhysicalPath="", AccessPerms="513" 17:46:32.577
66. i HANDLER_CHANGED OldHandlerName="", NewHandlerName="ExtensionlessUrlHandler-Integrated-4.0", NewHandlerModules="ManagedPipelineHandler", NewHandlerScriptProcessor="", NewHandlerType="System.Web.Handlers.TransferRequestHandler" 17:46:32.577
67. i VIRTUAL_MODULE_UNRESOLVED Name="FormsAuthentication", Type="System.Web.Security.FormsAuthenticationModule" 17:46:47.771
I am not sure what Handler changed is but it is taking a long time, any tips would be great on where to start looking.
It is hard to come up with a solution without having any piece of code in sight. Here are some general hints/tips you can follow in order to have great performances with an ASP.NET application.
The fastest way to do a request is to not do it in the first place. Try caching everything that can be cached. There are server-side caches and client-side caches. Each have their own uses, but you are not limited to only one type.
Make sure you do not cache and/or keep references of any request-related objects into memory. ASP.NET have a limited number of concurrent requests and keeping a request reference in memory will hang your server if it runs out of threads
Close the request as soon as you are done with it
Everything that is not needed by the client at the time of the request should be done in the background
Make sure you have no memory leak in your application. Garbage Collections are often the cause of hangs in ASP.NET application. When garbage collecting, all running threads are paused. This is especially true for Gen 2 garbage collections. You can enable background generation 2 garbage collections.
Isolate the problematic code. Use a profiler and see which type of request is CPU-intensive. Then dig deeper and see what inside that request makes it slow.
In any well-balanced application, objects should either be short-lived and live forever. In the case of an ASP.NET application, the objects created during the course of a request should ideally die within that request or during the next GC gen 0.
Consider object pooling for large objects and objects that are long to initialize
Make sure your app pool doesn't totally crash and restarts (look the IIS logs and/or the Windows Events)
Some useful debugging tools you can use:
LeanSentry. Great for diagnosing ASP.NET server hangs
windbg. High learning curve but by far the most powerful debugging tool you can use
PerfView. Useful for analyzing ETW events like I/O or CPU usage
There are many ways to improve server performance. But before that you should start with checking CPU usage during the "hang". An infinite loop in the application code may cause this behavior. Unless there is I/O, locking, or sleeps in the loop, you will be able to see it from the CPU usage as you will get exactly one full core's worth of CPU usage for each infinite loop.
Help link to improve server performance
More Info:
I can see entry related to VIRTUAL MODULE UNRESOLVED: which is related to bad use of Response.Redirect(url); Also make sure you have deployed your app on integrated mode on IIS.
here's a simple checklist you might want to reconsider:
Always pre-compiling your site, as opposed to copying it! you might gain a significant performance boost compiling your website before deployment: ASP.NET Precompilation Overview
Do not run the production application with debug="true" enabled, when debug flag is true in your web.config, Much more memory is used within the application at runtime, and since some additional debug paths are enabled, codes can execute much slower
Check your Web.config file to ensure trace is disabled in the section
IIS 7.5 comes with the Auto-Start Feature. WAS (Windows Process Activation Service) starts all the application pools that are configured to start automatically, ensure that your application pool is configured to AlwaysRunning in the IIS 7.5 applicationHost.config, check out here for more detail.
Every asp.net server can be well configured by aspnet.config file located in the root of the framework folder. Ensure that Publisher Evidence for Code Access Security (CAS) is set to false in your aspnet.config file, This might increase the initial page load when you restart the ASP.NET app pool. you can read more about it here.
Also you might want to try Application Initialization Module for IIS 7.5, this module also available on IIS 8.0 can decrease the response time for first requests by pre-loading worker processes
My shared hosting provider set up IIS recycle app pool every 3 minutes for idle.
So my session factory often recreates (at application startup). As I have about 70-100 entities it takes about 2-5 seconds to construct factory. So cold start of my application is rather long. I haven't access to IIS setting.
You can offset a lot of the cost of setting up your factory by generating your proxies at build-time instead of runtime. This article explains the steps how.
Being realistic, the simplest change is to ask that the app-pool isn't recycled so frequently (since this is an expensive operation for your application). I'm sure they've set the timeout very low as a "performance" setting, but really this is generating work and slowing things down.
You might not have access to the IIS settings directly, but this shouldn't stop you from contacting your supplier's technical support and getting it resolved.
If you are in a full trust environment (doubtful, but provider may be willing to work with you on this), you can try serializing your configuration so it doesn't need to be rebuilt each time. Merging all your entity mappings into a single XML doc can help also (just do this as build step so its not a nightmare to work with mappings).
More info here: http://nhibernate.info/blog/2009/03/13/an-improvement-on-sessionfactory-initialization.html
Have you tried to stop your site from being idle in the first place? I use uptime robot that is FREE and pings your site every 5 minutes. The benefit of this service is that it only requests the headers of the page you set up as a monitor and therefore does not affect logging such as Google Analytics.
However said you will need to test this to see when your app does indeed recycle to see if uptime robot works with your shared hosting provider. The best way is to log every time the session factory is re-built.
not much you can do. app pool recycle shuts down your app...
I guess you could try to fool the recycler by having the application do something every 2:45.
I'm trying to implement some fail safes on a client's web server which is running two of their most important sites (ASP.NET on IIS7). I'm going to set up application pool limiting so that if any w3wp process uses 90%+ CPU for longer than a minute then it gets killed (producing a temporary 503 Service Unavailable message to any visitors), and based on my local testing will be restarted within a minute - a much better solution than having one CPU-hogging process taking down the whole server for any length of time.
This seems to work, however during my fiddling on my local IIS7 instance I've noticed that if a request calls my "Kill.aspx", even when the site comes back up IIS will not serve the session that caused it to hang. I can only restart the test site from a different session - but as soon as I clear my cookies on the "killer" browser I can get to the site again.
So, whatever malicious behaviour IIS is trying to curb with this would not work against an even slightly determined opponent. In most cases, if excrement does hit fan it will be coding/configuration error and not the fault of the user who happened to request a page at that time.
Therefore, I'd like to turn this feature off as the theoretical user would have no idea that they need to clear their cookies before they can access the site again. I would really appreciate any ideas on how this might be possible.
Yous should be using ASP.Net Session StateServer instead of In-Proc (see msdn for details). That way, you session will run in different process and won't be affected by IIS crash.
Turn what "feature" off? If the worker process is reset (and your using in-proc session) then the session is blown away on a reset.
You might want to investigate moving your session storage to a state server or some other out of process scenario.
Also, you might want to set the application pool to use several worker processes (aka: web garden) this way if one process is killed the others continue serving content.
Next, as another option you might want to set up multiple web servers and load balance them.
Finally, you might want to profile the app to see exactly how they are causing it to spin into nothingness. My guess is that there are a number of code issues you are simply covering up with this idea.
How do you work-around the fact that sessions are dropped every time you deploy certain code files to an ASP.NET website? Sometimes we need to deploy a crucial fix in the middle of the day but don't want to boot off all our users for it.
By default Sessions are stored InProc. You should choose an out-of-process option.
Maybe just a StateServer is enough in your scenario
One way would be to have a load-balanced server set-up. You could direct all traffic to server A, patch Server B and then repeat the other way around.
Alternatively, as #Curtisk states, better to get to the stage where you don't need to do "hot patches" through rigourous testing and then proceed to planned outages advertised in advance.
Hope this helps.
The reason why this happens is that deploying the new code causes the application pool to get recycled. You then lose everything you have in memory.
The way to get around this is then not to save anything in memory.
How difficult this is depends on your architecture.
One solution could be to save your session information in SQL Server, using the out of process state. Note do not use an in memory state server, as if the app pool is recycled you will lose this.
We have an ASP.Net 2.0 web application running in a web farm which is using the ASP.Net State service to store sessions.
We have been having problmes with the service intermittently and have changed a few things such as the machineKey in the machine.config.
My actual question is around the monitoring of the state service service. We have all 4 available performance counters running on the server that hosts the service and as yet we have not seen a single session time out. We have also seen the number of active sessions slowly rise over a period of time, but never become less.
Does the state service recognise when sessions time out? Is there something we should be doing manually?
Edit: We have given up on the state service and gone with SQL server sessions.
To answer the questions below, it seems that sessions go up forever until the service falls over and it is very doubtful that any oen threads are linked to the state server. This is a fairly basic web app at the end of the day.
It seems from the reading I am doing that anumber of other people have experienced similar things, but there seems to be a general lack of common sense and knowledge in any responses flying about.
MS seem to have almost no documentation on this topic.
In ASP.Net session time outs can be configured in web.config and machine.config. The default time out assuming nothing has changed will be 20 mins. The machine.config file can be set to not allow overriding, which means that any changes specified in web.config files will not override these settings.
Have you ensured that the appropriate settings are in place in both machine and web config files?
The state service should drop each session after 20 mins of inactivity assuming the default settings.
At what point are your inactive sessions dropped? I assume they are not exponentially increasing, unless your are restarting the service in order to clear them they must be being dropped at some point.
Do you have something that might be hitting the session and keeping it alive without you knowing? Is there are thread being spawned somehwere that is doing work inteh background and holding on to your session? As far as my expeireince goes the timeout is set int he web config file and it just doesn't it's magic from there.
In my experience we've found out that native state server or even using SQL Server for sessions is a very scary scenario as both have issues.
I think you can explore other products for this to achive the absolute best.
A free option would be Velocity but it is still not released.
And another comprehensive but proven product will be (Very expensive actually) NCache
Take a look and see which looks best for you.
About SQL Server, you server will die very soon if you have enough number of hits coming in (I belive you have some hits already which yielded you to do Web Farm or you do it just for the sake of redundancy)
I am sure this will get modded down, but I have to say it.
If you are having issues with the state server, then there is likely an error somewhere in your web application. Charles' comment above seems like good places to start checking, but somewhere there is a life cycle issue.
Go back over the code and check your assumptions. Take a new computer, visit your website (create a session) and let it sit for an hour. If your session is still alive, then something is wrong. Create a new web application that just has a single page reporting the the age of the current session and try the same thing. You should find that after an hour (default is 20 minutes) the session is no longer valid. Now you have a system that is working as expect and one that is not, both using the same session server, so you can rule that out as the problem, now start going through code/configuration and see where you could be keeping it alive (or preventing the time-out).
Here, by the way, is a 'valid' session config. If you don't have your looking something like this, you have likely found your issue:
<sessionState
mode="StateServer"
stateConnectionString="tcpip=10.1.1.1:55455"
cookieless="false"
timeout="20" />
Also make sure you are not overriding your web.config with your machine.config to have a longer timeout.