I'm trying to implement some fail safes on a client's web server which is running two of their most important sites (ASP.NET on IIS7). I'm going to set up application pool limiting so that if any w3wp process uses 90%+ CPU for longer than a minute then it gets killed (producing a temporary 503 Service Unavailable message to any visitors), and based on my local testing will be restarted within a minute - a much better solution than having one CPU-hogging process taking down the whole server for any length of time.
This seems to work, however during my fiddling on my local IIS7 instance I've noticed that if a request calls my "Kill.aspx", even when the site comes back up IIS will not serve the session that caused it to hang. I can only restart the test site from a different session - but as soon as I clear my cookies on the "killer" browser I can get to the site again.
So, whatever malicious behaviour IIS is trying to curb with this would not work against an even slightly determined opponent. In most cases, if excrement does hit fan it will be coding/configuration error and not the fault of the user who happened to request a page at that time.
Therefore, I'd like to turn this feature off as the theoretical user would have no idea that they need to clear their cookies before they can access the site again. I would really appreciate any ideas on how this might be possible.
Yous should be using ASP.Net Session StateServer instead of In-Proc (see msdn for details). That way, you session will run in different process and won't be affected by IIS crash.
Turn what "feature" off? If the worker process is reset (and your using in-proc session) then the session is blown away on a reset.
You might want to investigate moving your session storage to a state server or some other out of process scenario.
Also, you might want to set the application pool to use several worker processes (aka: web garden) this way if one process is killed the others continue serving content.
Next, as another option you might want to set up multiple web servers and load balance them.
Finally, you might want to profile the app to see exactly how they are causing it to spin into nothingness. My guess is that there are a number of code issues you are simply covering up with this idea.
Related
Since we have moved to azure, we have numerous session lost issues only on production.
We have InProc, cookie based, sticky session, large timeout, no high traffic and no high memory/process usage.
We use HAProxy as loadbalancer.
I have done basic research and none of the following seems to be the cause:
session timeout
application pool settings/recycling
memory size and usage thresholds
no eaten exceptions
there is no changes to file system to cause a restart
I'm particularly more suspicious about how loadbalancer/ssl and application work together and if http headers are fine, but I don't know any tools to really monitor that.
I'm assigned to find a solution at the same time I have no privilege to access the machines.
Logs(Log4Net) are all stored in database but doesn't help to give a clear understanding of what is going on the system and cannot follow a user session using them.
I'm allowed to find the problem by adding required logs to code or to develop some kind of monitoring module or to use profiling/debugging tools.
Only once a month there will be a production deployment so I'm trying to use the opportunity as best as possible.
Question:
Is there any useful monitoring/profiling tool that can give me a clear view of what is happening in the system by aggregating information I may need? for example following a user/session between requests from time of login until session drop plus information about headers and other system application parameters.
if there is not such a tool out there, please give me your ideas to write one?
This is a common issue in load balanced environment. As mentioned in this answer for a similar question,
InProc mode, which stores session state in memory on the Web server. Which means that session data is maintained inside your web server on a given VM and is not shared outside of the VM. So when you have multiple server for load balancing, the session state isn't shared with each other. To solve this, you must store your session state external to the web server.
Use Redis, or SQL Database, or something else.
I am having some performance issues with my iis webserver. It is hanging randomly and I am trying to figure out how to speed up the server. I enabled Failed request tracing on the server and set it to generate a log when the request is over 3 seconds.
The resulting logs(xml) dont show much but there is a point in the compact performance log that indicates what part of the log the server is hanging on. Below is the part of the log where the large time loss is occurring.
65. i GENERAL_GET_URL_METADATA PhysicalPath="", AccessPerms="513" 17:46:32.577
66. i HANDLER_CHANGED OldHandlerName="", NewHandlerName="ExtensionlessUrlHandler-Integrated-4.0", NewHandlerModules="ManagedPipelineHandler", NewHandlerScriptProcessor="", NewHandlerType="System.Web.Handlers.TransferRequestHandler" 17:46:32.577
67. i VIRTUAL_MODULE_UNRESOLVED Name="FormsAuthentication", Type="System.Web.Security.FormsAuthenticationModule" 17:46:47.771
I am not sure what Handler changed is but it is taking a long time, any tips would be great on where to start looking.
It is hard to come up with a solution without having any piece of code in sight. Here are some general hints/tips you can follow in order to have great performances with an ASP.NET application.
The fastest way to do a request is to not do it in the first place. Try caching everything that can be cached. There are server-side caches and client-side caches. Each have their own uses, but you are not limited to only one type.
Make sure you do not cache and/or keep references of any request-related objects into memory. ASP.NET have a limited number of concurrent requests and keeping a request reference in memory will hang your server if it runs out of threads
Close the request as soon as you are done with it
Everything that is not needed by the client at the time of the request should be done in the background
Make sure you have no memory leak in your application. Garbage Collections are often the cause of hangs in ASP.NET application. When garbage collecting, all running threads are paused. This is especially true for Gen 2 garbage collections. You can enable background generation 2 garbage collections.
Isolate the problematic code. Use a profiler and see which type of request is CPU-intensive. Then dig deeper and see what inside that request makes it slow.
In any well-balanced application, objects should either be short-lived and live forever. In the case of an ASP.NET application, the objects created during the course of a request should ideally die within that request or during the next GC gen 0.
Consider object pooling for large objects and objects that are long to initialize
Make sure your app pool doesn't totally crash and restarts (look the IIS logs and/or the Windows Events)
Some useful debugging tools you can use:
LeanSentry. Great for diagnosing ASP.NET server hangs
windbg. High learning curve but by far the most powerful debugging tool you can use
PerfView. Useful for analyzing ETW events like I/O or CPU usage
There are many ways to improve server performance. But before that you should start with checking CPU usage during the "hang". An infinite loop in the application code may cause this behavior. Unless there is I/O, locking, or sleeps in the loop, you will be able to see it from the CPU usage as you will get exactly one full core's worth of CPU usage for each infinite loop.
Help link to improve server performance
More Info:
I can see entry related to VIRTUAL MODULE UNRESOLVED: which is related to bad use of Response.Redirect(url); Also make sure you have deployed your app on integrated mode on IIS.
here's a simple checklist you might want to reconsider:
Always pre-compiling your site, as opposed to copying it! you might gain a significant performance boost compiling your website before deployment: ASP.NET Precompilation Overview
Do not run the production application with debug="true" enabled, when debug flag is true in your web.config, Much more memory is used within the application at runtime, and since some additional debug paths are enabled, codes can execute much slower
Check your Web.config file to ensure trace is disabled in the section
IIS 7.5 comes with the Auto-Start Feature. WAS (Windows Process Activation Service) starts all the application pools that are configured to start automatically, ensure that your application pool is configured to AlwaysRunning in the IIS 7.5 applicationHost.config, check out here for more detail.
Every asp.net server can be well configured by aspnet.config file located in the root of the framework folder. Ensure that Publisher Evidence for Code Access Security (CAS) is set to false in your aspnet.config file, This might increase the initial page load when you restart the ASP.NET app pool. you can read more about it here.
Also you might want to try Application Initialization Module for IIS 7.5, this module also available on IIS 8.0 can decrease the response time for first requests by pre-loading worker processes
My shared hosting provider set up IIS recycle app pool every 3 minutes for idle.
So my session factory often recreates (at application startup). As I have about 70-100 entities it takes about 2-5 seconds to construct factory. So cold start of my application is rather long. I haven't access to IIS setting.
You can offset a lot of the cost of setting up your factory by generating your proxies at build-time instead of runtime. This article explains the steps how.
Being realistic, the simplest change is to ask that the app-pool isn't recycled so frequently (since this is an expensive operation for your application). I'm sure they've set the timeout very low as a "performance" setting, but really this is generating work and slowing things down.
You might not have access to the IIS settings directly, but this shouldn't stop you from contacting your supplier's technical support and getting it resolved.
If you are in a full trust environment (doubtful, but provider may be willing to work with you on this), you can try serializing your configuration so it doesn't need to be rebuilt each time. Merging all your entity mappings into a single XML doc can help also (just do this as build step so its not a nightmare to work with mappings).
More info here: http://nhibernate.info/blog/2009/03/13/an-improvement-on-sessionfactory-initialization.html
Have you tried to stop your site from being idle in the first place? I use uptime robot that is FREE and pings your site every 5 minutes. The benefit of this service is that it only requests the headers of the page you set up as a monitor and therefore does not affect logging such as Google Analytics.
However said you will need to test this to see when your app does indeed recycle to see if uptime robot works with your shared hosting provider. The best way is to log every time the session factory is re-built.
not much you can do. app pool recycle shuts down your app...
I guess you could try to fool the recycler by having the application do something every 2:45.
I wrote a test page that does a bunch of busy work in a method called at page load. This process as I have it now takes around 12 seconds.
If I try to load another page while the first long running page is loading, this second page doing nothing except writing out a hello world, it doesn't load until the first long running page is finished.
Why is this the case? I would think IIS would be able to handle multiple concurrent connections, it seems crazy that one long running page would stop every other page in the application from loading. I must be missing something or not understand how IIS works.
I would think multiple independent requests would be spawned on different threads. Is this only the case if the requests are from different sessions entirely? Are all requests from a single session bound to a single thread?
bd
You need to turn off sessionState... if session is on, any sub-sequent request is waiting on the previous thread to finish in order to continue on that particular session.
So turn session off, it will not need to wait for any previous session.
in web.config put
<system.web>
<sessionState mode="Off" />
</system.web>
Now you should have those request process concurrently.
I just hit this issue too, and for me it turns out it is the session feature of ASP.NET. Basically if you have sessions enabled, each request will return in the order it arrived per user.
Try using 2 different browsers, for me it no longer blocked. Here is a similar question:
IIS 5.1 on XP Classic ASP appears to be in single threaded mode
It's a threading problem. While IIS can handle multiple connections simultaneously, ASP.NET, by default, is configured in single-threaded mode.
This choice by Microsoft was to prevent dummy developers to make common concurrency mistakes. ASP.NET for Mono doesn't show this behaviour and if you access shared resources without prior locking you might be... f... you know ;) by a yellow page of death.
I don't remember the exact procedure, but in the properties of your website you can choose the application pool and also the number of threads. I saw with my eyes that IIS6 sets it to 1 by default.
If you can't find it, tell me and tomorrow I'll take a look on my development server in lab.
Hope to have been of help.
We have an ASP.Net 2.0 web application running in a web farm which is using the ASP.Net State service to store sessions.
We have been having problmes with the service intermittently and have changed a few things such as the machineKey in the machine.config.
My actual question is around the monitoring of the state service service. We have all 4 available performance counters running on the server that hosts the service and as yet we have not seen a single session time out. We have also seen the number of active sessions slowly rise over a period of time, but never become less.
Does the state service recognise when sessions time out? Is there something we should be doing manually?
Edit: We have given up on the state service and gone with SQL server sessions.
To answer the questions below, it seems that sessions go up forever until the service falls over and it is very doubtful that any oen threads are linked to the state server. This is a fairly basic web app at the end of the day.
It seems from the reading I am doing that anumber of other people have experienced similar things, but there seems to be a general lack of common sense and knowledge in any responses flying about.
MS seem to have almost no documentation on this topic.
In ASP.Net session time outs can be configured in web.config and machine.config. The default time out assuming nothing has changed will be 20 mins. The machine.config file can be set to not allow overriding, which means that any changes specified in web.config files will not override these settings.
Have you ensured that the appropriate settings are in place in both machine and web config files?
The state service should drop each session after 20 mins of inactivity assuming the default settings.
At what point are your inactive sessions dropped? I assume they are not exponentially increasing, unless your are restarting the service in order to clear them they must be being dropped at some point.
Do you have something that might be hitting the session and keeping it alive without you knowing? Is there are thread being spawned somehwere that is doing work inteh background and holding on to your session? As far as my expeireince goes the timeout is set int he web config file and it just doesn't it's magic from there.
In my experience we've found out that native state server or even using SQL Server for sessions is a very scary scenario as both have issues.
I think you can explore other products for this to achive the absolute best.
A free option would be Velocity but it is still not released.
And another comprehensive but proven product will be (Very expensive actually) NCache
Take a look and see which looks best for you.
About SQL Server, you server will die very soon if you have enough number of hits coming in (I belive you have some hits already which yielded you to do Web Farm or you do it just for the sake of redundancy)
I am sure this will get modded down, but I have to say it.
If you are having issues with the state server, then there is likely an error somewhere in your web application. Charles' comment above seems like good places to start checking, but somewhere there is a life cycle issue.
Go back over the code and check your assumptions. Take a new computer, visit your website (create a session) and let it sit for an hour. If your session is still alive, then something is wrong. Create a new web application that just has a single page reporting the the age of the current session and try the same thing. You should find that after an hour (default is 20 minutes) the session is no longer valid. Now you have a system that is working as expect and one that is not, both using the same session server, so you can rule that out as the problem, now start going through code/configuration and see where you could be keeping it alive (or preventing the time-out).
Here, by the way, is a 'valid' session config. If you don't have your looking something like this, you have likely found your issue:
<sessionState
mode="StateServer"
stateConnectionString="tcpip=10.1.1.1:55455"
cookieless="false"
timeout="20" />
Also make sure you are not overriding your web.config with your machine.config to have a longer timeout.