I have a ASP.NET application running on IIS 8. My application sometimes does not respond to request with clients logging a timeout and then IIS starts sending out 503 Unavailable response codes. Each outage seems to last 5 minutes which corresponds to a 5 minute limit interval for Kill w3wp.
I have the App Pool configured for a Kill w3wp limit action with limit percentage of 80%.
While it might seems the pool is getting killed due to high CPU usage, I could not find any records in the event viewer. In fact, it had gone to idle state because of no requests in about 25 minutes.
The crash happened with 3 simple identical requests issued when the pool was idle.It cannot be reproduced manually in any environment.
This is happening about once a week. Since its a critical application, its running on a separate server with no other applications installed.
I have changed the Limit from Kill to throttle but I am not sure its a good idea. Anyways I've got to find a way to reproduce this.
Can the pool fail to exit an idle state?
If there is no traffic, w3wp.exe may go idle aka stop running. But that is not the same as it being truly stopped and it would not cause a 503. It would just start back up on the next request.
A 503 is more likely because your IIS App Pool is crashing or literally being stopped.
I would check Windows Event Viewer to see if says anything around that time frame around w3wp crashing. You can also set IIS to do more detailed logging anytime your w3wp app pool is stopped, started, recycled.
Docs about enabling more app pool logging:
https://technet.microsoft.com/en-us/library/cc753412%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396
You could also search various IIS logs for errors:
https://stackify.com/beyond-iis-logs-find-failed-iis-asp-net-requests/
If you set auto recycle option then remove it and make it schedule recycle instead.
It seems as the memory consumption by your application tends to saturation level, IIS being recycled
Related
I have two Application Pools, Prod_Apps_Pool and Prod_Mobs_Pool. For some reason, the Prod_Apps_Pool will keep stopping, but the Prod_Mobs_Pool will keep running just fine. I do understand that it's the Rapid-Fail Protection that is turning off the pool; what I don't understand is what failing specific process is causing this.
Looking at Event Viewer, I do get a System Error that says "Application pool 'Prod_Apps_Pool' is being automatically disabled due to a series of failures in the process(es) serving that application pool." Prior to this error, I have several warnings about processes serving the pool exceeding time limits during shut down, as well as processes requesting a recycle due to reaching a virtual memory limit.
I do not see anything in the Application Log, nor anything in the W3SVC# logs IIS logs. If I had to guess, I would say that these processes that are exceeding time limits are counting as "failures", the maximum number of failed processes in a given time limit is being hit, and then the Rapid-Fail Response is being triggered stopping the application pool. I would further guess that one of the applications in the pool specifically is causing these issues.
How can I specifically find out which application is causing this to happen? The only thing I have seen suggested so far is to put every single application in the pool into it's own pool, then find out which pool is shutting it down. But aside from the impracticality of separating a pool with dozens of applications into dozens of separate pools, this still doesn't tell me what error is causing all of this.
What's the best approach to narrow down the application and error causing this?
Recently, some of my IIS application is having no response. I check the event log and found that some
error happened in WAS service which cause some application pool to be stopped.
The following is the error message.
I can see only the process id causing
the problem.
I checked the error and find in one apppool a process is exceeded time limits during shut down.
Is that possible the process cannot shutdown and cause the other apppool to shutdown. My solutions is set the Test_pool application pool timeout to 1440 minutes(24 hour) and set recylcing time at a specific time
of the day to avoid the shutdown failure of process and prevent the auto shutdown of application pool. Is that workable?
A process serving application pool 'CEHL_POOL' exceeded time limits during shut down. The process id was '529004'.
A worker process '504596' serving application pool 'TEST_POOL' failed to stop a listener channel for protocol 'http' in the allotted time. The data field contains the error number.
A worker process with process id of '794136' serving application pool 'TEST_POOL' was shutdown due to inactivity. Application Pool timeout configuration was set to 20 minutes. A new worker process will be started when needed.
Application pool 'TEST_POOL' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
The event logs included mutiple errors with different application pool. So please point out which application pool stop responding. The error message "exceeded time limits during shut down" means the application failed to finish all current requests inside the process before application pool shutdown timeout reached. So either threads hang up or locked up.
“being automatically disabled due to a series of failures。。。”is caused by rapid failure protection, so you need to check application events and you should be able to find the crash event.
No matter crash or hang, you need to reproduce the issue and monitor the request status inside worker process monitor. It will show the time taken and current handler. You may need to collect and analyze dump file to trace the root cause.
Besides, in some case, http.sys will cause this issue because it is no longer able to wake up a new appliction pool. You may need to check whether your application pool is able to be waked up after your application stop responding.
We have a web application deployed on IIS 7.5 target framework 4.0
the application perform slow when leave idle for few minutes for first time and then perform as expected this happened each time application is idle.
With the help of fiddler I found its TCP/IP connection which is taking time about 21 secs whilein subsequent calls this time is 0.
The Idle time out is also set high and connection time out is also high in the IIS settings.
server is - Windows 2008 R2.
there is nothing in the event viewer related to the website.
we used form authentication but the time out for that is also set about 10 hours in the config file.
Can anybody point me to the setting with is affecting the response time after the app is idle for some time.
Note - this was working proper when deployed withing the LAN but this problem starts when deployed out of the LAN or in separate domain.
Problem
here is the problem in IIS app pool idle time out, its by default set to 20 minutes, after 20 minutes app pool shutdown if no request within 20 minutes,
when any request comes after 20 minute its again start,
The problem is that the first visit to an app pool needs to create a new w3wp.exe worker process which is slow because the app pool needs to be created, ASP.NET or another framework needs to be loaded, and then your application needs to be loaded. so it may take time 20-30 seconds or depends on the application content size.
Solution
so to avoid this type of delay we need to set the idle time out to 0.
now it will always load fast.
app pool setting
The IIS application pool is shut down after 30 minutes of inactivity. After that, when you make a request IIS basically has to start the website up again, which leads to the behavior you are describing. You can change the idle time of your website in IIS though to avoid it.
You could also look into the Auto-Start feature of the 4.0 framework.
Well, a bit late, but may help someone else. I had the same problem, nothing in the logs, spent days, then looking at the network adapter properties / configuration / power management - Allow computer to shut down save power was checked. Unchecked and the problem was solved.
using SignalR in procution. at startup everything works fine, but after 8-9 hours, service stops working, without any exception, or any log information in Event Logs.
Info:
Online Users (who uses this service in this 8-9 hours): 3000
Online Concurrent Users Max (at same time): 200
Hubs Count: 1 (try catch in every method for logging)
after browser timeout, it returns "404 not found".
do u have any ideas?
What version of SignalR are you using? You should be using v0.5.2 as previous versions had issues with zombie connections which would shut down your app by causing either an OutOfMemoryException or exceed the allowed number of requests for the app pool.
Essentially what would happen is that the # of requests would get backed up (use the performance monitor to view ASP.NET/{Requests Current, Requests Queued, Requests Rejected} - See Performance Tuning) and/or you would max out your IIS requests and the service would shut down. You can manually override this and increase your Requests Current for ASP.NET. I increased this to about 20,000 on our production box.
If you're not maxing out your requests, your app pool may be shutting down due to increased memory usage or # of exceptions shutting down the app pool.
In the IIS manager, under Management, double-click on Configuration Editor.
Next, at the top, click on system.applicationHost/applicationPools then click on the RHS of the line that says "(Collection)". This will open up your application pool collection editor.
Select your SignalR app pool and check the Properties at the bottom. Here you can set the periodicRestart/memory threshold to whatever you want.
In our application I'm finding that we're good for about 45 minutes, due to the high traffic nature of the application.
Hopefully that helps.
In an ASP.NET website we are storing sessions in SQL Server. All is working fine except that sessions frequently recycle. I've a timeout period set to 30 minutes but some times it recycles within a few minutes. We've a dedicated server, and website running under a 'classic' application pool. I've searched a lot on this problem, but didn't find a satisfactory answer. Any help will be greatly appreciated.
Note: mostly it happens on a page where there is lots of use of viewstate, I'm curious that is there a link of viewstate with session recycling?
We've experienced this problem when we either have a web farm (more than one web server servicing clients) or a web garden (more than one worker process in an application pool).
If you have a web farm, then you need to ensure that all web servers have the same machine key and that all instances have the exact same application path.
If you have a web garden, try dropping the maximum worker processes back to 1 to see if that resolves the issue.
While you are checking the IIS settings, you should probably also ensure that the application pool is not recycling regularly. This can be due any of the following specified in the app pool:
1) Private Memory Limit (the app pool is reset if the maximum amount of memory has been exceeded)
2) Regular Time Interval recycling (the app pool is automatically recycled after a specified number of minutes, defaulting to 1740, and/or at specific times).
3) Idle Time-out (the number of minutes of inactivity that can elapse before the application pool is automatically shutdown).
You should also check the event logs for reports of the application pool crashing or otherwise being recycled.
Update:
An additional thought:
If you have an application, such as anti-virus or backup software, that monitors your application's bin directory and modifies or changes attributes (such as the backup flag or timestamp) of files in that directory OR your web.config file, this will cause application recycling as well.