In my web site, I turned off recycling on app pool recycling settings. I was wondering if worker process is still releasing its memory even though recycling is turned off? Since I turned off recycling, the memory usage by the web site is increasing without a limit. Does worker process create a new thread for each request? If so does each thread get killed after it serves the request?
Yes, each request causes a new thread to be created or taken from the thread pool. The number of worker threads available per processor is governed by the maxWorkerThreads in the processModel section of the web.config. The range for this value is 5 to 100, with the default value being 20.
So the answer to your question is that each request gets its own thread and if none are available, then the request is queued up and processed once a thread is available. The thread is not necessarily killed when the request finishes, because it may return to the thread pool.
Related
Recently, some of my IIS application is having no response. I check the event log and found that some
error happened in WAS service which cause some application pool to be stopped.
The following is the error message.
I can see only the process id causing
the problem.
I checked the error and find in one apppool a process is exceeded time limits during shut down.
Is that possible the process cannot shutdown and cause the other apppool to shutdown. My solutions is set the Test_pool application pool timeout to 1440 minutes(24 hour) and set recylcing time at a specific time
of the day to avoid the shutdown failure of process and prevent the auto shutdown of application pool. Is that workable?
A process serving application pool 'CEHL_POOL' exceeded time limits during shut down. The process id was '529004'.
A worker process '504596' serving application pool 'TEST_POOL' failed to stop a listener channel for protocol 'http' in the allotted time. The data field contains the error number.
A worker process with process id of '794136' serving application pool 'TEST_POOL' was shutdown due to inactivity. Application Pool timeout configuration was set to 20 minutes. A new worker process will be started when needed.
Application pool 'TEST_POOL' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
The event logs included mutiple errors with different application pool. So please point out which application pool stop responding. The error message "exceeded time limits during shut down" means the application failed to finish all current requests inside the process before application pool shutdown timeout reached. So either threads hang up or locked up.
“being automatically disabled due to a series of failures。。。”is caused by rapid failure protection, so you need to check application events and you should be able to find the crash event.
No matter crash or hang, you need to reproduce the issue and monitor the request status inside worker process monitor. It will show the time taken and current handler. You may need to collect and analyze dump file to trace the root cause.
Besides, in some case, http.sys will cause this issue because it is no longer able to wake up a new appliction pool. You may need to check whether your application pool is able to be waked up after your application stop responding.
I'm reading about AsyncControllers in ASP.NET MVC.
It seems that the sole reason why they exist is so that the IIS threads can be saved while the long running work is delegated to regular CLR threads, that seem to be cheaper.
I have a couple of questions here:
Why are these IIS threads so expensive to justify this whole architecture built to support asynchronous controllers?
How do I know/configure how many IIS threads are running in my IIS application pool?
ASP.NET processes requests by using threads from the .NET thread pool. The thread pool maintains a pool of threads that have already incurred the thread initialization costs. Therefore, these threads are easy to reuse. The .NET thread pool is also self-tuning. It monitors CPU and other resource utilization, and it adds new threads or trims the thread pool size as needed. You should generally avoid creating threads manually to perform work. Instead, use threads from the thread pool. At the same time, it is important to ensure that your application does not perform lengthy blocking operations that could quickly lead to thread pool starvation and rejected HTTP requests.
Disk I/O, web service calls, are all blocking. There are best optimized by using async calls. When you make an async call, asp.net frees your thread and the request will be assigned to another thread when the callback function is invoked.
To configure the number of threads you can set:
<system.web>
<applicationPool maxConcurrentRequestsPerCPU="50" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000"/>
</system.web>
Refer: ASP.NET Thread Usage on IIS 7.5, IIS 7.0, and IIS 6.0
These are the setting that Microsoft best practices recommend:
Set maxconnection to 12 * # of CPUs. This setting controls the maximum number of outgoing HTTP connections that you can initiate from a client. In this case, ASP.NET is the client. Set maxconnection to 12 * # of CPUs.
Set maxIoThreads to 100. This setting controls the maximum number of I/O threads in the .NET thread pool. This number is automatically multiplied by the number of available CPUs. Set maxloThreads to 100.
Set maxWorkerThreads to 100. This setting controls the maximum number of worker threads in the thread pool. This number is then automatically multiplied by the number of available CPUs. Set maxWorkerThreads to 100.
Set minFreeThreads to 88 * # of CPUs. This setting is used by the worker process to queue all the incoming requests if the number of available threads in the thread pool falls below the value for this setting. This setting effectively limits the number of requests that can run concurrently to maxWorkerThreads - minFreeThreads. Set minFreeThreads to 88 * # of CPUs. This limits the number of concurrent requests to 12 (assuming maxWorkerThreads is 100).
Set minLocalRequestFreeThreads to 76 * # of CPUs. This setting is used by the worker process to queue requests from localhost (where a Web application sends requests to a local Web service) if the number of available threads in the thread pool falls below this number. This setting is similar to minFreeThreads but it only applies to localhost requests from the local computer. Set minLocalRequestFreeThreads to 76 * # of CPUs.
Note: The recommendations that are provided in this section are not rules. They are a starting point.
You would have to benchmark your application to find what works best for your application.
IIS threads are taken from the default thread pool, which is limited by default based on number of processor cores. If this thread pool queue becomes backed up, IIS will stop responding to requests. By using async code, the thread pool thread can be returned to the pool while the async operation takes place, allowing IIS to service more requests overall.
On the other hand, spawning a new thread on your own does not utilize a thread pool thread. Spawning an unchecked number of independent threads can also be a problem, so it's not a cure all fix to the IIS thread pool issue. Async IO is generally preferred either way.
As for changing the number of threads in the thread pool, check here. However, you should probably really avoid doing so.
Our Webservice need from time to time to serve 100 requets/second while the rest of the time it's 1 request/second. Analazyng IIS logs we found out it tooks around 28s when burst occurs to serve such calls.
Applying Microsoft best practices as cited by #nunespascal drasticly reduced time to 1s in our case.
Below is a the Powershell script we currently use when we deploy our production servers. It updates machine.config taking count of logical Processor number.
<# Get and backup current machine.config #>
$path = "C:\Windows\Microsoft.Net\Framework\v4.0.30319\Config\machine.config";
$xml = [xml] (get-content($path));
$xml.Save($path + "-" + (Get-Date -Format "yyyyMMdd-HHmm" ) + ".bak");
<# Get number of physical CPU #>
$physicalCPUs = ([ARRAY](Get-WmiObject Win32_Processor)).Count;
<# Get number of logical processors #>
$logicalProcessors = (([ARRAY](Get-WmiObject Win32_Processor))[0] | Select-Object “numberOfLogicalProcessors").numberOfLogicalProcessors * $physicalCPUs;
<# Set Number of connection in system.net/connectionManagement #>
$systemNet = $xml.configuration["system.net"];
if (-not $systemNet){
$systemNet = $xml.configuration.AppendChild($xml.CreateElement("system.net"));
}
$connectionManagement = $systemNet.connectionManagement;
if (-not $connectionManagement){
$connectionManagement = $systemNet.AppendChild($xml.CreateElement("connectionManagement"));
}
$add = $connectionManagement.add;
if(-not $add){
$add = $connectionManagement.AppendChild($xml.CreateElement("add")) ;
}
$add.SetAttribute("address","*");
$add.SetAttribute("maxconnection", [string]($logicalProcessors * 12) );
<# Set several thread settings in system.web/processModel #>
$systemWeb = $xml.configuration["system.web"];
if (-not $systemWeb){
$systemWeb = $xml.configuration.AppendChild($xml.CreateElement("system.web"));
}
$processModel = $systemWeb.processModel;
if (-not $processModel){
$processModel = $systemWeb.AppendChild($xml.CreateElement("processModel"));
}
$processModel.SetAttribute("autoConfig","true");
$processModel.SetAttribute("maxWorkerThreads","100");
$processModel.SetAttribute("maxIoThreads","100");
$processModel.SetAttribute("minWorkerThreads","50");
$processModel.SetAttribute("minIoThreads","50");
<# Set other thread settings in system.web/httRuntime #>
$httpRuntime = $systemWeb.httpRuntime;
if(-not $httpRuntime){
$httpRuntime = $systemWeb.AppendChild($xml.CreateElement("httpRuntime"));
}
$httpRuntime.SetAttribute("minFreeThreads",[string]($logicalProcessors * 88));
$httpRuntime.SetAttribute("minLocalRequestFreeThreads",[string]($logicalProcessors * 76));
<#Save modified machine.config#>
$xml.Save($path);
This solution came to us from a blog article witten by Stuart Brierley back in 2009. We sucessfully tested it with Windows Server from 2008 R2 to 2016.
Actually what is written in article you have linked is not true.
Async pattern isn't there to free "super costly IIS Worker Threads" and use in background some other "cheap threads".
Async pattern is there simply to free threads.
You can benefit from it in scenarios where you do not need your threads (and best even not your local machine).
I can name two example scenarios (both I/O):
First:
BeginRequest
Begin async file read
during file read you don't need your thread - so other requests can use it.
File read ends - you get thread from app pool.
Request finishes.
And almost identical second:
BeginRequest
Begin async call to WCF service.
we can leave our machine and don't need our thread - so other requests can use it.
We get response from remote service - we get some thread from app pool to continue.
Request finishes.
It's usually safe to read msdn. You can get information about async pattern here.
I met an issue in ASP.Net. In any Web App config Web.config file there is a section called httpRuntime, and it has an attriube: shutdownTimeout. According to MSDN documentation, this attr specify how long the idle time of the worker process is allowed before ASP.Net runtime to terminate the worker process. On another side, under IIS's ApplicationPool's -> Default AppPool -> properties -> performance tab, there is a setting: "shutdown worker process after being idle for (20) minutes".
I guess that under IIS, this setting is for all worker process which is used to handle incoming request not only the process where a specific ASP.Net runtime resides. And if Web.Config's shutdownTimeout has not taken effect yet, the IIS's setting would then do its work.
However from my observation, although the httpRuntime's shutdownTimeout default value is 90 seconds, my web application was always shut down after idle for 20 minutes. It seems the IIS setting take priority for this aspect.
It is much appreciated if somebody could clarify on this: what is wrong with my guess.
I did some digging and found out the answer:
the attriubte shutdownTimeout controls how long the ASP.Net runtime will shut down the worker process before it gracefully exits by itself if it is asked to terminate by the ASP.Net runtime.
Is this right, any opinion is greatly appreciated.
https://msdn.microsoft.com/en-us/library/e1f13641(v=vs.100).aspx
Specifies the number of minutes that are allowed for the worker process to shut down. When the time-out expires, ASP.NET shuts down the worker process.
The default is 90.
So basically, the worker process has X minutes to shut down the worker process. If it reaches X it is killed.
When recycling happens, I want to move all new request to new w3wp (this is done automatically) and leave the previous w3wp to exit when it gets idle (whatever time it takes).
There's a shutdown time limit config for application pools (I use IIS 7.0) which doesn't take 0 as a value. How can I let the previous w3wp to shutdown when it is done and idle.
The shutdown time limit hint leaves the old worker process running for up to the number of seconds indicated. If all requests are completed prior to that time, then it will shut down earlier.
When recycling happens, a new worker process spins up and immediately begins taking new requests. The old worker process continues working on old in-flight requests until completed, or until the shutdown timelimit is reached. By the way, the maximum shutdown timelimit seconds appears to be 4294967 which is one second shy of 50 days. OMG I hope no one is waiting that long for a request to complete!
Parenthetically, it is possible for a new worker process to NOT actually be able to start, especially if there is not enough free memory.
I have a web service that creates a thread to process some data in the background. I've seen a "System.Threading.ThreadAbortException: Thread was being aborted" message in one of my logs where the thread was killed. I am currently under the assumption that the thread will run as long as it takes to execute the tasks that it's working on, however after googling the exeception I've seen several posts making mention of increasing the ExecutionTimeOut property of the application in the web.config file. My question is:
What is the maximum execution time of a thread executed in ASP.NET? Is this timeout unlimited or still bound by the ExecutionTimeOut property of the application?
This is a similar question.
How to know who kills my threads
You can run it as long as the pool is running. If the pool thats keep the thread is restart then its wait for all the threads to exits, but the web service to run the pages stop working at this moment, and you have problems. Also if you have setup your pool then in pool recycle if the thread is not stop after some time its kill it.
In My program one index thread it was running for more than 26 hours doing hard database work, and was on static function.