I have a ASP.NET web application running on an IIS6 server. The application is making potentially long running calls to a xml service on a remote machine. Some of the service calls on the remote machine are taking a long time to execute (sometimes up to 4 minutes). The long term solution would be to make the calls asyncronous, but as a short term solution we want to increase the timeout for the calls and the overall httpRequest time out.
My fear with this is that the long running calls will fill up the request queue and prevent the "normal" page requests from completing. How can the server, IIS and application settings be tuned to temporarely resolve the issue?
Currently there are approximately 200 page requests/minute and this results in 270 service requests/minute.
The current executionTimeout is 360 (6 minutes)
The current service call time out is 2 minutes
There's this article on Microsoft's Knowledgebase that has some pretty much all the information you might need:
* Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
I will give you some research I have done regarding some of the specific items handled in the article above. This information below applies to IIS6, comments for IIS7 where applicable.
Increase the Processor Worker Thread pool from 25 to at least 100
The default values for the Threadpool size is 100 because the default value for autoConfig is true.
The values covered by autoConfig is
maxWorkerThreads
maxIoThreads
maxConnection
There is one value that is still 25 that must change is – ASPProcessorThreadMax, this can only be set in the IIS metabase (via adsutil tool) in IIS6. [IIS7’s equivalent is the processorThreadMax value]
So I'm opting not to change the machine.config settings as they are fine and there are other paramaters that would be affected by turning off autoconfig, but rather change ASPProcessorThreadMax from 25 to 100 via the IIS metabase (the only way to change this value).
e.g.
cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\<nowiki>adsutil.vb</nowiki>s SET W3SVC/AspRequestQueueMax 100
Max connections per server
maxconnection
The autoconfig sets this value to 12*number of cpu’s, that’s how many connections can be made to each address you are connecting to at one time.
Debugging
Here are some things you can do:
Monitor if requests are waiting in the queue
Monitor the following counter:
Run perfmon
Add counter: ASP.NET Applications/Requests In Application Queue
This will show us if work items are queued because of a shortage of workers.
Check Identity Used by Application Pool
Open IIS Manager
Check which application pool is used by your site in IIS manager.
Choose the Application pool being used in the Application Pools list, then Right Click -> properties and see what account identity is being used.
It should be Network Service by default.
Related
There is an ASP.NET application with a set of web services, and there is a WinForms application that calls these web services.
The w3wp.exe (which handles asmx calls) run special processes that do a really "hard" jobs (like zipping, xml parsing, etc.). It is a long story why we need it to be done in separate processes and why we don't run this code inside w3wp.exe - this doesn't matter here. It must be like that.
If multiple requests came simulteneously from many users, there can be 30-40 such processes, and they consume CPU. They consume it so much that almost no CPU resources is left for IIS itself.
So, the problem is next: when the following HTTP requests are coming to this IIS, IIS cannot handle them: it cannot even pass them to w3wp.exe because server is "bombed". And, as a result, in peak time 30 users have run their queries, and 31st user is getting a WebException in the WinForms client "timeout expired".
I have found a lot of articles in the internet which explain how to tweak IIS, but noone take into account the fact that OTHER processes and/or applications can run on the same machine. And noone tells what to do to make so that IIS is of higher priority.
So, the question is next: is there a way to "explain" IIS that it should start by default the apppool (svchost.exe) and w3wp.exe with hiher priority?
I faced this issue in Windows Server 2008 r2.
I had a windows service (which used parallel processing) which used to bloat up my CPU a lot & left nothing for IIS & SQL Server.
So I set processor affinity to the windows service to consume only specific processors.
You get in Task Manager->RightClick Process -> Set Affinity.
This left other processors free to use for SQL & IIS.
To an extent, It decreased the productivity of the windows service which was now working with a limited set of processors but it did not matter that much.
In an ASP.NET website we are storing sessions in SQL Server. All is working fine except that sessions frequently recycle. I've a timeout period set to 30 minutes but some times it recycles within a few minutes. We've a dedicated server, and website running under a 'classic' application pool. I've searched a lot on this problem, but didn't find a satisfactory answer. Any help will be greatly appreciated.
Note: mostly it happens on a page where there is lots of use of viewstate, I'm curious that is there a link of viewstate with session recycling?
We've experienced this problem when we either have a web farm (more than one web server servicing clients) or a web garden (more than one worker process in an application pool).
If you have a web farm, then you need to ensure that all web servers have the same machine key and that all instances have the exact same application path.
If you have a web garden, try dropping the maximum worker processes back to 1 to see if that resolves the issue.
While you are checking the IIS settings, you should probably also ensure that the application pool is not recycling regularly. This can be due any of the following specified in the app pool:
1) Private Memory Limit (the app pool is reset if the maximum amount of memory has been exceeded)
2) Regular Time Interval recycling (the app pool is automatically recycled after a specified number of minutes, defaulting to 1740, and/or at specific times).
3) Idle Time-out (the number of minutes of inactivity that can elapse before the application pool is automatically shutdown).
You should also check the event logs for reports of the application pool crashing or otherwise being recycled.
Update:
An additional thought:
If you have an application, such as anti-virus or backup software, that monitors your application's bin directory and modifies or changes attributes (such as the backup flag or timestamp) of files in that directory OR your web.config file, this will cause application recycling as well.
I have a WCF service hosted in IIS which takes a long time (around 5 hours) to execute. the WCF service basically generates some reports using SSRS (SQL server reporting services) and saves them to a locaton on the server. this service was actually stopping after generating few reports, so I disabled the "recycling of worker process", "shut down worker processes after being idle" and "limit kernel request queue" in application pool and that fixed the issue and all the reports were generated regardless of the amount taken to generate them. but I am not sure if this is the right fix for this and I would like to know what is the impact of unchecking these settings in application pool for the WCF service in IIS? and is there any better way to get around this problem?
For any long running process it is much better to do it outside of IIS.
In this case I would have a regular windows service running that monitored a request queue. When a request comes in to generate a report, it would then spin off a thread to perform the generation.
The web service would be responsible for 3 things. First, adding an item to the queue to be handled. Second, checking status on the queue as to whether the report is ready. Third, sending the completed report back to the calling client.
This would allow the client to essentially do a fire and forget on the report request and call back later to check on it's status. Further it would mean that if IIS recycled for whatever reason you are still OK.
For bonus points I would add some error handling code that when the windows service restarted it could restart report jobs that were in the middle of execution. This would make it a bit more robust and allow you to reboot the server at any point.
I have disable also all the automatic shut down process from iis for my application with out any issue. I have monitor the memory limits and of course the program work smoothly with out any issues on memory.
I think that this automatic shut down triggers are designed mostly for process that keep too many web sites together and possible some of them have not good programming. But if you are the master of your iis, and you have check your program that have not memory issue, then is better to not shut it down, or at least control the shut down process with some way.
Ok is better to make long running process outside IIS, but is not so simple to develop it, not so simple to install it, not so simple to check it out.
I have a WCF service installed on IIS7. I noticed that the first call to my service is always very very slow. The subsequent calls are much faster & acceptable.
If there are no calls made to the service for some time, it again goes to sleep mode. After this the next call again takes a long long time.
Any remedies for this problem?
It is because of process management on IIS. When there are no calls for certain period of time IIS release the recourses and stops the process.
This is why you can notice that it is slow for first request and for requests after a long delay. Because while the first request or after long period of silence IIS loads everything from scratch. JIT complier runs and etc...
Also note :
When you are hosting WCF services on IIS, the WCF services enjoy all the features of ASP.NET applications. You have to be aware of these features because they can cause unexpected behavior in the services world. One of the major features is application recycling, including application domain recycling and process recycling. Through the IIS Management Console, you can configure different rules when you want the recycling to happen. You can set certain thresholds on memory, on time, and on the amount of processed requests. When IIS recycles a worker process, all the application domains within the worker process will be recycled as well
If you need automatic starting: The Windows Service Control Manager allows you to set the startup type to automatic, so that as soon as Windows starts, the service will be started, without an interactive logon on the machine. So you can use Windows service as a host.
More details you can check in Hosting and Consuming WCF Services.
There is another approach through which you can make it better. We have some kind of scehduled process which keeps hitting our server like every 5 mins with very light 'fetch' requests to keep all servers "hot" (by loading most of the required dlls etc) so that user experience is far better.
I agree it is not a fool proof way but still is something you can consider apart from increasing the recycling settings in IIS.
This should be a simple question but I haven't managed to find the answer on google.
I would like to know, in terms an idiot can understand, exactly what application lifetime means in ASP.NET (and therefore when you can expect application start and end events to run).
I assumed it would be when you run and stop the app in IIS, but I've read things that suggest it's related to number of requests.
By default the lifetime starts with the first request to the app. And it ends after an idle timeout.
But this is configurable based on various things (including request count) in IIS.
And IIS7.5 has the ability to start an application when IIS starts, rather than waiting for the first request.
You do have to consider how the Application Pool that your site is running in is configured. Applications can be dumped in a pool with other apps or it can have its own. The pool can be restarted based on memory usage beyond a certain point, by a timer so to speak (reset daily at 3am for example) and I believe by a number of requests beyond a certain configurable number. Not a super expert on IIS so verify before you buy ;-)