There is an ASP.NET application with a set of web services, and there is a WinForms application that calls these web services.
The w3wp.exe (which handles asmx calls) run special processes that do a really "hard" jobs (like zipping, xml parsing, etc.). It is a long story why we need it to be done in separate processes and why we don't run this code inside w3wp.exe - this doesn't matter here. It must be like that.
If multiple requests came simulteneously from many users, there can be 30-40 such processes, and they consume CPU. They consume it so much that almost no CPU resources is left for IIS itself.
So, the problem is next: when the following HTTP requests are coming to this IIS, IIS cannot handle them: it cannot even pass them to w3wp.exe because server is "bombed". And, as a result, in peak time 30 users have run their queries, and 31st user is getting a WebException in the WinForms client "timeout expired".
I have found a lot of articles in the internet which explain how to tweak IIS, but noone take into account the fact that OTHER processes and/or applications can run on the same machine. And noone tells what to do to make so that IIS is of higher priority.
So, the question is next: is there a way to "explain" IIS that it should start by default the apppool (svchost.exe) and w3wp.exe with hiher priority?
I faced this issue in Windows Server 2008 r2.
I had a windows service (which used parallel processing) which used to bloat up my CPU a lot & left nothing for IIS & SQL Server.
So I set processor affinity to the windows service to consume only specific processors.
You get in Task Manager->RightClick Process -> Set Affinity.
This left other processors free to use for SQL & IIS.
To an extent, It decreased the productivity of the windows service which was now working with a limited set of processors but it did not matter that much.
Related
for IIS7
Does a webapplication run faster when Maximum Worker Processes is more than one?
By increasing the Maximum Worker Processes over 1 you're creating a Web Garden. So the short answer is: likely no... unless:
To quote Chris Adams an ex IIS PM's article I have flowers... should I get a Web Garden?:
Web gardens was designed for one single reason – Offering applications that are not CPU-bound but execute long running requests the ability to scale and not use up all threads available in the worker process.
The examples might be things like -
Applications that make long running database requests (e.g. high computational database transaction)
Applications that have threads occupied by long-running synchronous and network intensive transactions
The question that you must ask yourself -
What is the current CPU usage of the server?
What is the application’s threads executing and what type of requests are they?
Based on the criteria above, you should understand better when to use Web Gardens. Web Gardens, in the metabase, equals the MaxProcesses metabase property if you are not using the User Interface to configure this feature.
cscript adsutil.vbs set w3svc/apppools/defaultapppool/maxprocesses 4
I hope that I get some mileage out of having this blog and more importantly I hope it helps you understand this better…
You may want to look at "What is Web Garden?" from Deploying ASP.NET Websites on IIS 7.0 [codeproject.com] which says:
By default each Application Pool runs with a Single Worker Process (W3Wp.exe). We can assign multiple Worker Processes With a Single Application Pool. An Application Pool with multiple Worker process is called "Web Gardens". Many worker processes with the same Application Pool can sometimes provide better throughput performance and application response time. And each worker process should have their own Thread and Own Memory space.
WebGarden is faster than single worker process in case if application contains locks that prevent its parallelization. For example, GDI+ based image processing.
See this and this questions for more info.
My environment: Windows Server 2008, IIE 7.0, ASP.NET
I developed a Silverlight client.
This client gets updates from the ASP.NET host through a WCF web service.
We get 100% CPU usage and connection drops when we have a very low number of users (~50).
The server should clearly be able to handle a lot more than that.
I ran some tests on our DEV server and indeed 100 requests / s maxes out the CPU.
What's odd is that even if the service is replaced by a dummy sending back hardcoded data the service still maxes out the CPU.
The thread count looked very low at about 20 so I thought there was some contention somewhere.
I changed all configuration options I could find to increase the worker threads (processModel, httpRuntime and the MaxRequestsPerCPU registry entry).
Nothing changed.
Then I stopped the IIS server and ran the web service as a console (removing all the ASP authentication references).
The service maxed out the CPU as well.
Then comes the magic part: I killed the console app and restarted IIS and now the service runs a 5-60% CPU with 100 requests / s and I can see 50+ worker threads.
I did the same thing on our preprod machine and had the same magic effect.
Rebooting the machines keeps the good behaviour.
So my question is: what happened to fix my IIS server? I really can't understand what fixed it.
Cheers.
Find out the root cause of the high CPU usage, and then you can find a fix,
http://support.microsoft.com/kb/919791
I have a WCF service hosted in IIS which takes a long time (around 5 hours) to execute. the WCF service basically generates some reports using SSRS (SQL server reporting services) and saves them to a locaton on the server. this service was actually stopping after generating few reports, so I disabled the "recycling of worker process", "shut down worker processes after being idle" and "limit kernel request queue" in application pool and that fixed the issue and all the reports were generated regardless of the amount taken to generate them. but I am not sure if this is the right fix for this and I would like to know what is the impact of unchecking these settings in application pool for the WCF service in IIS? and is there any better way to get around this problem?
For any long running process it is much better to do it outside of IIS.
In this case I would have a regular windows service running that monitored a request queue. When a request comes in to generate a report, it would then spin off a thread to perform the generation.
The web service would be responsible for 3 things. First, adding an item to the queue to be handled. Second, checking status on the queue as to whether the report is ready. Third, sending the completed report back to the calling client.
This would allow the client to essentially do a fire and forget on the report request and call back later to check on it's status. Further it would mean that if IIS recycled for whatever reason you are still OK.
For bonus points I would add some error handling code that when the windows service restarted it could restart report jobs that were in the middle of execution. This would make it a bit more robust and allow you to reboot the server at any point.
I have disable also all the automatic shut down process from iis for my application with out any issue. I have monitor the memory limits and of course the program work smoothly with out any issues on memory.
I think that this automatic shut down triggers are designed mostly for process that keep too many web sites together and possible some of them have not good programming. But if you are the master of your iis, and you have check your program that have not memory issue, then is better to not shut it down, or at least control the shut down process with some way.
Ok is better to make long running process outside IIS, but is not so simple to develop it, not so simple to install it, not so simple to check it out.
I have a WCF service installed on IIS7. I noticed that the first call to my service is always very very slow. The subsequent calls are much faster & acceptable.
If there are no calls made to the service for some time, it again goes to sleep mode. After this the next call again takes a long long time.
Any remedies for this problem?
It is because of process management on IIS. When there are no calls for certain period of time IIS release the recourses and stops the process.
This is why you can notice that it is slow for first request and for requests after a long delay. Because while the first request or after long period of silence IIS loads everything from scratch. JIT complier runs and etc...
Also note :
When you are hosting WCF services on IIS, the WCF services enjoy all the features of ASP.NET applications. You have to be aware of these features because they can cause unexpected behavior in the services world. One of the major features is application recycling, including application domain recycling and process recycling. Through the IIS Management Console, you can configure different rules when you want the recycling to happen. You can set certain thresholds on memory, on time, and on the amount of processed requests. When IIS recycles a worker process, all the application domains within the worker process will be recycled as well
If you need automatic starting: The Windows Service Control Manager allows you to set the startup type to automatic, so that as soon as Windows starts, the service will be started, without an interactive logon on the machine. So you can use Windows service as a host.
More details you can check in Hosting and Consuming WCF Services.
There is another approach through which you can make it better. We have some kind of scehduled process which keeps hitting our server like every 5 mins with very light 'fetch' requests to keep all servers "hot" (by loading most of the required dlls etc) so that user experience is far better.
I agree it is not a fool proof way but still is something you can consider apart from increasing the recycling settings in IIS.
I have a ASP.NET web application running on an IIS6 server. The application is making potentially long running calls to a xml service on a remote machine. Some of the service calls on the remote machine are taking a long time to execute (sometimes up to 4 minutes). The long term solution would be to make the calls asyncronous, but as a short term solution we want to increase the timeout for the calls and the overall httpRequest time out.
My fear with this is that the long running calls will fill up the request queue and prevent the "normal" page requests from completing. How can the server, IIS and application settings be tuned to temporarely resolve the issue?
Currently there are approximately 200 page requests/minute and this results in 270 service requests/minute.
The current executionTimeout is 360 (6 minutes)
The current service call time out is 2 minutes
There's this article on Microsoft's Knowledgebase that has some pretty much all the information you might need:
* Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
I will give you some research I have done regarding some of the specific items handled in the article above. This information below applies to IIS6, comments for IIS7 where applicable.
Increase the Processor Worker Thread pool from 25 to at least 100
The default values for the Threadpool size is 100 because the default value for autoConfig is true.
The values covered by autoConfig is
maxWorkerThreads
maxIoThreads
maxConnection
There is one value that is still 25 that must change is – ASPProcessorThreadMax, this can only be set in the IIS metabase (via adsutil tool) in IIS6. [IIS7’s equivalent is the processorThreadMax value]
So I'm opting not to change the machine.config settings as they are fine and there are other paramaters that would be affected by turning off autoconfig, but rather change ASPProcessorThreadMax from 25 to 100 via the IIS metabase (the only way to change this value).
e.g.
cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\<nowiki>adsutil.vb</nowiki>s SET W3SVC/AspRequestQueueMax 100
Max connections per server
maxconnection
The autoconfig sets this value to 12*number of cpu’s, that’s how many connections can be made to each address you are connecting to at one time.
Debugging
Here are some things you can do:
Monitor if requests are waiting in the queue
Monitor the following counter:
Run perfmon
Add counter: ASP.NET Applications/Requests In Application Queue
This will show us if work items are queued because of a shortage of workers.
Check Identity Used by Application Pool
Open IIS Manager
Check which application pool is used by your site in IIS manager.
Choose the Application pool being used in the Application Pools list, then Right Click -> properties and see what account identity is being used.
It should be Network Service by default.