I have a site that is running on the 4.5 framework which uses Task.Run during the pre-init event to fetch and process multiple pieces of data in an effort to reduce the response time. I'm seeing an avg response time under 500ms that is pretty consistent over the course of the day however every few mins I see response times for individual requests that hit anywhere from 5000ms - 30000ms (which is the configured page request timeout). The machine.config is set with autoConfig = true in the process model section with no overrides for min/max for either WorkerThreads or IOThreads in either the machines config or web config.
I'm referencing these documents for some details on how autoconfig works and tuning.
asp-net-thread-usage-on-iis-7-5-iis-7-0-and-iis-6-0
how-to-tune-machine-config-settings-for-improved-performance
I'm trying to determine if I'm running out of ASP.net WorkerThreads or IoThreads and nothing stands out in the Performance Counters to be able to see this.
Is there a way to see what the running values of the Min/MaxWorkerThreads and Min/MaxIoThreads are? Are there counters on how many Worker and IO threads are being consumed as requests come in?
Server Details:
4 Core Intel Xeon Platinum 8000 series
8 GB Memory
Windows Server 2016
Related
I am just optimizing the WebService performance of my ASP.NET website.
The services from user's point of view feel fast enough but Stackify Retrace shows me that most WebService-Calls go to the queue for about 500ms before being executed.
The site serves only about 20-30 WCF-requests/s on a dedicated 8 core + HT machine. CPU and memory is healthy: There should be no reason at all to queue those requests.
I've noticed that queueing become worse when I started to use WebSockets more extensively. I moved a lot of WCF traffic to WebSockets, but the average execution time for the WCF-calls has increased (because of the queueing)
I've played around a lot with these settings:
maxWorkerThreads
minWorkerThreads
maxIoThreads
minFreeThreads
minLocalRequestFreeThreads
maxconnection
executionTimeout
No luck. By now I learned that those are relevant mostly for .NET 2 and have intelligent defaults in .NET 4 anyways. That explains why all my meddling didn't have any effect.
Question: Are there any relevant performance counters which would help me find out why so many requests are getting queued?
Are WebSockets/WebServices using some shared resources?
I understand that IIS uses a thread from Threadpool to serve an incoming Http Request and releases the thread once after it completes serving the request.
I want to play around this to understand, how many threads are possible in an IIS server for a specific hardware configuration and how many threads it can handle concurrently, and more related scenarios.
I'm looking for the way/tools that will help visually monitor the thread allocation/usage # IIS.
I appreciate Any pointers/suggestions?
To see how many threads are possible in an IIS server for specific hardware, Click on the server. Then on the right side pane, double click on ASP like this.
The ASP Threads Per Processor Limit property specifies the maximum number of worker threads per processor that IIS creates. The default value of Threads Per Processor Limit is 25. The maximum recommended value for this property is 100.
To increase the value for the Threads Per Processor see this.
I have an ASP.NET web service that I am hosting on IIS 6 (may change to IIS 7 in the future). The .asmx page may receive many requests at the same time. It takes approximately 3s per request per CPU to return a response after a request is received (so two requests will also come back in 3s on a dual-core). However, when multiple requests come in at once (or close enough), the service seems to try to make it look like it is processing all of them at the same time. For example, if 6 requests come in, they all return in around 9s instead of the first two coming back in 3s, the next 2 in 6s, and the final 2 in 9s. My questions are: What is going on (briefly or elaborately if you have the patience :)), and how can I limit the number of requests or threads created from the service point-of-view, preferably without making any changes to IIS or machine.config?
Thanks in advance!
EDIT:
Just to clarify, I'm trying to make the web service perform as 1st in 1st out sort of situation, where the first n requests are processed (n = number of processors), then the next ones. Right now, if I send 10 requests at once, the service gathers all of them together and splits the processing up among all processors. It seems to me that in theory, if I can tell the service to limit the number of simultaneous processing to n (corresponding to the # of processors), then I will achieve my goal. But I don't know how to do this.
In Pro ASP.NET Web API: HTTP Web Services in ASP.NET book -
With .NET Framework 4.0, the default configuration settings are
suitable for most scenarios. For example, in ASP.NET 4.0 and 4.5, the
MaxConcurrentRequestsPerCPU value is set to 5000 by default (it had
been a very low number in earlier versions of .NET). According to team
members of IIS, there’s nothing special about the value 5000. It was
set because it is a very large number and will allow plenty of async
requests to execute concurrently. This setting should be fine, so
there is no need to change it.
Tip Windows 7, Windows Vista, and all other Windows client operating
systems handle a maximum of 10 concurrent requests. You need to have a
Windows Server operating system or use IIS Express to see the benefits
of asynchronous methods under high load.
We have a fairly popular site that has around 4 mil users a month. It is hosted on a Dedicated Box with 16 gb of Ram, 2 procc with 24 cores.
At any given time the CPU is always under 40% and the memory is under 12 GB but at the highest traffic we see a very poor performance. The site is very very slow. We have 2 app pools one for our main site and one for our forum. Only the site is being slow. We don't have any restrictions on cpu or memory per app pool.
I have looked at he Performance counters and I saw something very interesting. At our peek time for some reason Request are being queued. Overall context switching numbers are very high around 30 - 110 000 k.
As i understand high context switching is caused by locks. Can anyone give me an example code that would cause a high number of context switches.
I am not too concerned with the context switching, and i don't think the numbers are huge. You have a lot of threads running in IIS (since its a 24 core machine), and higher context switching numbers re expected. However, I am definitely concerned with the request queuing.
I would do several things and see how it affects your performance counters:
Your server CPU is evidently under-utilized, since you run below 40% all the time. You can try to set a higher value of "Threads per processor limit" in IIS until you get to a 50-60% utilization. An optimal value of threads per core by the books is 20, but it depends on the scenario, and you can experiment with higher or lower values. I would recommend trying setting a value >=30. Low CPU utilization can also be a sign of blocking IO operations.
Adjust the "Queue Length" settings in IIS properties. If you have configured the "Threads per processor limit" to be 20, then you should configure the Queue Length to be 20 x 24 cores = 480. Again, if the requests are getting Queued, that can be a sign that all your threads are blocked serving other requests or blocked waiting for an IO response.
Don't serve your static files from IIS. Move them to a CDN, amazon S3 or whatever else. This will significantly improve your server performance, because 1,000s of Server requests will go somewhere else! If you MUST serve the files from IIS, than configure IIS file compression. In addition use expire headers for your static content, so they get cached on the client, which will save a lot of bandwidth.
Use Async IO wherever possible (reading/writing from disk, db, network etc.) in your ASP.NET controllers, handlers etc. to make sure you are using your threads optimally. Blocking the available threads using blocking IO (which is done in 95% of the ASP.NET apps i have seen in my life) could easily cause the thread pool to be fully utilized under heavy load, and Queuing would occur.
Do a general optimization to prevent the number of requests that hit your server, and the processing time of single requests. This can include Minification and Bundling of your CSS/JS files, refactoring your Javascript to do less roundtrips to the server, refactoring your controller/handler methods to be faster etc. I have added links below to Google and Yahoo recommendations.
Disable ASP.NET debugging in IIS.
Google and Yahoo recommendations:
https://developers.google.com/speed/docs/insights/rules
https://developer.yahoo.com/performance/rules.html
If you follow all these advices, i am sure you will get some improvements!
I have a ASP.NET web application running on an IIS6 server. The application is making potentially long running calls to a xml service on a remote machine. Some of the service calls on the remote machine are taking a long time to execute (sometimes up to 4 minutes). The long term solution would be to make the calls asyncronous, but as a short term solution we want to increase the timeout for the calls and the overall httpRequest time out.
My fear with this is that the long running calls will fill up the request queue and prevent the "normal" page requests from completing. How can the server, IIS and application settings be tuned to temporarely resolve the issue?
Currently there are approximately 200 page requests/minute and this results in 270 service requests/minute.
The current executionTimeout is 360 (6 minutes)
The current service call time out is 2 minutes
There's this article on Microsoft's Knowledgebase that has some pretty much all the information you might need:
* Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
I will give you some research I have done regarding some of the specific items handled in the article above. This information below applies to IIS6, comments for IIS7 where applicable.
Increase the Processor Worker Thread pool from 25 to at least 100
The default values for the Threadpool size is 100 because the default value for autoConfig is true.
The values covered by autoConfig is
maxWorkerThreads
maxIoThreads
maxConnection
There is one value that is still 25 that must change is – ASPProcessorThreadMax, this can only be set in the IIS metabase (via adsutil tool) in IIS6. [IIS7’s equivalent is the processorThreadMax value]
So I'm opting not to change the machine.config settings as they are fine and there are other paramaters that would be affected by turning off autoconfig, but rather change ASPProcessorThreadMax from 25 to 100 via the IIS metabase (the only way to change this value).
e.g.
cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\<nowiki>adsutil.vb</nowiki>s SET W3SVC/AspRequestQueueMax 100
Max connections per server
maxconnection
The autoconfig sets this value to 12*number of cpu’s, that’s how many connections can be made to each address you are connecting to at one time.
Debugging
Here are some things you can do:
Monitor if requests are waiting in the queue
Monitor the following counter:
Run perfmon
Add counter: ASP.NET Applications/Requests In Application Queue
This will show us if work items are queued because of a shortage of workers.
Check Identity Used by Application Pool
Open IIS Manager
Check which application pool is used by your site in IIS manager.
Choose the Application pool being used in the Application Pools list, then Right Click -> properties and see what account identity is being used.
It should be Network Service by default.