I have an ASP.NET web service that I am hosting on IIS 6 (may change to IIS 7 in the future). The .asmx page may receive many requests at the same time. It takes approximately 3s per request per CPU to return a response after a request is received (so two requests will also come back in 3s on a dual-core). However, when multiple requests come in at once (or close enough), the service seems to try to make it look like it is processing all of them at the same time. For example, if 6 requests come in, they all return in around 9s instead of the first two coming back in 3s, the next 2 in 6s, and the final 2 in 9s. My questions are: What is going on (briefly or elaborately if you have the patience :)), and how can I limit the number of requests or threads created from the service point-of-view, preferably without making any changes to IIS or machine.config?
Thanks in advance!
EDIT:
Just to clarify, I'm trying to make the web service perform as 1st in 1st out sort of situation, where the first n requests are processed (n = number of processors), then the next ones. Right now, if I send 10 requests at once, the service gathers all of them together and splits the processing up among all processors. It seems to me that in theory, if I can tell the service to limit the number of simultaneous processing to n (corresponding to the # of processors), then I will achieve my goal. But I don't know how to do this.
In Pro ASP.NET Web API: HTTP Web Services in ASP.NET book -
With .NET Framework 4.0, the default configuration settings are
suitable for most scenarios. For example, in ASP.NET 4.0 and 4.5, the
MaxConcurrentRequestsPerCPU value is set to 5000 by default (it had
been a very low number in earlier versions of .NET). According to team
members of IIS, there’s nothing special about the value 5000. It was
set because it is a very large number and will allow plenty of async
requests to execute concurrently. This setting should be fine, so
there is no need to change it.
Tip Windows 7, Windows Vista, and all other Windows client operating
systems handle a maximum of 10 concurrent requests. You need to have a
Windows Server operating system or use IIS Express to see the benefits
of asynchronous methods under high load.
Related
I have a site that is running on the 4.5 framework which uses Task.Run during the pre-init event to fetch and process multiple pieces of data in an effort to reduce the response time. I'm seeing an avg response time under 500ms that is pretty consistent over the course of the day however every few mins I see response times for individual requests that hit anywhere from 5000ms - 30000ms (which is the configured page request timeout). The machine.config is set with autoConfig = true in the process model section with no overrides for min/max for either WorkerThreads or IOThreads in either the machines config or web config.
I'm referencing these documents for some details on how autoconfig works and tuning.
asp-net-thread-usage-on-iis-7-5-iis-7-0-and-iis-6-0
how-to-tune-machine-config-settings-for-improved-performance
I'm trying to determine if I'm running out of ASP.net WorkerThreads or IoThreads and nothing stands out in the Performance Counters to be able to see this.
Is there a way to see what the running values of the Min/MaxWorkerThreads and Min/MaxIoThreads are? Are there counters on how many Worker and IO threads are being consumed as requests come in?
Server Details:
4 Core Intel Xeon Platinum 8000 series
8 GB Memory
Windows Server 2016
There is an ASP.NET application with a set of web services, and there is a WinForms application that calls these web services.
The w3wp.exe (which handles asmx calls) run special processes that do a really "hard" jobs (like zipping, xml parsing, etc.). It is a long story why we need it to be done in separate processes and why we don't run this code inside w3wp.exe - this doesn't matter here. It must be like that.
If multiple requests came simulteneously from many users, there can be 30-40 such processes, and they consume CPU. They consume it so much that almost no CPU resources is left for IIS itself.
So, the problem is next: when the following HTTP requests are coming to this IIS, IIS cannot handle them: it cannot even pass them to w3wp.exe because server is "bombed". And, as a result, in peak time 30 users have run their queries, and 31st user is getting a WebException in the WinForms client "timeout expired".
I have found a lot of articles in the internet which explain how to tweak IIS, but noone take into account the fact that OTHER processes and/or applications can run on the same machine. And noone tells what to do to make so that IIS is of higher priority.
So, the question is next: is there a way to "explain" IIS that it should start by default the apppool (svchost.exe) and w3wp.exe with hiher priority?
I faced this issue in Windows Server 2008 r2.
I had a windows service (which used parallel processing) which used to bloat up my CPU a lot & left nothing for IIS & SQL Server.
So I set processor affinity to the windows service to consume only specific processors.
You get in Task Manager->RightClick Process -> Set Affinity.
This left other processors free to use for SQL & IIS.
To an extent, It decreased the productivity of the windows service which was now working with a limited set of processors but it did not matter that much.
I have faced strange behavior of my ASP.Net application on the server (IIS7 on Windows Server 2008 x64, processor Xeon Quad).
The web application is the simple page which about one second calculates some math, and then displays result. That is it almost does not consume a network, a disk, memory, but completely uses processor resources.
The following phenomenon appears at load testing: IIS7 utilize processor no more than on 25% and not for the world does not wish to utilize it more. This 25% are equal to one core, but spread out on all four according to task manager performance tab. On the other computer (IIS7, Win 7, Quad) all works as well as should: the processor is utilized on all of 100%.
For each of behavior variants (peak loading of 25% and 100% on 4 core processors) I have found on 2 computers. The similar situation is described here. What can cause such behavior?
This 25% are equal to one core, but
spread out on all four according to
task manager performance tab.
Reality check: when you use up one core, the CPU scheduler will move the load between cores before 2008 R2. Staring with 2008 R2 it will keep it on one core to actually move the other cores into deep sleep.
So, what you see is basically an application that uses one CPU core. Point.
What can cause such behavior?
Either your code, or your request generation (well, together with your code) make sure that the requests are serialized and not handled in parallel.
During load testing... do you accept / keep the session cookie (like: ONE) and in your asp.net page do you have session state enabled? This would serialize all page requests to the one session in memory and is one very likely culprit. Another one is doing "stupid" things in code the result in a block and make the algorithm effectively single threaded - but this can no be evaluated without a lot more information from you on how you program and what you effectively do. For example I have seen a bunch of monkeys once code an online shop using ONE database connection (to not overload the database) that was kept in the aplication object and using the lock / unlock methods there to effectively turn their asp application into a single threaded thing. This was obvious - but there are a lot of other things that can go wrong. The questions basically are:
Are you by configuration / test scenario doing something to force IIS to serialize (which would be among web farming settings or bad usage of session state)?
Do you do anything within the pages that is effectively locking them to single threaded?
IIS per se answers requests through work items (i.e. uses a LOT of threads) unless it HA to serialize them (sessions only are ever assigned to ONE thread at a time, so a second request for the same session is serialized).
I doubt it's spread out. More likely the algorithm is not parallelised and so the code runs in a single core.
I have understood, that on those 2 computers where loading was 100%, 32 bit Windows has been installed. On the same 2 computers, where peak loading was 25% - 64 bit. But customisation change "Enable 32 bit applications"=true has not helped.
If your server is using multiple worker processes and you are sure that your load testing software is issuing requests in parallel, then something in your application is likely becoming serial.
This is actually pretty common (we do a lot of load testing for our customers) - it could be as simple as a database pool with a size of one or as complex as some shared resource being locked at some level deep within the application or within a library the application is using. We've seen cases where the first step in serving an application page opens a transaction that is not committed until the page is done. If that transaction is locking a table that is needed for the same purpose by every other page, then only one page request can be serviced at a time.
Good luck hunting down the problem - be sure to let us know what you find!
The problem has been solved after installation of fresher OS. "Windows Server 2008 Enterprise SP1 (c) 2009" instead of "Windows Server 2008 Standard SP2 (c) 2007".
This should be a simple question but I haven't managed to find the answer on google.
I would like to know, in terms an idiot can understand, exactly what application lifetime means in ASP.NET (and therefore when you can expect application start and end events to run).
I assumed it would be when you run and stop the app in IIS, but I've read things that suggest it's related to number of requests.
By default the lifetime starts with the first request to the app. And it ends after an idle timeout.
But this is configurable based on various things (including request count) in IIS.
And IIS7.5 has the ability to start an application when IIS starts, rather than waiting for the first request.
You do have to consider how the Application Pool that your site is running in is configured. Applications can be dumped in a pool with other apps or it can have its own. The pool can be restarted based on memory usage beyond a certain point, by a timer so to speak (reset daily at 3am for example) and I believe by a number of requests beyond a certain configurable number. Not a super expert on IIS so verify before you buy ;-)
I've got an asp.net web page that is making 7 async requests to a WCF service on another server. Both boxes are clean without anything else installed.
I've also increased maxconnections in web.config to 20.
I run a single call through the system and the page returns in 800ms. The long and short of it is I think that the threadpool is being being overwhelmed as, once placed underload I cannot get more that 8 requests per second, even though both quad core boxes are running at 20% CPU load and the sql server it's connected to is returning the querys in under 10ms per call.
I've changed the service behaviour to concurrency.multiple but that's not seeming to help.
Any ideas anyone.
There are many different factors that could be in play here. Taking a stab at the remark that changing your instancing model on the service had zero effect (big IF here) then its possible the 'bottleneck' is upstream from the service. Either at the web server, or the client load generator.
You've got several areas to review for tuning: client, web server, wcf service server - that's assuming there are no network devices in the middle. Pick an end and work towards the other end. Since I'm already making an assumption that its not the service, then I'd start at the client and work my way towards the wcf service.
Client
What machine is driving the load against the web server? A laptop? A desktop? A dedicated test agent, or a shared one? The client acting as the load generator for purposes of this test is also susceptible to maxConnections limitation as this is a client setting.
What is the CPU utilization of the client generating load? Could it be that the test driver is just unable to generate enough load to push these boxes? Can you add additional test clients to your test?
Web Server
What does the system.net/processModel element look like in machine.config on the ASP.NET web server? Try setting autoConfig = true. This will allow the configuration to auto size based on the 'size' of the machine its running on.
WCF Service
Review WCF service for any throttling defaults that might be in play and tweak appropriately. See ServiceThrottlingBehavior on MSDN.
Let us know any changes in behavior you might observe (if any) if you make any changes!
The real answer here that everyone missed is that you're using an ASP.NET web page. That means your client is some form of web browser. Modern web browsers have a limit of 2 concurrent async requests at any time. This means that 5 of your requests were queued up and waiting for the first two to finish. Once those first two, it served the next two, then the next two, then the last one.
All of these round-trips and handshakes simply take time. I'm guessing that your roundtrip time is around 200ms, unfortunately you have to do it 4 times.
I also really dislike the "max 2" browser limitation on making webservice calls.
Is this service hosted in IIS, WAS or a Windows Service?
You should try to set Windows to run services on a higher priority. Your WCF Service is probably creating the threads it needs but they should be running at a low priority.
Hope that helps.