I am testing an asp.net web application iis7.Appserver- dual core,8gb, Webserver -dualcore 8gb.Running with 50 users for 1 hour.Requests/sec is 550 and CPU maxes out. Requests queue & current are 4 and 21 on an average. We run on default configuration. Number of logical and physical threads on webserver are 55 and 48 on an avg.Private bytes consumed went up to 0.4gb by end of load test.# of Exceps thrown/sec increases but errors total/sec is 0. Cache total turnover rate is high.% GC time is 4.73 on an avg.There are no errors in the load test and no. of passed transactions are also good.The concerns that we have are:
-How to improve the response time
-Should we limit the request/sec and are we stressing the server coz of high no. of requests
-Is 50 users high for the current configuration.We dont have any biz requirement or SLAs as of now.
I changed the process config to he following and that improved the resp time by 1-4 secs.
memoryLimit="60"
autoConfig="false"
maxWorkerThreads="100"
maxIoThreads="100"
minWorkerThreads="40"
minIoThreads="30"
Any insights will be appreciated.
Related
We are currently using an M2 Large server from Amazon.
In the Advanced Settings of an application pool in IIS 10, we are using these options,
Idle Time-out Action: Suspend
Idle Time-out (minutes): 1
Start Mode: OnDemand
Recycling Time Interval (minutes) : 1740
When the application pool is in suspended mode the next day, the startup of the application is still slow. Could we improve the startup while using these settings?
Your Recycling Time Interval is 1740 minutes or 29 hours. From https://weblogs.asp.net/owscott/why-is-the-iis-default-app-pool-recycle-set-to-1740-minutes ,
However, since you likely know your environment, it’s best to change this. I recommend setting to a fixed time like 4:00am if you’re on the East coast of the US, 1:00am on the West coast, or whatever seems to make sense for your audience when you have the least amount of traffic. Setting it to a fixed time each day during low traffic times will minimize the impact and also allow you to troubleshoot easier if you run into any issues. If you have multiple application pools it may be wise to stagger them so that you don’t overload the server with a lot of simultaneous recycles
You would probably have to use application initialization (see https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization) and increase your Idle Time Out as well to make sure your app is hot for the first user.
I am running a large scale ERP system on the following server configuration. The application is developed using AngularJS and ASP.NET 4.5
Dell PowerEdge R730 (Quad Core 2.7 Ghz, 32 GB RAM, 5 x 500 GB Hard disk, RAID5 configured) Software: Host OS is VMWare ESXi 6.0 Two VMs run on VMWare ESXi .. one is Windows Server 2012 R2 with 16 GB memory allocated ... this contains IIS 8 server with my application code Another VM is also Windows Server 2012 R2 with SQL Server 2012 and 16 GB memory allocated .... this just contains my application database.
You see, I separated the application server and database server for load balancing purposes.
My application contains a registration module where the load is expected to be very very high (around 10,000 visitors over 10 minutes)
To support this volume of requests, I have done the following in my IIS server -> increase request queue in application pool length to 5000 -> enable output caching for aspx files -> enable static and dynamic compression in IIS server -> set virtual memory limit and private memory limit of each application pool to 0 -> Increase maximum worker process of each application pool to 6
I then used gatling to run load testing on my application. I injected 500 users at once into my registration module.
However, I see that only 40% / 45% of my RAM is being used. Each worker process is using only a maximum amount of 130 MB or so.
And gatling is reporting that around 20% of my requests are getting 403 error, and more than 60% of all HTTP requests have a response time greater than 20 seconds.
A single user makes 380 HTTP requests over a span of around 3 minutes. The total data transfer of a single user is 1.5 MB. I have simulated 500 users like this.
Is there anything missing in my server tuning? I have already tuned my application code to minimize memory leaks, increase timeouts, and so on.
There is a known issue with the newest generation of PowerEdge servers that use the Broadcom Network Chip set. Apparently, the "VM" feature for the network is broken which results in horrible network latency on VMs.
Head to Dell and get the most recent firmware and Windows drivers for the Broadcom.
Head to VMWare Downloads and get the latest Broadcom Driver
As for the worker process settings, for maximum performance, you should consider running the same number of worker processes as there are NUMA nodes, so that there is 1:1 affinity between the worker processes and NUMA nodes. This can be done by setting "Maximum Worker Processes" AppPool setting to 0. In this setting, IIS determines how many NUMA nodes are available on the hardware and starts the same number of worker processes.
I guess the 1 caveat to the answer you received would be if your server isn't NUMA aware/uses symmetric processing, you won't see those IIS options under CPU, but the above poster seems to know a good bit more than I do about the machine. Sorry I don't have enough street cred to add this as a comment. As far as IIS you may also want to make sure your app pool doesn't use default recycle conditions and pick a time like midnight for recycle. If you have root level settings applied the default app pool recycling at 29 hours may also trigger garbage collection against your child pool/causing delays even in concurrent gc where it sounds like you may benefit a bit from Gcserver=true. Pretty tough to assess that though.
Has your sql server been optimized for that type of workload? If your data isn't paramount you could squeeze faster execution times with delayed durability, then assess queries that are returning too much info for async io wait types. In general there's not enough here to really assess for sql optimizations, but if not configured right (size/growth options) you could be hitting a lot of timeouts due to growth, vlf fragmentation, etc.
I am load testing an .Net 4.0 MVC application hosted on IIS 7.5 (default config, in particular processModel autoconfig=true), and am observing odd behavior in how .Net manages the threads.
http://msdn.microsoft.com/en-us/library/0ka9477y(v=vs.100).aspx mentions that "When a minimum is reached, the thread pool can create additional threads or wait until some tasks complete".
It seems the duration that threads are blocked for, plays a role in whether it creates new threads or waits for tasks to complete. Not necessarily resulting in optimal throughput.
Question: Is there any way to control that behavior, so threads are generated as needed and request queuing is minimized?
Observation:
I ran two tests, on a test controller action, that does not do much beside Thread.Sleep for an arbirtrary time.
50 requests/second with the page sleeping 1 second
5 requests/second with the page sleeping for 10 seconds
For both cases .Net would ideally use 50 threads to keep up with incoming traffic. What I observe is that in the first case it does not do that, instead it chugs along executing some 20 odd requests concurrently, letting the incoming requests queue up. In the second case threads seem to be added as needed.
Both tests generated traffic for 100 seconds. Here are corresponding perfmon screenshots.
In both cases the Requests Queued counter is highlighted (note the 0.01 scaling)
50/sec Test
For most of the test 22 requests are handled concurrently (turquoise line). As each takes about a second, that means almost 30 requests/sec queue up, until the test stops generating load after 100 seconds, and the queue gets slowly worked off. Briefly the number of concurrency jumps to just above 40 but never to 50, the minimum needed to keep up with the traffic at hand.
It is almost as if the threadpool management algorithm determines that it doesn't make sense to create new threads, because it has a history of ~22 tasks completing (i.e. threads becoming available) per second. Completely ignoring the fact that it has a queue of some 2800 requests waiting to be handled.
5/sec Test
Conversely in the 5/sec test threads are added at a steady rate (red line). The server falls behind initially, and requests do queue up, but no more than 52, and eventually enough threads are added for the queue to be worked off with more than 70 requests executing concurrently, even while load is still being generated.
Of course the workload is higher in the 50/sec test, as 10x the number of http requests is being handled, but the server has no problem at all handling that traffic, once the threadpool is primed with enough threads (e.g. by running the 5/sec test).
It just seems to not be able to deal with a sudden burst of traffic, because it decides not to add any more threads to deal with the load (it would rather throw 503 errors than add more threads in this scenario, it seems). I find this hard to believe, as a 50 requests/second traffic burst is surely something IIS is supposed to be able to handle on a 16 core machine. Is there some setting that would nudge the threadpool towards erring slightly more on the side of creating new threads, rather than waiting for tasks to complete?
Looks like it's a known issue:
"Microsoft recommends that you tune the minimum number of threads only when there is load on the Web server for only short periods (0 to 10 minutes). In these cases, the ThreadPool does not have enough time to reach the optimal level of threads to handle the load."
Exactly describes the situation at hand.
Solution: Slightly increased the minWorkerThreads in machine.config to handle expected traffic burst. (4 would give us 64 threads on the 16 core machine).
We have an asp.net 4.0 (integrated mode) web application that runs on iis 7.5 x64 (w2k8) with 12 GB ram and 16 cores that has problem with spikes of requests queued. Normally the queue is zero, but occasionally (probably aroud 15 times over a 10 minute period) the queue spikes up to about 20-100 in the queue.
Sometimes this queue also correlate with a higher number of requests/sec. But that isn't always the case.
Requests current seems to always be between 15-30.
nbr of current logical and physical threads is as low as 60-100
CPU load is avg of 6%
requests/sec is around 150-200
connections active seems to be slowly increasing. It's about 7000.
connections established seems faily consitent around 130-140.
Since we are running .net 4.0 in integrated mode I suppose that we should be able to handle up to 5000 simultanously requests, or atleast 1000 (http.sys kernel)
http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx
What could be causing .net to queue the requests even though there are threads left and requests/sec is low?
Just a guess: Garbage collection suspends all threads, so perhaps the period immediately following the garbage collection would look like a request spike, since IIS would be piling up requests during the GC. Can you correlate the spikes with garbage collections? If your application is I/O-bound, it may not be possible to drive the CPU load very high, since the threads will spend most of their time blocked.
The apparent leak of active connections is disturbing though, if it really is ever-increasing.
We have a vanilla ASP.NET application (ASP.NET web forms, Entity Framework, SQL Server 2005) without any explicit multithreading from code. It has been deployed on the stagging environment (OS - Windows Server 2008 R2 64 bit, CPU - Intel Xeon E5507 # 2.27 GHz 2.34 GHz, RAM - 7.5 GB). This environment is composed of a web, a database and a reporting server each a separate instance in the cloud (Amazon EC2). When testing for concurrency, observations are as under:
1 user - CPU usage ~25%, Response time 2-4 seconds
2 users - CPU usage 40-50%, Response time 3-6 seconds
4 users - CPU usage 60-80%, Response time 4-8 seconds
8 users - CPU usage 80-100%, Response time 4-10 seconds
My questions are:
Is CPU usage is relative to no. of concurrent users? And the response time can vary to a great extent as seen in the above observations?
As from the above observations, CPU will be maxed out when concurrent user counts is ~10. Shouldn't the CPU handle much more concurrent users seamlessly without a drastic increase in the response time? In ideal scenario, in case of a basic ASP.NET application, how many concurrent users a CPU can handle?
If yes in the above question, what could be the problem here for high CPU/long response time? What ways we should go ahead for effective debugging to find out bottlenecks in code/IIS settings?
PS: IIS settings (i.e. in machine.config) which have been changed:
maxWorkerThreads = 100
MinWorkerThreads = 50
maxIOThreads = 100
minIOThreads = 50
minFreeThreads = 176
maxConnections = 100
The high CPU usage could be caused by a wide variety of things. The easiest way to find out what's going on is to use a profiling tool:
ANTS Memory Profiler:
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/
ANTS Performance Profiler:
http://www.red-gate.com/products/dotnet-development/ants-performance-profiler/
They're very affordable, but you should be able to work unrestricted with the trial versions. These tools do a fantastic job of identifying bottlenecks and memory leaks.