I've begun to track my asp.net application metrics, but for Servers -> Process CPU (avg) I'm getting values above 100% (194% or more). What does that mean?
Probably it means it's a multi-threaded process that's keeping 1.94 CPUs busy, on average.
Related
Should I take in consideration CPU utilization, network traffic or http response time checks? I've run some tests with Apache AB (from the same server - eq: ab -k -n 500000 -c 100 http://192.XXX.XXX.XXX/) - and I monitored the load average. Even if the load was between 1.0 - 1.50(one core server), "time per request"(mean) was pretty solid, 140ms for a simple dynamic page with one set/get Redis operation. Anyway, I'm confused as the general advice is to launch a new instance when you pass the 70% CPU utilization threshold.
70% CPU utilization is a good rule of thumb for CPU-bound applications like nginx. CPU time is kind of like body temperature: it actually hides a lot of different things, but is a good general indicator of health. Load average is a separate measure of how many processes are waiting to be scheduled. The reason the rule is 70% (or 80%) utilization is that, past this point, CPU-bound appliations tend to suffer contention-induced latency and non-linear performance.
You can test this yourself by plotting throughput and latency (median and 90th percentile) against CPU utilization on your setup. Finding the inflection point for your particular system is important for capacity planning.
A very good writeup of this phenomenon is given in Facebook's original paper on Dyno, their system for measuring throughput of PHP under load.
https://www.facebook.com/note.php?note_id=203367363919
we have requirement to handle 10000 concurrent user.
Let me explain the system. Machine has two processors. ProcessModel in machine.config is set as autoconfig = true. so that makes maxWorkerThreads = 20.
When I Load run my case with 30 users and watch CPU usage it is maximing to 100. and number of threads on w3wp.exe is more then 70. As my default is 20 * 2 (CPU's) = 40.
Once cpu touches 100% most of the transaction fail or talking maximum time to respond
Now questions
1. how do i get 30 more threads assigned to the same workerprocess?
2. How can reduce CPU usage here?
You have a bit of an issue here. Increasing the # of threads will further increase CPU usage. (Your 2 goals are incompatible.) Not only are you asking each CPU to do more, but you'll have additional overhead with context switching.
You could investigate using a non-blocking IO library, which would essentially mean only 1-2 threads per CPU. However, this could be a significant architecture change to your project (probably not feasible) - and what you might actually find is that most of the CPU was actually spent due to the work your code is performing, and not because of anything threading-related.
It sounds like you need to do some performance tuning and optimization of your application.
You should take a look at making async calls so that your threads are not remaining active while the caller is waiting for a response.
http://msdn.microsoft.com/en-us/magazine/cc163463.aspx
We have an asp.net 4.0 (integrated mode) web application that runs on iis 7.5 x64 (w2k8) with 12 GB ram and 16 cores that has problem with spikes of requests queued. Normally the queue is zero, but occasionally (probably aroud 15 times over a 10 minute period) the queue spikes up to about 20-100 in the queue.
Sometimes this queue also correlate with a higher number of requests/sec. But that isn't always the case.
Requests current seems to always be between 15-30.
nbr of current logical and physical threads is as low as 60-100
CPU load is avg of 6%
requests/sec is around 150-200
connections active seems to be slowly increasing. It's about 7000.
connections established seems faily consitent around 130-140.
Since we are running .net 4.0 in integrated mode I suppose that we should be able to handle up to 5000 simultanously requests, or atleast 1000 (http.sys kernel)
http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx
What could be causing .net to queue the requests even though there are threads left and requests/sec is low?
Just a guess: Garbage collection suspends all threads, so perhaps the period immediately following the garbage collection would look like a request spike, since IIS would be piling up requests during the GC. Can you correlate the spikes with garbage collections? If your application is I/O-bound, it may not be possible to drive the CPU load very high, since the threads will spend most of their time blocked.
The apparent leak of active connections is disturbing though, if it really is ever-increasing.
I have a multi-threaded web application with about 1000~2000 threads at production environment.
I expect CPU usage on w3wp.exe but System Idle Process eats CPU. Why?
The Idle process isn't actually a real process, it doesn't "eat" your CPU time. the %cpu you see next to it is actually unused %cpu (more or less).
The reason for the poor performance of your application is most likely due to your 2000 threads. Windows (or indeed any operating system) was never meant to run so many threads at a time. You're wasting most of the time just context switching between them, each getting a couple of milliseconds of processing time every ~30 seconds (15ms*2000=30sec!!!!).
Rethink your application.
the idle process is simply holding process time until a program needs it, its not actually eating any cycles at all. you can think the system idle time as 'available cpu'
System Idle Process is not a real process, it represents unused processor time.
This means that your app doesn't utilize the processor completely - it may be memory-bound or CPU-bound; possibly the threads are waiting for each other, or for external resources? Context switching overhead could also be a culprit - unless you have 2000 cores, the threads are not actually running all at the same time, but assigned time slices by the task scheduler, this also takes some time.
You have not provided a lot of details so I can only speculate at this point. I would say it is likely that most of those threads are doing nothing. The ones that are doing something are probably IO bound meaning that they are spending most of their waiting for the external resource to respond.
Now lets talk about the "1000~2000 threads". There are very few cases (maybe none) where having that many threads is a good idea. I think your current issue is a perfect example of why. Most of those threads are (apparently anyway) doing nothing but wasting resources. If you want to process multiple tasks in parallel, espcially if they are IO bound, then it is better to take advantage of pooled resources like the ThreadPool or by using the Task Parallel Library.
I'm trying to recognize a run away threads in my own application and close them for good before they render machine inaccessible.
However, I can only get CPU time for the thread, that is limitation of API I'm using. Is there any way to evaluate CPU utilization from that data?
I was thinking about comparing it to real time and if it is close - than that thread is loading CPU too much. What do you think about that heuristic, will it work?
CPU time divided by real time will give you CPU utilization.