I have inherited a somewhat complex system (and problem) that I need help with.
I have a webserver w/ the following specs:
Hardware:
Server 2003 32bit
IIS 6
8 cores (16 w/ hyperthreading)
12gb RAM
ASP.NET site
3 app pools, so 3 instances of w3wp.exe running.
This system serves a large number of people and bandwidth is fairly constant during business hours reaching ~ 68,000kbit/s
There are moments when the system "comes down" - site gets very slow which generates a lot of phone calls. Things usually slow down for 60 seconds, but has varied greatly in length. Sometimes only a few seconds and sometimes 3 minutes or more.
I have my app pools set to recycle somewhere about 600mb of consumed memory. That's not exact but they recycle on their own with much success. At times I recycle the "main" pool manually to clear the problem I'm describing.
This is what I know is going on when things are running slow.
Network bandwidth takes a considerable dip.
Requests Queued in the ASP.NET performance counters goes up.
In tandem w/ the Requests Queued rising page latency increases. (I employ a simple ASP page that makes a SQL call and just says "The system is live" - this page is monitored for latency)
Overall CPU usage rises.
Overall memory consumption of w3wp.exe rises.
In my mind here is what I imagine is happening.
Someone asks the system to generate a report or glob of data. This spins up a process that consumes a large number of threads (ie, all available threads) This causes all other requests to the system to wait in the ASP.NET que pool which essentially kills the site. The lack of activity causes the network traffic to dip.
I have read many articles about thread queues, thread pools, etc. This is a good example: http://williablog.net/williablog/post/2008/12/02/Increase-ASPNET-Scalability-Instantly.aspx and does what I believe is a clue to help me solve my problem... but I'm not sure. My "Machine.config" file for the version of asp.net that I am using does not specify any of the thread values listed in the article so we are default for everything which I believe is incorrect given our situation.
If you were me; What would you do next? Where do you think the problem is?
edit: Here is a screenshot. It should be obvious when the problem is happening.
http://i.imgur.com/5BJlq.png
edit:
I want to change these values for our setup. A few questions first:
1) After making the changes, what needs to be restarted for them to take effect?
2) How do these settings look for a system with 8 physical cores?
maxconnection = 96
maxIoThreads = 100
maxWorkerThreads = 100
minFreeThreads = 704
minLocalRequestFreeThreads = 608
Not fun.
Many root causes share common symptoms which makes it difficult to diagnose without getting dirty with the application. :) Pardon if some of these steps were implied.
Some next steps might be:
Review the IIS logs of each site looking for things like:
HTTP response codes (5xx,4xx,3xx)
Request response times
Review Windows Event Logs
How often are application pools cycling?
Application errors, etc.
Verify processModel settings as suggested by #vinayc to make sure predecessor didn't get 'tricky'
Install DebugDiag, its a surprisingly good tool for some basic analysis of memory and crash related problems.
This can also help you capture memory snaps to diagnose later.
Tess Ferrandez blog can help make heads/tails of memory snap analysis.
Understand how many web applications are running in each AppPool.
Investigate using a 'web garden' to possibly help minimize number of users impacted by 'slow down'
Is a virus scanner enabled? Is it running? If so, verify exclusions.
Are application teams available to help troubleshoot? Identify if they have any custom application instrumentation that might help diagnose problem.
Is the behavior 'new'? Or has it always been there? If 'new', can you track down which deployment might have caused the new behavior?
Could the the description given of the 'slow down' behavior be attributed to an apppool recycle and resulting jitting of the application again? ala - the first request syndrome.
Reviewing the logs helps understand how the sites/applications are being used, which can be especially important if you don't own the codebase. Logparser is an excellent tool for doing some IIS log analysis (as well as other formats).
Good luck!
Z
The settings that your are talking are part of processModel element under system.web element from machine.config. For IIS6, following are applicable:
autoConfig
maxIoThreads
maxWorkerThreads
minIoThreads
minWorkerThreads
requestQueueLimit
responseDeadlockInterval
Typically, you will only find autoConfig="true" and not other elements. Auto-config sets the values as per your machine configuration - the tuning is done as per recommended values (see Threading Explained section from this article) which are same as sighted by the link that you have provided.
The article although dated, i excellent resource if you want to tune up these settings manually.
On the other hand, at the load that you are serving, I would recommend two things (if you haven't tried already)
Use output caching aggressively - even if the data is dynamic, caching for say 30-60 seconds can give a definite boost at your load
If you suspect certain requests are hogging too many threads then attempt to move those resources under different app-pool (you can use different web-site with different sub-domain or you can use different virtual directory/application and choose different app-pool)
Related
We have a fairly popular site that has around 4 mil users a month. It is hosted on a Dedicated Box with 16 gb of Ram, 2 procc with 24 cores.
At any given time the CPU is always under 40% and the memory is under 12 GB but at the highest traffic we see a very poor performance. The site is very very slow. We have 2 app pools one for our main site and one for our forum. Only the site is being slow. We don't have any restrictions on cpu or memory per app pool.
I have looked at he Performance counters and I saw something very interesting. At our peek time for some reason Request are being queued. Overall context switching numbers are very high around 30 - 110 000 k.
As i understand high context switching is caused by locks. Can anyone give me an example code that would cause a high number of context switches.
I am not too concerned with the context switching, and i don't think the numbers are huge. You have a lot of threads running in IIS (since its a 24 core machine), and higher context switching numbers re expected. However, I am definitely concerned with the request queuing.
I would do several things and see how it affects your performance counters:
Your server CPU is evidently under-utilized, since you run below 40% all the time. You can try to set a higher value of "Threads per processor limit" in IIS until you get to a 50-60% utilization. An optimal value of threads per core by the books is 20, but it depends on the scenario, and you can experiment with higher or lower values. I would recommend trying setting a value >=30. Low CPU utilization can also be a sign of blocking IO operations.
Adjust the "Queue Length" settings in IIS properties. If you have configured the "Threads per processor limit" to be 20, then you should configure the Queue Length to be 20 x 24 cores = 480. Again, if the requests are getting Queued, that can be a sign that all your threads are blocked serving other requests or blocked waiting for an IO response.
Don't serve your static files from IIS. Move them to a CDN, amazon S3 or whatever else. This will significantly improve your server performance, because 1,000s of Server requests will go somewhere else! If you MUST serve the files from IIS, than configure IIS file compression. In addition use expire headers for your static content, so they get cached on the client, which will save a lot of bandwidth.
Use Async IO wherever possible (reading/writing from disk, db, network etc.) in your ASP.NET controllers, handlers etc. to make sure you are using your threads optimally. Blocking the available threads using blocking IO (which is done in 95% of the ASP.NET apps i have seen in my life) could easily cause the thread pool to be fully utilized under heavy load, and Queuing would occur.
Do a general optimization to prevent the number of requests that hit your server, and the processing time of single requests. This can include Minification and Bundling of your CSS/JS files, refactoring your Javascript to do less roundtrips to the server, refactoring your controller/handler methods to be faster etc. I have added links below to Google and Yahoo recommendations.
Disable ASP.NET debugging in IIS.
Google and Yahoo recommendations:
https://developers.google.com/speed/docs/insights/rules
https://developer.yahoo.com/performance/rules.html
If you follow all these advices, i am sure you will get some improvements!
I am writing a High Performance ASP.NET Json API with soon > 1000 Request/Second. All my logic and processing is done in an IHttpHandler. I measured via Stopwatch Class and the handler finishes a request in around 0,1 - 0,5 Millisecond.
But it seems IIS and/or other HTTPHandlers (Modules?) are taking away a lot of performance. Can i measure that somehow ? How much overhead will a request produce in IIS when configured for best performance ?
Will removing all those HTTPHandlers help, or are there other tricks to speed it up? I dont need much of the ASP.NET Featureset besides Session (could even workaround that if it give a significant performance boost).
Measuring performance of a web server is no trivial task. A few things to consider:
Find the actual bottleneck. This can be memory, disk access, caching, database access, network latency etc. Use a memory profiler, or other performance profiler to find out.
Use WireShark to find the difference between how long the request is on your machine and how long your code runs.
Try other configurations. Give ASP.NET more memory. Upgrade the test system. I.e., going from 8GB / 2.5GHz with 600 requests/sec to 16GB / 3.0GHz can yield 6500 requests/sec. Performance growth is often not linear. See this document from Microsoft.
Consider adding an extra machine. This can yield up to a 50 or even higher performance upgrade depending on how you configure it. See again that document from MS.
Check these hints by Jon Skeet. The comment thread reveals some non-obvious potential bottlenecks as well.
NOTE 1: know your tools. ASP.NET runs each request in its own thread. Thread swapping is faster than process swapping, but it still requires time. If other handlers take time because they are in the request chain, it's beneficial to disable them.
NOTE 2: one of the original side-goals of stackoverflow was to create a site in ASP.NET that had great performance on max 2 servers and could handle > 1Mln visitors per hour. They managed to do that. I believe they wrote some blogposts on it, but I don't remember where they are.
This is a very good question. I have noticed the same once you get into the single-millisecond range of response times, ASP.NET overhead starts to be noticable. I can confirm your observation.
What I have done successfully is to find out, which HttpModules are registered (using IIS Manager) and disable all of them which I could possibly get rid of. The standard ASP.NET pipeline has a lot of modules and functionality configured.
If you need ultimate performance, you could of course use a tiny HTTP server library and get rid of almost all overhead that way. This would be so incredibly fast.
I have faced strange behavior of my ASP.Net application on the server (IIS7 on Windows Server 2008 x64, processor Xeon Quad).
The web application is the simple page which about one second calculates some math, and then displays result. That is it almost does not consume a network, a disk, memory, but completely uses processor resources.
The following phenomenon appears at load testing: IIS7 utilize processor no more than on 25% and not for the world does not wish to utilize it more. This 25% are equal to one core, but spread out on all four according to task manager performance tab. On the other computer (IIS7, Win 7, Quad) all works as well as should: the processor is utilized on all of 100%.
For each of behavior variants (peak loading of 25% and 100% on 4 core processors) I have found on 2 computers. The similar situation is described here. What can cause such behavior?
This 25% are equal to one core, but
spread out on all four according to
task manager performance tab.
Reality check: when you use up one core, the CPU scheduler will move the load between cores before 2008 R2. Staring with 2008 R2 it will keep it on one core to actually move the other cores into deep sleep.
So, what you see is basically an application that uses one CPU core. Point.
What can cause such behavior?
Either your code, or your request generation (well, together with your code) make sure that the requests are serialized and not handled in parallel.
During load testing... do you accept / keep the session cookie (like: ONE) and in your asp.net page do you have session state enabled? This would serialize all page requests to the one session in memory and is one very likely culprit. Another one is doing "stupid" things in code the result in a block and make the algorithm effectively single threaded - but this can no be evaluated without a lot more information from you on how you program and what you effectively do. For example I have seen a bunch of monkeys once code an online shop using ONE database connection (to not overload the database) that was kept in the aplication object and using the lock / unlock methods there to effectively turn their asp application into a single threaded thing. This was obvious - but there are a lot of other things that can go wrong. The questions basically are:
Are you by configuration / test scenario doing something to force IIS to serialize (which would be among web farming settings or bad usage of session state)?
Do you do anything within the pages that is effectively locking them to single threaded?
IIS per se answers requests through work items (i.e. uses a LOT of threads) unless it HA to serialize them (sessions only are ever assigned to ONE thread at a time, so a second request for the same session is serialized).
I doubt it's spread out. More likely the algorithm is not parallelised and so the code runs in a single core.
I have understood, that on those 2 computers where loading was 100%, 32 bit Windows has been installed. On the same 2 computers, where peak loading was 25% - 64 bit. But customisation change "Enable 32 bit applications"=true has not helped.
If your server is using multiple worker processes and you are sure that your load testing software is issuing requests in parallel, then something in your application is likely becoming serial.
This is actually pretty common (we do a lot of load testing for our customers) - it could be as simple as a database pool with a size of one or as complex as some shared resource being locked at some level deep within the application or within a library the application is using. We've seen cases where the first step in serving an application page opens a transaction that is not committed until the page is done. If that transaction is locking a table that is needed for the same purpose by every other page, then only one page request can be serviced at a time.
Good luck hunting down the problem - be sure to let us know what you find!
The problem has been solved after installation of fresher OS. "Windows Server 2008 Enterprise SP1 (c) 2009" instead of "Windows Server 2008 Standard SP2 (c) 2007".
I'm profiling a asp(classic) web service. The web service makes database calls, reads/writes to files, and processes xml. On a windows server 2003 box(2.7ghz, 4 core, 4gb ram) how many requests per second should I be able to handle before things start to fail.
I'm building a tool to test this, but I'm looking for a number of requests per second to shoot for.
I know this is fairly vague, but please give the best estimate you can. If you need more information, please ask.
95% of the performance of any data-driven app is dependent on the database: 1) the way you do your calls, 2) the indexes, 3) the hardware under the database (disk subsystem in particular).
I have seen a machine, like you are describing, handle 40 requests per second (2500/minute), but numbers like 10 per second (600/minute) are more common. I would expect even lower if you are running your DB on the same machine, and even lower still if that DB is SQLExpress or MSAccess.
Also, at capacity, your app will probably not fail, but IIS will Queue requests, once it is saturated, and may timeout some of those requests if it can't service them before the timeout expires.
Btw, instead of building a tool to test your app, you may want to look into using a test tool such as Microsoft WCAT. It is pretty smooth and easy to use.
How fast should it be? Fast enough.
How fast is fast enough? That's a question that only you and your users can answer. If your service is horrifically inefficient and keeps up with demand, it's fast enough. If your service is assembly-optimized, lightning-fast, and overwhelmed with requests, it's not fast enough.
If the server is handling its actual workload, then don't worry about how fast it "should" be. When the server is having trouble, or when you anticipate that it soon will, then you should look at improving the code or upgrading the hardware. Remember Knuth's Law – premature optimization is the root of all evil. Any work you do now to make it faster may never pay off, and you may be forced to make compromises with flexivility or maintainability. Remember, too, an older adage – if it ain't broke, don't fix it.
Yes I would also say 10 per second is a good benchmark. For a high performance app you would want to get more than this, but if you have no specific goal you should generally be able to get at least 10 requests per sec for a general web page with a bunch of database queries.
Environment:
Windows 2003 Server (32 bit); IIS6, ASP.NET 2.0 (3.5); 4Gb Ram; 1 Worker Process
We have a situation where we have a very large System.XmlDocument is being loaded into memory, and then it heads into a complied XSL transform.
What is happening is when a web request comes in the server is sitting in an idle state with 2500Mb of available system memory.
As the XML DOM is populated, the available memory drops approx 500Mb at which point we get a System.OutOfMemoryException event. At this point the system should theoretically still have 2000Mb of available memory available to service the request (according to Perfmon).
The related questions I have are:
1) At what level in the stack is this out of memory limitation being met? OS? IIS? ASP.NET? worker process? Is this a per individual web request limit?
2) Is this limit configurable somewhere?
3) Why can’t this web request access the full available system memory?
1) I would guess at the worker process but this should be configurable within IIS to the limit of memory that a worker process can use. Another factor is what level of bits does your software use, e.g. 32 bit has a physical limit of 4 GB since this is the total address space.
2) Probably but don't forget that memory fragmentation may play a role in getting to out of memory faster than you think, e.g. if there is a memory request for a contiguous 1000 Mb piece of memory then this may not necessarily be found in the current memory.
3) Have you examined dump data to see what is in the memory when the exception gets thrown? If not, there are ways to get a snapshot of the memory to see what it looks like as this may give you more clues about what is going on.
You are running in a process. A process can only access 2 gigs of memory. This task is sharing memory with everything else running in this process, so this bit of code does not get the full 2 gig -- even if it is available.
There is a 3 gig switch on the os as well. I believe it is a registry setting. But you will have to search MSDN to find that info.
But realistically, you need to do this another way. Possibly by switching to a SAX style xml parser.
I'm sure there are some bright heads here that can answer your specific questions, but have you asked yourself if there is another way to do what you want? I specifically mean that you probably do not want to process a very large XML document, but you probably more specifically want to return something back to the client. Could you rewrite the code to avoid this XML document altogether, or perhaps not load it all into memory at the same time, and still produce the same end-result?
1) Dunno. Check your logs.
2) IIS limits memory divvied out to websites/application pools. Check your settings.
3) Servers are all about uptime; if an single app hogs all the resources everybody else suffers. Thats why enterprise apps like IIS limit memory to prevent runaways from taking down the entire server.