We are starting to write more and more code for an ASP.Net web application uses a new thread to complete long running tasks. I can find no solid documentation that give any useful guide to any limitations of restrictions of using threads within IIS (6). Any advice to this end would be appreciated - specifically the following:
What (if any) is the max number of threads
Is there a recommended max number
Are there any pitfalls of using threads within an ASP.Net IIS web application?
Thanks for any advice
I assume you have already looked into Asynchronous ASP.NET page processing?
Improving .NET Application Performance and Scalability
http://msdn.microsoft.com/en-us/library/ms998530.aspx
10 Tips for Writing High-Performance Web Applications
http://msdn.microsoft.com/en-us/magazine/cc163854.aspx
I can find no solid documentation that
give any useful guide to any
limitations of restrictions of using
threads within IIS (6).
Mainly because this is a bad idea. Long running processes should be converted into windows services which either run continuously and occasionally check the database or whatever else for work to do or services that can be woken up by your asp.net app.
I myself have frequently done the same thing. What i found was that there is a maximum which is based on a "n number of threads per CPU" these can be adjusted and fine tuned in the web.config and machine.config files. This post has a reasonable explanation of this.
The recommended maximum would be the default setting, at least according to the documentation I have read from Microsoft on this topic sometime ago.
The biggest pitfall you will find you need to cross is how to report progress or the results back to the user. I typically use a polling mechanism from the client to call back to a page which checks the session state for progress. The session state is of course being updated from the main thread. If you want to see this approach working in real life see the House of Travel website and do a search for flights.
Was going to make this a comment, but I realized it was more relevant than I thought. Eric Lippert has heard this set of questions before, and states that it is unanswerable.
So, in short, don't even go there.
Come up with a design that uses a
small number of threads and tune that.
Make sure that you only use threads when you're going to benefit. If your long-running code is CPU-intensive, then you won't actually benefit from making the call asynchronous (in fact, performance will decrease as there is an overhead).
Use threads for I/O operations or calling Web Services.
Each application is different. Simply setting the ThreadPool to max isn't the answer, or it would already be set at this level!
The higher you set the ThreadPool, the more you'll saturate the CPU, so IF you have CPU-intensive code then this will just compound the problem even more.
Of course, you could off-load these CPU-intensive calls onto another machine.
http://msdn.microsoft.com/en-gb/magazine/cc163327.aspx
http://msdn.microsoft.com/en-us/magazine/cc163725.aspx
Related
I've heard that only 1 cpu core can be used per asp.net 4.0 website in IIS 7. Using more cpu cores means entering webfarm territory (as a result session management should be done respectively). But I could not find any references confirming this.
So are there any limitations on cpu core count that can be used per website, where session still can be in-proc? Any references?
Absolut wrong.
The IIS7 and asp.net use all cpu and the power of the server. Also include in that and the SQL server that run in parallel with asp.net/iis and also use all the cpus and (all) the memory. Also if you make any system Thread, you also potential use a different cpu.
What you have "hear" probably is the session is blocking the asynchronous processing of the pages that is not totally bad you know, is help in many case. Again that is not mean that is use one CPU, only that the calls on the same site are synchronous (the one wait the other)
Few words about the session. Personally I have totally replace it with my custom made session handler, but for any beginner site and small sites the current session is perfect because is help you to synchronize the calls.
Without the current session module, you need to handle the synchronization by case manually - that is not so easy. If you not do that, the results is usually double and triple submissions of the same data (all that from experience).
Now if you design your site for web garden, and design it good, you take care of the synchronization of the calls and make it work fast, and correct.
Read about the session blocking on:
Web app blocked while processing another web app on sharing same session
Replacing ASP.Net's session entirely
What perfmon counters are useful for identifying ASP.NET bottlenecks?
Trying to make Web Method Asynchronous
I got a dedicated server running both IIS 7.5 and SQL Server 2010. Server CPU load is often near 100%. The SQL server does not take too much but the w3wp process is taking a significant amount of CPU (often 70+%).
I'd like to find out, what is causing this pressure:
* Too many requests of static files (a CDN could be added)
* Too many ajax requests (I am thinking about comet/web sockets anyways)
* Single asp.net pages consuming too much processing power (should be easy to optimize)
Where would you start looking to find out where to start optimizing?
The easiest possible way is to profile the app in production. Not sure if that is possible in your case. Some options:
look into the logs and look at the duration of the requests. Long requests are likely to put load on the system
Remote debug w3wp with Visual Studio and pause the debugger 10 times to see where it stops most. That is the hot spot
Use XPerf or PerfView to capture (managed) stacks. This has almost no impact on production performance
A good starting point would be to fire up the development tools (F12 in IE / Chrome) and look at the timings under the network tab. That will show you a waterfall-style diagram for how the page has loaded and should help you identify any particularly slow-loading static files which might be sensibly moved off to a cdn, any unnecessary requests being made, how much time is being spent getting the actual page itself, etc.
After that, profile the application with a performance profiler. A good profiler like ANTS Performance Profiler will let you look at things like execution time / hit counts for different methods, as well as what database queries are being run and how long they’re taking. A new version of ANTS (currently in EAP) will also group that activity by http request so you can see if specific pages need optimisation or are being hit too many times.
You'd also do well to check that caching is working as you intend it so that users aren’t unnecessarily re-requesting pages.
There's also a nice article on ASP.NET performance which you might want to read at http://aspalliance.com/1533_ASPNET_Performance_Tips.7.
Disclaimer: I work for Red Gate which makes ANTS.
I found an easy way to see what's going on on the server.
Nevertheless, the professional way is probably to go and use a profiling tool.
What did I do?
In IIS Console you can get a list of all current worker threads and if you choose one you can see what this thread is working on. So I was able to see that the thread was handling 100 requests in parallel, 70 of those were tracing back to the same ajax call.
The immediate solution was to reduce the frequency of that call (from every 10 to every 30 seconds). The next step will be to further optimize the call on the server side since I do have other ajax calls with the same frequency (every 10 seconds) which nearly never showed up in the active requests list since they were so fast.
Probably the easiest way to figure it out would be to install New Relic on the server. The trial lasts 30 days I think so it should give you enough time to get to the bottom of this. It'll show you long-running SQL queries, .NET methods, as well as just about everything else you can think of. It makes it very easy to identify bottlenecks.
By the way, I suggested New Relic because it sounds like your problem is in a production environment. New Relic isn't an incredibly detailed profiler. It gathers enough information to be helpful, but not so much as to slow down the server. That makes it well suited to this purpose.
If, however, you could reproduce the problem in a development environment you might try something like the free Eqatec profiler.
I consider myself aware of the concepts of threading and why certain code is or isn’t “thread-safe,” but as someone who primarily works with ASP.NET, threading and thread safety is something I rarely think about. However, I seem to run across numerous comments and answers (not necessarily for ASP.NET) on Stack Overflow to the effect of “warning – that’s not thread-safe!,” and it tends to make me second guess whether I’ve written similar code that actually could cause a problem in my applications. [shock, horror, etc.] So I’m compelled to ask:
Do ASP.NET developers really need to be concerned with thread safety?
My Take: While a web application is inherently multi-threaded, each particular request comes in on a single thread, and all non-static types you create, modify, or destroy are exclusive to that single thread/request. If the request creates an instance of a DAL object which creates an instance of a business object and I want to lazy-initialize a collection within this object, it doesn’t matter if it’s not thread-safe, because it will never be touched by another thread. ...Right? (Let’s assume I’m not starting a new thread to kick off a long-running asynchronous process during the request. I’m well aware that changes everything.)
Of course, static classes, methods and variables are just the opposite. They are shared by every request, and the developer must be very careful not to have “unsafe” code that when executed by one user, can have an unintended effect on all others.
But that’s about it, and thus thread safety in ASP.NET mostly boils down to this: Be careful how you design and use statics. Other than that, you don’t need to worry about it much at all.
Am I wrong about any of this? Do you disagree? Enlighten me!
There are certain objects in addition to static items that are shared across all requests to an application. Be careful about putting items in the application cache that are not thread-safe, for example. Also, nothing prevents you from spawning your own threads for background processing while handling a request.
There are different levels of ASP.NET Developers. You could make a perfectly fine career as an ASP.NET Developer without knowing anything threads, mutexes, locks, semaphores and even design patterns because a high percentage of ASP.NET applications are basically CRUD applications with little to no additional business logic.
However, most great ASP.NET Developers which I have come across aren't just ASP.NET Developers, their skills run the gamut so they know all about threading and other good stuff because they don't limit themselves to ASP.NET.
So no, for the most part ASP.NET Developers do not need to know about thread safety. But what fun is there in only knowing the bare minimum?
Only if you create, within the processing stream for a single HTTPRequest, multiple threads of your own... For e.g., if the web page will display stock quotes for a set of stocks, and you make separate calls to a stock quote service, on independant threads, to retrive the quotes, before generating the page to send back to the client... Then you would have to make sure that the code you are running in your threads is thread-safe.
I believe you covered it all very well. I agree with you. Being focused on ASP.NET only it rarely (if at all) comes to multi-threading issues.
The situation changes however when it comes to optimizations. Whenever your start a long-lasting query, you may often want to let it run in a separate thread so that the page load does not stop until the server reports connection timeout. You may wish to have this page periodically check for completion status to notify the user. Here where it comes to multi-threading issues.
I'm seeing consistently high CPU usage for my ASP.NET web application (on the live production box only, naturally....!) and I'm trying to narrow down the cause - it's basically maxing out a quad core Xeon box and there's no way it should be able to do that!
The CPU usage of the web process is generally higher than that of the DB process - which rings alarm bells to me on its own (?).
However, using the standard profiling tools (dotTrace, Red Gate etc) only show you the time spent in individual methods (rather than actual CPU usage) - and ultimately still highlight methods that are DB-bound. While this might indicate opportunities for caching or better indexes, I don't see how that in itself would result in high CPU usage of the web application process?
Any suggestions or tips as to how I can narrow this down?
Thanks!
Some suggestions to try at the first place.
1.Deploy with Release Build
Check whether the deployed product is in release mode. By running in debug mode, lot of time is wasted loading the pdbs along with the assemblies.
2.Disable ViewState
Disable viewstate if its not required. ViewState is nothing but data stored in hidden fields to be persisted between requests. it increases the total payload of the page both when served and when requested. There is also an additional overhead incurred when serializing or deserializing view state data that is posted back to the server. Lastly, view state increases the memory allocations on the server.
3.Disable Session State:
If you are not going to use it disable Session State. By default it’s on. You can actually turn this off for specific pages or for the whole application.
There are some basic ASP.NET application performance monitoring, check these two MSDN articles
"Monitoring ASP.NET Application Performance" and Performance Counters for ASP.NET
Can you set up some unit tests to call various methods and see what their impact is on processor usage? Visual Studio has some testing tools built in if you're using Team System, but even if you're not, you could write a multithreaded tester to call particular functions hundreds of times.
If you'd like some pointers on how to do this, I can help you build some basic unit testing.
are you recording/reporting unhandled exception? If not do so and check if any of them correspond with your high CPD spikes you may have a stack overflow causing the spikes.
http://msdn.microsoft.com/en-us/library/ms998306.aspx
You could also look into recoding the time of each request by using a HttpModule and checking which requests are taking up the most time which may indicate the pages that are causing the issue.
As Pradeepno notes, the place to start with is really performance counters--they can give you a very good idea of what is consuming what part of the CPU.
The web app usage being higher than DB usage isn't entirely suprising. If you have decent db design, most web apps are barely going to cause a decently powered DB server to break a sweat.
I'm quickly falling in love with ASP.NET MVC beta, and one of the things I've decided I won't sacrifice in deploying to my IIS 6 hosting environment is the extensionless URL. Therefore, I'm weighing the consideration of adding a wildcard mapping, but everything I read suggests a potential performance hit when using this method. However, I can't find any actual benchmarks!
The first part of this question is, do you know where I might find such benchmarks, or is it just an untested assumption?
The second part of the question is in regards to the 2 load tests I ran using jMeter on our dev server over a 100Mbs connection.
Background Info
Our hosting provider has a 4Gbs burstable internet pipe with a 1Gbs backbone for our VLAN, so anything I can produce over the office lan should translate well to the hosting environment.
The test scenario was to load several images / css files, since the supposed performance hit comes when requesting files that are now being passed through the ASP.NET ISAPI filter that would not normally pass through it. Each test contained 50 threads (simulated users) running the request script for 1000 iterations each. The results for each test are posted below.
Test Results
Without wildcard mapping:
Samples: 50,000
Average response time: 428ms
Number of errors: 0
Requests per second: 110.1
Kilobytes per second: 11,543
With wildcard mapping:
Samples: 50,000
Average response time: 429ms
Number of errors: 0
Requests per second: 109.9
Kilobytes per second: 11,534
Both tests were run warm (everything was in memory, no initial load bias), and from my perspective, performance was about even. CPU usage was approximately 60% for the duration of both tests, memory was fine, and network utilization held steady around 90-95%.
Is this sufficient proof that wildcard mappings that pass through the ASP.NET filter for ALL content don't really affect performance, or am I missing something?
Edit: 11 hours and not a single comment? I was hoping for more.. lol
Chris, very handy post.
Many who suggest a performance disadvantage infer that the code processed in a web application is some how different/inferior to code processed in the standard workflow. The base code type maybe different, and sure you'll be needing the MSIL interpreter, but MS has shown in many cases you'll actually see a performance increase in a .NET runtime over a native one.
It's also wise to consider how IIS has to be a "jack of all trades" - allowing all sorts of configuration and overrides even on static files. Some of those are designed for performance increase (caching, compression) and - indeed - will be lost unless you reimplement them in your code, but many of them are for other purposes and may not ever be used. If you build for your needs (only) you can ignore those other pieces and should be realising some kind of performance advantage, even though there's a potential ASP.NET disadvantage.
In my (non-.NET) MVC testing I'm seeing considerable (10x or more) performance benefits over webforms. Even if there was a small hit on the static content - that wouldn't be a tough pill to swallow.
I'm not surprised the difference is almost negligible in your tests, but I'm happy to see it backed up.
NOTE: You can disable wildcard mapping from static directories (I keep all static files in /static/(pics|styles|...)) in IIS. Switch the folder to an application, remove the wildcard mapping, and switch it back from an application and - voilà - static files are handled by IIS without pestering your ASP.NET.
I think there are several additional things to check:
Since we're using the .Net ISAPI filter, we might be using threads used to run application for serving static assets. I would run the same load test while reviewing performance counters of threads - Review this link
I would run the same load test while running Microsoft Performance Analyzer and compare the reports.
I was looking for benchmark like this for a long time. Thanx!
In my company we did wildcard mapping on several web sites (standard web forms, .net1.1 and 2, iis6), and sys admins said to me that they didn't noticed any performance issues.
But, it seems you stressed network, not server. So maybe the scores are so similar because network bottleneck? Just thinking...
That's quite an impressive post there, thanks very much for that.
We're also assessing the security and performance concerns with removing a piece of software that's always been in place to filter out unwanted traffic.
Will there by any further benchmarking on your part?
Cheers,
Karl.
Seems the bottleneck in your test is network utilization. If the performance degradation is expected to be on the CPU usage (I'm not sure it is, but it's reasonable), then you wouldn't notice it with the test you did.
Since this is a complex system, with many variables - it does not mean that there is no performance degradation. It means that in your scenario - the performance degradation is probably negligible.