We have a very large three-tier .NET 4.0 web application (ASP.NET, WCF, SQL Server 2008 R2) with some performance issues I'm trying to diagnose. The ASP.NET tier uses the SQL session state provider, and there is reasonably extensive use of session state.
<sessionState mode="SQLServer" sqlConnectionString="..." cookieless="false" timeout="20"/>
I'm trying to figure out how much time it takes to load and save the session to the database. Note that this is different than using SQL profiler or other database means to measure time on that tier; I want to measure total time taken from the ASP.NET tier. I.e. that would include session object serialization/deserialization, wire time, and SQL save/load time.
I am pretty experienced developer (since the early 80's), and I have Googled the web for hours now -- there are tons of articles contrasting session state approaches and general performance concepts etc, but I can find nothing on how to actually measure this total time for the SQL session state provider. I initially thought of using an HttpModule to hook events and use a stopwatch, but there doesn't appear to be a clean path to this. (I would settle for an approach that's reasonably accurate to few percent...)
Any ideas/suggestions are much appreciated, thanks.
Could you not just alter your session timeout to seconds, and then to a lower and lower amount, until it does not load and save the session?
The managed stack can be profiled in the VS profiler. See:
Find Application Bottlenecks with Visual Studio Profiler
How to: Profile a Web Site or Web Application Using the Performance Wizard
Analyzing Application Performance by Using Profiling Tools
For low level stuff use Xperf, see Two Minute Drill: Introduction to XPerf
For the SQL Server side there are many methodologies for performance troubleshooting, like Waits and Queues. For specific statement duration/IO cost you can use profiler trace, see Scenarios for Using SQL Server Profiler.
Now for a very specific and invasive way in your case you can use Performance Counters. Not the stock kind, but create your own. See Using XSLT to generate Performance Counters for an ease way to geenrate the plumbing needed to augment your app with its own perf counters, then hook up your code with performance tracing. Use the standard perofrmance counters tool chain (logman.exe, perfmon.exe) to capture and analyze.
The easiest way to do this is to profile the app and look at where the time is spent deep in the bowels of ASP.NET. You will be able to quickly spot session-related activity by class and method name.
Perfview is fine for this. It has a sampling profiler which means that it will give you accurate, unskewed profiling information. But it can't tell you how many seconds were spent in what method. All it tells you are percentages.
I'd spin up a load test against an empty page and let perfview run for 30s.
dottrace is a simple profiler that also shows you the amount of CPU time spent per call, I believe.
The first thing that comes to mind is to create a page that doesn't really do anything other than perhaps load/store the session state variables. From there you can set up a coded UI test that accesses the page a number of times and builds a statistical estimate for the variability and distribution involved in load times for the page.
WCAT (Web Capacity Analysis Tool) could also be used to see how performance changes under load...
Related
I've heard that only 1 cpu core can be used per asp.net 4.0 website in IIS 7. Using more cpu cores means entering webfarm territory (as a result session management should be done respectively). But I could not find any references confirming this.
So are there any limitations on cpu core count that can be used per website, where session still can be in-proc? Any references?
Absolut wrong.
The IIS7 and asp.net use all cpu and the power of the server. Also include in that and the SQL server that run in parallel with asp.net/iis and also use all the cpus and (all) the memory. Also if you make any system Thread, you also potential use a different cpu.
What you have "hear" probably is the session is blocking the asynchronous processing of the pages that is not totally bad you know, is help in many case. Again that is not mean that is use one CPU, only that the calls on the same site are synchronous (the one wait the other)
Few words about the session. Personally I have totally replace it with my custom made session handler, but for any beginner site and small sites the current session is perfect because is help you to synchronize the calls.
Without the current session module, you need to handle the synchronization by case manually - that is not so easy. If you not do that, the results is usually double and triple submissions of the same data (all that from experience).
Now if you design your site for web garden, and design it good, you take care of the synchronization of the calls and make it work fast, and correct.
Read about the session blocking on:
Web app blocked while processing another web app on sharing same session
Replacing ASP.Net's session entirely
What perfmon counters are useful for identifying ASP.NET bottlenecks?
Trying to make Web Method Asynchronous
I need some information from you.I have used session.TimeOut=540 in application.Is that effects on my Application performance after some time.When number of users increases its getting very slow. response time nearly more that 2 minutes for a button click also.This is hosted in server in Application pool .I don't know about Application pool much.If Session Timeout is the problem i will remove it.Please suggest me the way to for more users.
Job Numbers,CustomerID,Tasks will come from one database.when the user click start Button then the data saved in another Database.I need this need to be faster for more Users
I think that you have some page(s) that make some work that takes time, or for some reason or a bug is keep open for more time than the usual.
This page is keep lock the session and hold the rest page from response because the session holds all the pages.
Now, together with the increase of the timeout this page is lock everything and here is you response time near to 2 minutes.
The solution is to locate the page that have the long running problem and fix it or make it faster by optimize the process, or if this page must keep the long time running, then disable the session for that one.
relative:
Web app blocked while processing another web app on sharing same session
What perfmon counters are useful for identifying ASP.NET bottlenecks?
Replacing ASP.Net's session entirely
Trying to make Web Method Asynchronous
Does ASP.NET Web Forms prevent a double click submission?
About server
Now from the other hand, if your server suffer from hardware, or bad setup then here is one other answer with points that you need to check to make it faster.
Find out where the time is spent
add the StopWatch in the method which you said "more that 2 minutes for a button click". you can find which statment spent the most time.
If it is a query on DB that cost time. Check your sql statement.
are you using "SELECT Count(*)" instead of "SELECT Count(Id)"? the * is always slower. also, don't try "SELECT * FROM...."
Use cache.
there are many ways to do cache. both in ASPX pages and your biz layer.
the OutputCache is the most easy way.
and also, cache the page (for example a blog post) on the first time when a user visit it.
Did you use memory paging?
be careful when doing paging on gridview or other list. If you just call DataSource=xxx and DataBind(), even with PagedDataSource, this is likely a memory paging. It cost a lot of performance. Please use stored procedures to do paging.
Check your server environment
where did you deploy the website? many ISP will limit brandwide and IIS connection count and also CPU time to your account.
if you have RD access to your server. you can watch CPU and memory usage to see if they are high when many user comes to your site. If the site is slow and neither CPU nor memory useage is high, it may be a network brandwide problem.
Here are some simple steps to narrow down the issue -
1) Get HTTPWatch (theres a free Basic version) available and check whats really taking time from an end user perspective. Look at number of requests, number of resources downloaded, and the payload. If there is nothing to worry move on to next
2) If its not client, then its usually the processing time on the server. Jump on to DB first - since this is quite easier to eliminate quickly. Look at how many DB calls are made (run profiler in staging or dev) and see if there are any long running queries, missing indexes or statistics, and note the IO. If all is well, move on
3) Check your app code. You could get on with VS.NET in build profiler or professional tools such as Ants. If code is fine then its your network or external calls that you make, check your network bandwidth. If you still cannot narrow down, check your environment/hardware
The best way to get to it is to apply load - You could use simple tools such as ab.exe (that comes as part of Apache Web server) to have concurrent hits on your server and run the App, DB profilers in the background to get to the issue.
Hope this helps!
I have found a lot of great information comparing InProc, StateServer, and SQLServer for ASP.NET state management, but I can't seem find any performance benchmark comparisons. It is clear that InProc is faster than StateServer which in turn is faster than SQLServer, but it isn't clear how much faster. I realize that it's going to vary greatly by application and environment, but having a relative idea of how they compare would be valuable.
Do you know of any benchmarks that have been performed that you could share? or have any personal experience with this? Thank you!
I have personal experience but no benchmarks or actual recorded metrics to share. We initially created an Asp.Net site which stored a larger than usual user object in session using the InProc method. We found that the size of the object and the nature of our error handling libraries caused 2 undesired behaviors. The first was a recycling of the application pool at random intervals during processes. Because the w3wp.exe process would recycle itself midstream, it would essentially dump the session and the object would be lost. This caused other code to kick in and repair the session, and it increased the latency of our transactions. We also found (although it was not a terrible problem and I only discovered while attempting to debug the memory loss I just described) that the size of the object in session along with some of the objects being loaded in libraries for the page itself would cause the w3wp.exe to page itself in and out repeatedly. For smaller requests that only involved either the session object or these library objects but not both, there was no odd paging behavior on the process.
In moving from InProc to StateServer, we discovered an immediate return on the recycling. The pool actually ended up recycling less, and even when it did our session objects stayed in separate memory. We also noticed that this created a universal "library only" condition as described above with respect to paging and we did not experience it again (though admittedly I stopped checking after 1 month of uptime). We did pick up a very small latency in accessing certain framework libraries at the time, but when we upgraded from 2.0 to 3.5, these access anomalies disappeared entirely. We're hoping for similar behavior when we upgrade from 3.5 to 4.0 soon.
A test site using SQLServer as a state controller was attempted, and the only latency we found was the initial session creation/storage. Subsequent calls to update/access the session in SQL provided no real difference from the StateServer option. I don't have any metrics, but there was nothing on any of the systems that indicated a difference in behavior. Timestamps had comparable differences in all aspects. A benefit we did gain, though it was of rare usage potential, was that we were able to couple our user database directly with the session state server and compare timestamps, statuses, and other specialized session variables directly. We had no real need for this feature, and the StateServer option was already in full swing on our production servers, so a determination to leave it as it was.
In the end, it wasn't speed so much as memory usage that persuaded us to dump InProc for StateServer. The benefits of access speed definitely did not outweigh the need to have the data in memory in the first place.
There's a good benchmarks the DevOps Guys.
http://www.slideshare.net/devopsguys/best-performing-aspnet-session-state-providers comparing
ASP.Net In-Proc
ASP.Net Session State Server
ASP.Net Sql Server
CouchBase
MongoDb
RavenDb
Redis (this one, TheCloudlessSky, not this one AngiesList)
AppHarbor also recommends memcached, but doesn't have a benchmark.
http://support.appharbor.com/kb/tips-and-tricks/using-memcached-backed-sessionprovider
and provides a Session Provider https://github.com/friism/Memcached-Providers
I am developing large data collecting ASP.Net/Windows service application-pair that uses Microsoft SQL Server 2005 through LINQ2Sql.
Performance is always the issue.
Currently the application is divided into multiple larger processing parts, each logging the duration of their work. This is not detailed and does not help us with anything. It would be nice to have some database tables that contain statistics that the application itself collected from its own behavior.
What logging tips and data structures do you recommend to spot the parts that cause performance problems?
Edit:
Mostly I am looking for parts of the application that can cripple the whole system when excessively used. There are peaks during the day when some parts of the application are under heavy load. Some advanced logging would help me isolate the parts that need more attention and optimizing.
Don't use logging for this, use Performance Counters instead. The runtime impact of performance counters is minor and you can simple have them always on. To collect and monitor the performance, you can rely on the existing performance counters infrastructure (perfmon.exe, logman.exe, relog.exe etc).
I personally use XML and XSLT to generate the counters. I can then decorate all my code with performance counters tracking functions being run, average call duration time, number of executions, rate of executions per second and so on and so forth. Good choice of counters will give an immediate, accurate, performance picture much faster than logging can. While logging can give more insight on certain event paths (ie. order of events that lead to certain state), logging can seldom be 'always on' as the impact on performance is significant, not only on performance but most importantly on concurrency as most existing logging infrastructures add contention.
This is not a job for logging. It's a job for a profiler.
Try one of these:
JetBrains' dotTrace - http://www.jetbrains.com/profiler/index.html
Red-Gate ANTS - http://www.red-gate.com/products/ants_profiler/index.htm
Automated QA's AQTime - http://www.automatedqa.com/products/aqtime/index.asp
While I haven't (yet) tried it for myself, it may be worth looking at Gibraltar which can be used with PostSharp to put declarative performance logging into your code.
When dealing with problems like this I try and not add any extra headache by manually adding logging / tracing & timing into the application itself. If all you want is to tune the application then I suggest getting a profiler which will show you what areas of code are an issue. I recommend Red-Gate's Ant's Profiler.
Now if you want to collect statistics for monitoring or trending purposes then a profiler is not the right tool. I have had success using PerformanceCounters which let's many third party tools pull the performance information out of the application.
So what are you trying to do solve performance problems or monitor to ensure you catch a performance problem before it becomes severe?
EDIT
Based on your comment, I would look at using performance monitors around critical sections of code, timing how long it took to complete an operation. Then you can use the built in performance monitoring tools, or any number of third party tools to monitor and trend the stats.
SQL Server keeps track of some things for you, so try running some of these queries on your system:
Wayback Machine: Uncover Hidden Data to Optimize Application Performance
here is an example from the link:
--Identifying Most Costly Queries by I/O
SELECT TOP 10
[Average IO] = (total_logical_reads + total_logical_writes) / qs.execution_count
,[Total IO] = (total_logical_reads + total_logical_writes)
,[Execution count] = qs.execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
(CASE WHEN qs.statement_end_offset = -1
THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
ELSE qs.statement_end_offset END - qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
ORDER BY [Average IO] DESC;
the link contains many queries including ones for: Costly Missing Indexes, Logically Fragmented Indexes, Identifying Queries that Execute Most Often , etc...
I would start would diagnosing what is the real cause for the perf issue? Is it CPU, Memory, Disk or IO. This can be identified by few perfmon counters.
For example Linq2SQL uses Sync I/O which could be a big bottleneck for scalability. Because it uses Sync I/O windows threads get blocked and requests would end up waiting. This is usual suspect and might not be true. Here is an MSDN article how sync I/O could affect scalability.
If CPU is an issue then the next question is application CPU bound? Then you could use one of the profilers like mentioned above. Also look for time spent on GC perfmon counter that is another usual suspect?
I'm seeing consistently high CPU usage for my ASP.NET web application (on the live production box only, naturally....!) and I'm trying to narrow down the cause - it's basically maxing out a quad core Xeon box and there's no way it should be able to do that!
The CPU usage of the web process is generally higher than that of the DB process - which rings alarm bells to me on its own (?).
However, using the standard profiling tools (dotTrace, Red Gate etc) only show you the time spent in individual methods (rather than actual CPU usage) - and ultimately still highlight methods that are DB-bound. While this might indicate opportunities for caching or better indexes, I don't see how that in itself would result in high CPU usage of the web application process?
Any suggestions or tips as to how I can narrow this down?
Thanks!
Some suggestions to try at the first place.
1.Deploy with Release Build
Check whether the deployed product is in release mode. By running in debug mode, lot of time is wasted loading the pdbs along with the assemblies.
2.Disable ViewState
Disable viewstate if its not required. ViewState is nothing but data stored in hidden fields to be persisted between requests. it increases the total payload of the page both when served and when requested. There is also an additional overhead incurred when serializing or deserializing view state data that is posted back to the server. Lastly, view state increases the memory allocations on the server.
3.Disable Session State:
If you are not going to use it disable Session State. By default it’s on. You can actually turn this off for specific pages or for the whole application.
There are some basic ASP.NET application performance monitoring, check these two MSDN articles
"Monitoring ASP.NET Application Performance" and Performance Counters for ASP.NET
Can you set up some unit tests to call various methods and see what their impact is on processor usage? Visual Studio has some testing tools built in if you're using Team System, but even if you're not, you could write a multithreaded tester to call particular functions hundreds of times.
If you'd like some pointers on how to do this, I can help you build some basic unit testing.
are you recording/reporting unhandled exception? If not do so and check if any of them correspond with your high CPD spikes you may have a stack overflow causing the spikes.
http://msdn.microsoft.com/en-us/library/ms998306.aspx
You could also look into recoding the time of each request by using a HttpModule and checking which requests are taking up the most time which may indicate the pages that are causing the issue.
As Pradeepno notes, the place to start with is really performance counters--they can give you a very good idea of what is consuming what part of the CPU.
The web app usage being higher than DB usage isn't entirely suprising. If you have decent db design, most web apps are barely going to cause a decently powered DB server to break a sweat.