To start off - I have 2 separate websites and a database (IIS 7.5, ASP.NET and SQL Server 2008, using Linq-To-SQL for database access).
I have a separate administrative website that sometimes, during usage needs to trigger long running operations (more than 10 seconds) on database. The problem is that those operations cause sqlserver process to hit 100% CPU and then other, main customer website, can't access database promptly - there are some delays in accessing database.
I am OK with those administrative operations lasting 2x or 4x or nx times longer since they are lower priority.
I've tried using CPU Limit setting on AppPool in IIS, but that doesn't help, as w3wp.exe process never uses much of CPU... rather it's sqlservr.exe. Thanks in advance for your suggestions!
If your admin queries are consuming all the CPU on the box, there is almost certainly some tuning opportunities there - likely some indexing optimizations.
In lieu of the time to invest in those, and until you get your Resource Governor configuration settled, you can simply reduce their impact to a single CPU, which may provide short-term symptom relief, by adding the MAXDOP hint to your admin query:
OPTION (MAXDOP 1);
Yes, it might make you feel a little dirty, but the rest of your CPUs will be freed up to work on your more important queries.
The real answer is to tune your admin queries. Just because it's ok that they run long does not mean it's good for your server or the experience of your users. You'll never be able to completely isolate them from the effects of other queries going on on the box, especially if you are experiencing high CPU that is compensating for slow I/O. I/O does not have any knobs in resource governor - you can only control CPU and memory, and not even 100%.
Sounds like you want to look into Resource Governor which is built into SQL server as of SQL 2008. BOL link should get you started.
http://msdn.microsoft.com/en-us/library/bb933866(v=SQL.100).aspx
Essentially, you can throttle CPU and memory usage for the resource pools and workloads you define. This throttling will only kick in when the server is under load. Be aware that you cannot control disk IO utilization. If the process in your admin database is IO bound and your other DBs share drives you will inevitably still see performance issues and moving databases to separate spindles or query tuning will be necessary.
Example of the classifier function that will ensure the user you define is throttled by the desired resource pool based on workload group:
/* Classifier function */
CREATE FUNCTION dbo.rgov_classifier_db ()
RETURNS sysname
WITH SCHEMABINDING
AS
BEGIN
DECLARE #rgWorkloadGrp sysname
IF SUSER_SNAME() = 'adminWebsiteDB'
SET #rgWorkloadGrp = 'workloadGroupName'
ELSE
SET #rgWorkloadGrp = 'defaultWorkloadGroupName'
RETURN #rgWorkloadGrp
END;
GO
/* Register the function with Resource Governor and then start Resource Governor. */
ALTER RESOURCE GOVERNOR
WITH (CLASSIFIER_FUNCTION = dbo.rgov_classifier_db);
GO
ALTER RESOURCE GOVERNOR RECONFIGURE;
GO
Related
I have an ASP .NET 4.5 application. On a maintenance page is a text box which allows administrative users to write SQL which is executed directly against the SQL Server 2008 database.
Occasionally one of the administrative users writes some inefficient SQL and the SQL Server process starts using up all the memory and CPU cycles on the server. We then have to start and stop the service to get the server responsive again.
Is there any way that I can stop these from queries consuming all the resources? These queries will not return fast enough for the user to see them so it's okay to cancel the query.
Edit
I realise it would be better to prevent users from writing SQL queries, but unfortunately I cannot remove this functionality from the application. The admin users don't want to be educated.
You can set the query governor at the server level but not sure about a per user or per connection / application limit.
http://technet.microsoft.com/en-us/magazine/dd421653.aspx
That said, it is likely a poor / dangerous practice to allow users to directly enter SQL queries.
First off I would make sure these queries are not locking any tables (either use NOLOCK or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED for each query ran).
As far as resources go, I would look at something like the Resource Governor. We use this for any adhoc reports on our production OLTP system.
Pinal Dave has a great blog entry on it. You can also look at Technet or other MS sites for information on what it is and how to set it up.
http://blog.sqlauthority.com/2012/06/04/sql-server-simple-example-to-configure-resource-governor-introduction-to-resource-governor/
Create a user which will have no more and no less privileges than necessary.
Create some login data to the user, you will not share this with the admins who will write the queries.
Create a panel/page/application where you let them write their queries. Here you can add additional constraints unavailable at RDBMS system-level.
You let your users access this panel/page/application and they will run their queries through this. If they are doing anything undesired uncovered by you, then you just issue your modifications for the privilege and the panel/page/application.
Leaving aside the pros/cons of allowing users to construct and run their own queries, one solution could be to use the Resource Governor that comes with SQL 2008/2012. I think it's only available in Enterprise editions which might mean you can't use it (you haven't specified an edition in your question).
Technet has an article on how to manage workloads, but basically you can limit a session (based on login details or any other decernible information) to only use a certain % of CPU and/or memory.
Best way to prevent SQL from using memory/CPU: don't let users run queries.
SQL Server is supposed to use all the memory it can get it hands on, so the memory usage (or the CPU usage for that matter) wouldn't be reason to restart SQL.
By default .NET SqlCommand timeout is 30 seconds. When it times out the query will be canceled. Queries can take a while to cancel, especially if you were making a data modification and rollbacks have to occur. It's possible while the query is running/canceling other user queries are being blocked. This might make it seem like the SQL Server is unresponsive, but eventually it will finish up and your CPU usage will go back to normal. The memory however will continue to be consumed by the SQL Server process.
I have a huge (>10GB) sqlite database that is shared among many (up to CPU core count) processes (same executable). This is a specialized application so RAM is not an issue and I want to cache as much of the database in memory. I have found about PRAGMA cache_size; and I am successfully using it but this blows the RAM usage out of proportion as each of many processes has its own private cache.
Now, I found SQLite Shared-Cache Mode but I can't see if this applies to different processes or just threads in one process. I have run some tests which confirm the latter but I am not sure if I am doing something wrong or whether something else needs to be done to make this work.
That page explains that "the same cache can be shared across an entire process".
In theory, you could try to configure your OS so that the entire database is held in the file cache.
If the amount of data in individual queries is small, it might be worthwhile to use a client/server database so that the caching needs to be done only in the server process.
I am developing large data collecting ASP.Net/Windows service application-pair that uses Microsoft SQL Server 2005 through LINQ2Sql.
Performance is always the issue.
Currently the application is divided into multiple larger processing parts, each logging the duration of their work. This is not detailed and does not help us with anything. It would be nice to have some database tables that contain statistics that the application itself collected from its own behavior.
What logging tips and data structures do you recommend to spot the parts that cause performance problems?
Edit:
Mostly I am looking for parts of the application that can cripple the whole system when excessively used. There are peaks during the day when some parts of the application are under heavy load. Some advanced logging would help me isolate the parts that need more attention and optimizing.
Don't use logging for this, use Performance Counters instead. The runtime impact of performance counters is minor and you can simple have them always on. To collect and monitor the performance, you can rely on the existing performance counters infrastructure (perfmon.exe, logman.exe, relog.exe etc).
I personally use XML and XSLT to generate the counters. I can then decorate all my code with performance counters tracking functions being run, average call duration time, number of executions, rate of executions per second and so on and so forth. Good choice of counters will give an immediate, accurate, performance picture much faster than logging can. While logging can give more insight on certain event paths (ie. order of events that lead to certain state), logging can seldom be 'always on' as the impact on performance is significant, not only on performance but most importantly on concurrency as most existing logging infrastructures add contention.
This is not a job for logging. It's a job for a profiler.
Try one of these:
JetBrains' dotTrace - http://www.jetbrains.com/profiler/index.html
Red-Gate ANTS - http://www.red-gate.com/products/ants_profiler/index.htm
Automated QA's AQTime - http://www.automatedqa.com/products/aqtime/index.asp
While I haven't (yet) tried it for myself, it may be worth looking at Gibraltar which can be used with PostSharp to put declarative performance logging into your code.
When dealing with problems like this I try and not add any extra headache by manually adding logging / tracing & timing into the application itself. If all you want is to tune the application then I suggest getting a profiler which will show you what areas of code are an issue. I recommend Red-Gate's Ant's Profiler.
Now if you want to collect statistics for monitoring or trending purposes then a profiler is not the right tool. I have had success using PerformanceCounters which let's many third party tools pull the performance information out of the application.
So what are you trying to do solve performance problems or monitor to ensure you catch a performance problem before it becomes severe?
EDIT
Based on your comment, I would look at using performance monitors around critical sections of code, timing how long it took to complete an operation. Then you can use the built in performance monitoring tools, or any number of third party tools to monitor and trend the stats.
SQL Server keeps track of some things for you, so try running some of these queries on your system:
Wayback Machine: Uncover Hidden Data to Optimize Application Performance
here is an example from the link:
--Identifying Most Costly Queries by I/O
SELECT TOP 10
[Average IO] = (total_logical_reads + total_logical_writes) / qs.execution_count
,[Total IO] = (total_logical_reads + total_logical_writes)
,[Execution count] = qs.execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
(CASE WHEN qs.statement_end_offset = -1
THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
ELSE qs.statement_end_offset END - qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
ORDER BY [Average IO] DESC;
the link contains many queries including ones for: Costly Missing Indexes, Logically Fragmented Indexes, Identifying Queries that Execute Most Often , etc...
I would start would diagnosing what is the real cause for the perf issue? Is it CPU, Memory, Disk or IO. This can be identified by few perfmon counters.
For example Linq2SQL uses Sync I/O which could be a big bottleneck for scalability. Because it uses Sync I/O windows threads get blocked and requests would end up waiting. This is usual suspect and might not be true. Here is an MSDN article how sync I/O could affect scalability.
If CPU is an issue then the next question is application CPU bound? Then you could use one of the profilers like mentioned above. Also look for time spent on GC perfmon counter that is another usual suspect?
Last night I did a load test on a site. I found that one of my shared caches is a bottleneck. I'm using a ReaderWriterLockSlim to control the updates of the data. Unfortunately at one point there are ~200 requests trying to update the data at approximately the same time. This also coincided with CPU usage spikes.
The data being updated is in the ASP.NET Cache. What I'd like to do is if the CPU usage is around 75%, I'd like to just skip the cache and hit the database on another machine.
My problem is that I don't know how expensive it is to create a new performance counter to check the cpu usage. Also, if I would probably like the average cpu usage over the last 2 or 3 seconds. However, I can't sit there and calculate the cpu time as that would take longer than it's taking to update the cache currently.
Is there an easy way to get the average CPU usage? Are there any drawbacks to this?
I'm also considering totaling the wait count for the lock and then at a certain threshold switch over to the database. The concern I had with this approach would be that changing hardware might allow more locks with less of a strain on the system. And also finding the right balance for the threshold would be cumbersome and it doesn't take into account any other load on the machine. But it's a simple approach, and simple is 99% of the time better.
This article from Microsoft covers Tuning .Net Application Performance and highlights which counters to collect and compare to determine CPU and I/O bound applications.
You sound like you want to monitor this during execution and bypass your cache when things get intensive. Would this not just move the intensive processing from the cache calls to your database calls? Surely you have the cache to avoid expensive DB calls.
Are you trying to repopulate an invalidated cache? What is the affect of serving stale data from the cache? You could just lock on the re-populating function and serve stale data to other requests until the process completes.
Based on the above article, we collect the following counter objects during our tests and that gives us all the necessary counters to determine the bottlenecks.
.NET CLR Exceptions
.NET CLR Memory
ASP.NET Applications
ASP.NET
Memory
Paging File
Processor
Thread
The sections in the article for CLR Tuning and ASP.NET Tuning highlight the bottlenecks that can occur and suggest configuration changes to improve performance. We certainly made changes to the thread pool settings to get better performance.
Changing and Retrieving Performance Counter Values might help with accessing the existing Processor counter via code but this isn't something I've tried personally.
I'm seeing consistently high CPU usage for my ASP.NET web application (on the live production box only, naturally....!) and I'm trying to narrow down the cause - it's basically maxing out a quad core Xeon box and there's no way it should be able to do that!
The CPU usage of the web process is generally higher than that of the DB process - which rings alarm bells to me on its own (?).
However, using the standard profiling tools (dotTrace, Red Gate etc) only show you the time spent in individual methods (rather than actual CPU usage) - and ultimately still highlight methods that are DB-bound. While this might indicate opportunities for caching or better indexes, I don't see how that in itself would result in high CPU usage of the web application process?
Any suggestions or tips as to how I can narrow this down?
Thanks!
Some suggestions to try at the first place.
1.Deploy with Release Build
Check whether the deployed product is in release mode. By running in debug mode, lot of time is wasted loading the pdbs along with the assemblies.
2.Disable ViewState
Disable viewstate if its not required. ViewState is nothing but data stored in hidden fields to be persisted between requests. it increases the total payload of the page both when served and when requested. There is also an additional overhead incurred when serializing or deserializing view state data that is posted back to the server. Lastly, view state increases the memory allocations on the server.
3.Disable Session State:
If you are not going to use it disable Session State. By default it’s on. You can actually turn this off for specific pages or for the whole application.
There are some basic ASP.NET application performance monitoring, check these two MSDN articles
"Monitoring ASP.NET Application Performance" and Performance Counters for ASP.NET
Can you set up some unit tests to call various methods and see what their impact is on processor usage? Visual Studio has some testing tools built in if you're using Team System, but even if you're not, you could write a multithreaded tester to call particular functions hundreds of times.
If you'd like some pointers on how to do this, I can help you build some basic unit testing.
are you recording/reporting unhandled exception? If not do so and check if any of them correspond with your high CPD spikes you may have a stack overflow causing the spikes.
http://msdn.microsoft.com/en-us/library/ms998306.aspx
You could also look into recoding the time of each request by using a HttpModule and checking which requests are taking up the most time which may indicate the pages that are causing the issue.
As Pradeepno notes, the place to start with is really performance counters--they can give you a very good idea of what is consuming what part of the CPU.
The web app usage being higher than DB usage isn't entirely suprising. If you have decent db design, most web apps are barely going to cause a decently powered DB server to break a sweat.