SQL Server profiler is great for profiling SQL Server performance for web apps. However, when I'm testing my webapp I'd like a summary of database hits/duration per page.
Does anybody know of any utilities for giving you this kind of information?
If you want duration per page, I'd recommand Google Analytics.
If you want a summary of database hits (ie, you run three procedures during one page load so you want to show a count of three) then I would recommend adding auditing code to your sprocs.
Alternately (though more expensively in terms of processing) you could turn on either SQL Profiler or SQL Trace and then track the database hits that way to perform statistical analysis on them.
I would recommend setting a data access routine that will be used for all the site.
This routine/class/or whatever you like could log in the database or in a log all the "hits", their duration, error (is any), timeout, etc.
If you program it properly, you will be able to know how many DB hit / page load, avg(DBHit) + you will get as a free bonus the "longest SProc, shortest, more often called".
The positive side of this is that you don't need to modify any stored proc and you can have a nice little "wrapper" around your access to the DB.
For the "Duration per page", if you go with google analysis you will not be able to merge the information back with what you got on the database server. So I would recommend logging each access to a page in the DB.
Then you can infer that Page1.StartTime = getdate(), Page1.EndTime = (page2.Starttime-1 or session.log_off_time) for example. [This is a little basic but according to your environment you can improve it].
Related
We have a system running Windows Server 2008R2 x64 and SQL Server 2008R2 x64 with SSRS installed/configured. This is a shared reporting server used by a large number of people, with some fairly large inefficient databases (400-500gb of data ish), and these users use the system to generate ad-hoc reports based of a reporting model that sits on top of the aforementioned databases. Note that the users are using NTLM to logon and identify for running reports.
Most reports are quick, but if you are running a report for 1 or 2 years worth of data, they can take a while to return (5minutes ish). This is fine for most users, however some of the users are stuck behind a proxy, which has a connection timeout set at 2minutes. As SSRS 2008R2 does not seem to send back a "keep-alive" signal (confirmed via wireshark), when running one of these long reports the proxy server thinks the connection has died, and as such it just gives up and kills the connection. This gives the user a 401 or 503 error and obviously cancels the report (the incorrect error is a known bug in SSRS which Microsoft refuse to fix).
We're getting a lot of flak from the user's about this, even though it's not really our issue..so I am looking for a creative solution.
So far I have come up with:
1) Discovering some as yet unknown setting for SSRS that can make it keep the connection alive.
2) installing our own proxy in between the users and our reports server, which WILL send a keep-alive back (not sure this will work and it's a bit hacky, just thinking creatively!)
3) re-writing our reports databases to be more efficient (yes this is the best solution, but also incredibly expensive)
4) ask the experts :) :)
We have a call booked in with Microsoft Support to see if they can help - but can any experts on Stack help out? I appreciate that this may be a better question for server fault (and I may post it there) but it's a development question too really :)
Thanks!
A few things:
A. For SSRS overall on it's service:
I personally use a keep alive service as I believe the default recycle is 12 hours for SSRS server. I use a tool someone turned me onto called 'VisualCron' that can do many task processes automatically. You can also just make a call in a WCF service or similar to. Basically I know the first report from a user for the day is generally slow. Usually you need to hit http:// (servername)/ReportServer to keep it alive.
B. For cachine report level items:
If this does not help I would suggest possibly caching DataSets when possible. Some people have data that is up to the moment but for a lot of people that is not the case. You may create a shared dataset in SSRS and then cache that on a schedule. So if you have domain like tables that only need to be updated once in a blue moon put them there. Same with data that is nightly or in batches. If you are transactional based shop that is up to the moment this may not help but for batch based businesses this can help tremendously.
You can also cache the reports for their data as a continuation of this. Under 'Manage' drop down for a report when in the /Reports landing page you can set the data to run under a specific schedule. You can also set a snapshot which is an extension of this as it executes with some default parameters set on a schedule and is a copy of the report when it was ran.
You are mentioning ASP.NET so I am not certain how much some of this will work if you are doing this all through a site you are setting up internally as a pass through. But you could email or save files on a schedule as well through SSRS's subscription service.
C. Change how you store your data for reporting.
You can create a Report Warehouse of select item level values of queries. Create a small database that is just a few recent years of data and only certain fields and certain tables. Then index it to death and report off of that. In my experience this method will fly in terms of performance but it does take the extra overhead of setting it up. Generally most companies will whine about this but it often takes a single day to set up and then you create one SSMS job that does it all nightly or an SSIS package then you don't worry about it. I like this method as I know my data is not being reported off of production and is isolated personally.
I have a requirement to create a high performance asp.net web application. The page when requested needs to pull in some financial rates, these rates originate from a legacy system where directly calling it with each page load would result in a substantial performance loss.
My initial idea was to have a cached version of the rates sitting in SQL server and pull from there. This way when rates do change the legacy system can call a web service which can update this SQL table.
Still I wasn't happy leaving it like that - this would mean a lookup to SQL with each page request.
I started thinking about using a global variable. To hold the rates. But then I was left wondering how the legacy system is going to update the asp.net web application global variable value. This could get tricky.
So what is the best approach given performance is key. I'm thinking that call to SQL server might not be a bad idea, since it will be a straight table select. But then again - maybe there is another way to do this which is better?
Thanks in advance....
You can cache using SqlCacheDependency. So no need to worry about updating data. Once your data is changed in the database, it will reset the cache and then next request will rebuild the cache.
I have developed a CRM for my company. Next I would like to take that system and make it available for others to use in a hosted format. Very much like a salesforce.com. The question is what type of database structure would I use. I see two options:
Option 1. Each time a company signs up, I clone the master database for them.
The disadvantage of this is that I could end up with thousands of databases. Thats a lot of databases to backup every night. My CRM uses cron jobs for maintanance, those jobs would have to run on all databases.
Each time I upgrade the system with a new feature, and need to add a new column to the database, I will have to add that column to thousands of databases.
Option 2. Use only one database.
At the beginning of EVERY table add "CompanyID". In EVERY SQL statement add "and
companyid={companyid}".
The advantage of this method is the simplicity of only one database. Just one database to backup each night. Just one database to update when needed.
The disadvantage is what if I get 1000 companies signing up, and each wants to store data on 100,000 leads, that 100,000,000 rows in the lead table, which worries me.
Does anyone know how the online hosted CRMs like salesforce.com do this?
Thanks
Wouldn't you clone a table structure style to each new database id all sheets archived in master base indexed client clone is hash verified to access specific sheet run through a host machine at the front end of the master system. Then directing requests as primary role. Internal access is batched to read/write slave systems in groups. Obviously set raid configurations to copy real time and scheduled. Balance requests for load to speed of system resources. That way you separated the security flawed from ui and connection to the retention schema. it seems like simplified structures and reducing policy requests cut down request rss in the query processing. or Simply a man in the middle approach from inside out.
1) Splitting your data into smaller groups in a safe, unthinking way (such as one database per grouping) is almost always best if you want to scale. In this case, unless for some reason you want to query between companies, keeping them in separate databases is best.
2) If you are updating all of the databases by hand, you are doing something wrong if you want to scale. You'd want to automate the process.
3) Ultimately, salesforce.com uses this as the basis of their DB infrastructure:
http://blog.database.com/blog/2011/08/30/database-com-is-open-for-business/
My client is asking for a "suggestion" based lookup to be added to a particular form field.
In other words, as you start typing into a field there should be a "Google style" popup which suggests possible results to select from. There will be in the order of "tens of thousands" of possible suggestions - this is the best estimate I currently have on the quantity.
Using AJAX to send/retrieve the result, my question is whether it is better to store ALL the suggestions within .NET cache and process there, or whether it's better to run a stored-procedure based query on SQL Server for each request?
This would be using .NET 2.0 and SQL Server 2005
There is one trick I use every time when faced with such task. Do not do it on every keystroke. Put the launching of the search on a sliding timeout.
The intent here is to launch the search only when the user paused in his/her typing. Usually I set the timeout at .1 to .2 sec. Psychologically it is still instantaneous, but it considerably reduces the load on whatever you will use to search
Your bottle neck will be transporting the data from the server to the browser. You can easily produce a large result from the database in almost no time at all, but it takes forever to return to the browser.
You will have to find a way to limit the result that the server returns, so that you only fetch what the user has to see right now.
When you have the data traffic down to reasonable level, you can start looking at optimising the server part. With some caching and logic that should be easy, considering that when the user types the result is often a subset of the previous result, e.g. the match for "hat" is a subset of the match for "ha".
When I've seen the "suggest as you type" type searches done in SQL Server environments, I've seen the best performance using some sort of a caching mechanism, and typically a distributed approach - like a memcached, typically. Even if your code is optimized well, your database is tuned well and you have your query taking only a <= 10ms with the call to it, process and return, that is still 10ms as they type.
It depends on the number of items. If you can cache the items in a .NET process without running out of memory this will defenitely be faster.
But if that can't be done you are stuck with accessing the database on each request. A stored procedure would be a nice way to do that.
But there are some design patterns which can guide you. I've used the following two while developing something similar to what you are doing.
Submission throttling
Browser side cache
I was wondering what is the easiest way to see total number of database queries from my ASP.Net (.NET 2.0) application.
My application heavily use sql 2005 database because all data are dynamic and everything goes through one connection string in web.config. Connection pooling is enabled there.
So, I am wondering how many select statements are executed for particular page I load in my browser.
I don't care if I can see that information from .net side or from db side as long as I can see only connections to MY database. Not all connections to that db server because I use shared db server and there is a lot of other databases.
The best way to do this is to set up a profiler on your database and then make a single request to your ASP.NET application. The profiler will aggregate any data you wish and you will be able to use that data to determine what queries were sent to SQL Server from your application.
The SQL Server Profiler will list all actions performed on your DB. If you use a different db login name for your project (probably a really good idea if you are not) you can filter so it only shows actions from your login (see Events Selection, Column Filters then Login Name).
Use SQL Profiler. You can configure it to filter by the database you want and to just show select statements.
If you have some sort of database layer in your code, you could modify it to write out a log message every time you run a select statement. Then just load the page once and count the number of log statements. This may or may not work, depending on how your code is structured, but it's an option.
Edit: I misread the question. I thought you had multiple clients connecting to the same database, not the same database server. In that case, a profiler probably is the best choice.
Do you have access to SQL Server Profiler? You can set up traces to monitor this sort of thing by loading a page and looking at the effects in the profiler.
JUst be aware that Profiler can affect performance, so it is best to do this on dev.