I have a website that runs a stored procedure when you open home page. That stored procedure process data from 4 relational table and gives a result. Since DB records increased, completion of the stored procedure can take more than 10 seconds and it is too much for a home page.
So I think, inserting result of the stored procedure into a new table regularly and using that table for home page can be a good idea to solve the problem but I am not sure if it is a good practice for SQL Server.
Is there any better solution for my case?
Edit: Those 4 tables are updated every 15 minutes with about 30 insert.
If you are willing to have a "designated victim" update the cache as needed (which may also cause other users to wait) you can do something like this in a stored procedure (SP):
Start a transaction to block access to the cache.
Check the date/time of the cache entries. (This requires either adding a CacheUpdated column to the cache table or storing the value elsewhere.)
If the cached data is sufficiently recent then return the data and end the transaction.
Delete the cached data and run a new query to refill it with an appropriate CacheUpdated date/time.
Return the cached data and end the transaction.
If the update time becomes too long for users to wait, or the cache rebuild blocks too many users, you can run a stored procedure at a scheduled interval by creating a job in SQL Server Agent. The SP would:
Save the current date/time, e.g. as #Now.
Run the query to update the cache marking each row with CacheUpdated = #Now.
Delete any cache rows where CacheUpdate != #Now.
The corresponding SP for users would simply return the oldest set of data, i.e. Min( CacheUpdated ) rows. If there is only one set, that's what they get. If an update is in progress then they'll get the older complete set, not the work in progress.
As far as you have explained your issue I see no problem in doing that, but you must explain more, since we don't know what type of data you collecting and how it is increases every time, so as to provide you a better solution
Related
I have a SQL Azure database on which I need to perform some data archiving operation.
Plan is to move all the irrelevant data from the actual tables into Archive_* tables.
I have tables which have up to 8-9 million records.
One option is to write a stored procedure and insert data in to the new Archive_* tables and also delete from the actual tables.
But this operation is really time consuming and running for more than 3 hrs.
I am in a situation where I can't have more than an hour's downtime.
How can I make this archiving faster?
You can use Azure Automation to schedule execution of a stored procedure every day at the same time, during maintenance window, where this stored procedure will archive the oldest one week or one month of data only, each time it runs. The store procedure should archive data older than X number of weeks/months/years only. Please read this article to create the runbook. In a few days you will have all the old data archived and the Runbook will continue to do the job from now and on.
You can't make it faster, but you can make it seamless. The first option is to have a separate task that moves data in portions from the source to the archive tables. In order to prevent table lock escalations and overall performance degradation I would suggest you to limit the size of a single transaction. E.g. start transaction, insert N records into the archive table, delete these records from the source table, commit transaction. Continue for a few days until all the necessary data is transferred. The advantage of that way is that if there is some kind of a failure, you may restart the archival process and it will continue from the point of the failure.
The second option that does not exclude the first one really depends on how critical the performance of the source tables for you and how many updates are happening with them. It if is not a problem you can write triggers that actually pour every inserted/updated record into an archive table. Then, when you want a cleanup all you need to do is to delete the obsolete records from the source tables, their copies will already be in the archive tables.
In the both cases you will not need to have any downtime.
I have 5000 records I am calculating salary of one user and update his data in database. So it’s taking quit long to update 5000 records. I want to calculate all users’ salary first and then update to records in db.
Is there any other way we can update db in single click
It really depends on how your are managing your data access layer and what data you need for doing the calculation? Do you have all the data you need in just one table or for each record you need to fetch some other data from another tables?
One way is to retrieve each record and do the calculation in a transaction and then store it on the database. In this way, you can also take advantage of ajax UI to inform the user about the progress of calculation. In this way, you should use SqlDataReader to fetching the data as it is very optimized and has less overhead than using DataSet and DataTables and also you can prevent several type-castings. In addition, you can also make it optimized by taking advantage of TPL or make it configurable for fetching/updating N records each time. This approach works if you have the ID of the records. You also need to have a field for your records to track your calculations in case of any disconnection or crashes or iisreset execution so that you can resume the calculation instead of rerunning it again.
I am collecting data every second and storing it in a ":memory" database. Inserting data into this database is inside a transaction.
Everytime one request is sending to server and server will read data from the first memory, do some calculation, store it in the second database and send it back to the client. For this, I am creating another ":memory:" database to store the aggregated information of the first db. I cannot use the same db because I need to do some large calculation to get the aggregated result. This cannot be done inside the transaction( because if one collection takes 5 sec I will lose all the 4 seconds data). I cannot create table in the same database because I will not be able to write the aggregate data while it is collecting and inserting the original data(it is inside transaction and it is collecting every one second)
-- Sometimes I want to retrieve data from both the databses. How can I link both these memory databases? Using attach database stmt, I can attach the second db to the first one. But the problem is next time when a request comes how will I check the second db is exist or not?
-- Suppose, I am attaching the second memory db to first one. Will it lock the second database, when we write data to the first db?
-- Is there any other way to store this aggregated data??
As far as I got your idea, I don't think that you need two databases at all. I suppose you are misinterpreting the idea of transactions in sql.
If you are beginning a transaction other processes will be still allowed to read data. If you are reading data, you probably don't need a database lock.
A possible workflow could look as the following.
Insert some data to the database (use a transaction just for the
insertion process)
Perform heavy calculations on the database (but do not use a transaction, otherwise it will prevent other processes of inserting any data to your database). Even if this step includes really heavy computation, you can still insert and read data by using another process as SELECT statements will not lock your database.
Write results to the database (again, by using a transaction)
Just make sure that heavy calculations are not performed within a transaction.
If you want a more detailed description of this solution, look at the documentation about the file locking behaviour of sqlite3: http://www.sqlite.org/lockingv3.html
I'm generating pages based on an sql query.
This is the query:
CREATEPROCEDURE sp_searchUsersByFirstLetter
#searchQuery nvarchar(1)
AS
BEGIN
SET NOCOUNT ON;
SELECT UserName
FROM Users Join aspnet_Users asp on Users.UserId = asp.UserId
WHERE (LoweredUserName like #searchQuery + '%')
I can call this procedure for each letter in the alphabet, and get all the users that start with that letter. Then, I put these users into a list on one of my pages.
My question is this: would it be better to cache the list of users to my webserver, rather than query the database each time? Like this:
HttpRuntime.Cache.Insert("users", listOfUsersReturnedFromQuery, null, DateTime.Now.AddHours(1), System.Web.Caching.Cache.NoSlidingExpiration);
Its ok for me if the list of users is an hour out of date. Will this be more efficient that querying the database each time?
Using a cache is best reserved for situations where your query meets the following constraints:
The data is not time critical, i.e. make sure a cache hit won't break your application by causing your code to miss a recent update of the data.
The data isn't sequenced, i.e. A, B, C, D, E are cached, F is inserted by another user, your user inserts G and hits the cache, resulting in ABCDEG instead of ABCDEFG.
The data doesn't change much.
The data is queried and re-used frequently.
Size isn't really a factor unless it's going to really tax your RAM.
I have found that one of the best tables to cache is a settings table, where the data is practically static, gets queried on nearly every page request, and changes don't have to be immediate.
The best thing for you to do would be to test which queries are performed most, then select those that are taxing the database server highest. Out of those, cache anything you can afford to. You should also take a look at tweaking maximum cached object ages. If you're performing a query 100 times a second, you can cut that rate down by a factor of 99% by simply caching it for 1 second, which negates the update delay problem for most practical situations.
In case if you have few servers memory cashing isn't so good, because it will take memroy in each server and in each w3p process of every server.
It will be also hard to maintain consistent data.
I would advise to choose from:
basic output cache (assuming you are using MVC this is zero efforts and good imporevement)
Db cache using smaller pre-calculated table where you have mapping from input string to 10 possible results
It really depends. Do you bottleneck at your database server (I would hope that answer is no)? If you are hitting the database 26 times, that is nothing compared to what typically happens. You should be considering caching data in a Dataset or some other offline model if you are hitting the database hundreds of thousands of times.
So I would say, no. You should be fine with your round trips to the database.
But there is no replacement for testing. That'll tell you for sure.
Considering that each DB call is always expensive in terms of network and DB load I would prefer to avoid such extra operations and cache items even they are requested few times per hour.
Only one opposite case I see - when amount of users in terms of memory consumption is a tons of megabytes.
Well Caching data and get back is fastest but it also depends on the data size...If there is large amount of data than it will cause performance issue.
So it almost depends on your requirement.
I would like you to suggest make use of paging or make use of mix mode by loading half of the user put in cache and than load the other data when require....
I'm trying to figure out how to develop the on my database. E.g. there are some records in the database:
alt text http://img109.imageshack.us/img109/2962/datax.png
So, if the actual DateTime on the server is 2010-01-09 12:12:12 the record no.1 must be deleted.
OK, but what, if in the datebase are e.g. 1.000.000 records? The server has to search in the database on each second to check what rows must be deleted ? It's not efficient at all.
I'm totally new to Microsoft Server so I'd be grateful of any kind of help
There isn't a time based trigger in sql server. So you are going to have to implement this as a job or through some other scheduled mechanism.
Most likely you will want an index on the StartDate (end date?) column so that your deletion query doesn't have to perform a full table scan to find the data it needs to delete.
Usually you don't actually perform deletes every second. Instead the app should be smart enough to query the table in a way to eliminate those records from it's result set. Then, you can perform lazy deletes at some other time interval to do cleanup. Such as once an hour or once a day etc.