I am working with a large web application where there can be up to 100,000 objects, populated from a DB, in cache.
There is a table in the database which, given the object ID, will give you a last_updated value which is updated whenever any aspect of that object changes in the DB.
I have read creating an SqlCacheDependency (one row in a table per object) per object, which such high number of objects is a no-go.
I am looking for alternative solutions. One such possible solution I thought of is to cache the "last_updated" table as a datastructure and create a cache dependency to the table it is based on. Then whenever one of the 100,000 objects is requested, I check the cached "last_updated" table and if it is out of date, I fetch the object again from the database and re-cache it. If it is not out of date, I give the cached version. Does this seem like a reasonable solution?
But.. how you can do it for a single row of the table.. In ASP.Net you can create SQl Server dependency which uses a broker Service.. and puts the data into the cache and whenever the table is updated.. the cache will be rejected and new data is taken from db and put into the cache..
i Hope this might give you some idea!
Related
I have a website that runs a stored procedure when you open home page. That stored procedure process data from 4 relational table and gives a result. Since DB records increased, completion of the stored procedure can take more than 10 seconds and it is too much for a home page.
So I think, inserting result of the stored procedure into a new table regularly and using that table for home page can be a good idea to solve the problem but I am not sure if it is a good practice for SQL Server.
Is there any better solution for my case?
Edit: Those 4 tables are updated every 15 minutes with about 30 insert.
If you are willing to have a "designated victim" update the cache as needed (which may also cause other users to wait) you can do something like this in a stored procedure (SP):
Start a transaction to block access to the cache.
Check the date/time of the cache entries. (This requires either adding a CacheUpdated column to the cache table or storing the value elsewhere.)
If the cached data is sufficiently recent then return the data and end the transaction.
Delete the cached data and run a new query to refill it with an appropriate CacheUpdated date/time.
Return the cached data and end the transaction.
If the update time becomes too long for users to wait, or the cache rebuild blocks too many users, you can run a stored procedure at a scheduled interval by creating a job in SQL Server Agent. The SP would:
Save the current date/time, e.g. as #Now.
Run the query to update the cache marking each row with CacheUpdated = #Now.
Delete any cache rows where CacheUpdate != #Now.
The corresponding SP for users would simply return the oldest set of data, i.e. Min( CacheUpdated ) rows. If there is only one set, that's what they get. If an update is in progress then they'll get the older complete set, not the work in progress.
As far as you have explained your issue I see no problem in doing that, but you must explain more, since we don't know what type of data you collecting and how it is increases every time, so as to provide you a better solution
I have a page on which data table is created programmatically if the data is not there in the database tables. I have to use this data table in many events during postbacks. The data table may contain hundreds of records and there may be multiple users accessing the same page(Of course with different data source for each user). I am storing the data table in view state but I am afraid that this practice will make the page heavier. Is there any other way to preserve the data table across postbacks.The code is very long so I can not copy and past it here.
Using session will again make the whole application heavier...So is it better choice over viewstate??
You should use Session. Also it's possible to use Application or Cache but you'll have to generate and store a unique key on your page to negate possible interferention between requests from different users.
In your case the view state can get very big and will hurt page load performance. IMHO the best thing would be to revise the way your are handling the post back events.
Use Caching if more than one user needs the same data.
Using the Session if the data is specific for each zuser. But keep in mind, that if you are in a
clustered invironment it has some pitfalls.
Load the data from the database each time the user posts back to the server. No Statehandling on the server needs to be done but you loose performance while doing a network roundtrip.
For a quick fix I usually store the View State on the server. Refer to this page to read about it... http://aspguy.wordpress.com/2008/07/09/reducing-the-page-size-by-storing-viewstate-on-server/
May be you need to store the data in the Cache object
If the data table is different for each user, you should use Session or you could use Cache , assumed to make a different Cache object for each user.
But if the data table is very big probably is not a good idea to store it in memory instead of direct db access.
If data is user specific then you can use Session. However, in case of out of process session state, you may have issues because all that data needs to be marshaled back & forth from session store.
Otherwise Cache is a good option but you need to choose cache period and expiration policy on typical usage scenarios (and also need to handle cache expiry scenario gracefully).
Yet another option is to push the data into temp file - however, in such case, you need to manage file clean up etc.
I'm working on a web application, on one page I am inserting records in the database and I want to display the data in a GridView but on a diffrent page. How can I do this?
I know how to display records in a GridView, but I want to know if there are two web pages,
on one page provides the facility to insert the records and U want to display the records in the GridView bit on the second page.
While it is possible to retain the data being inserted without retrieving it from the database, I think it is better to save the data on the first page and retrieve it from the database on the second page.
You can do this by writing inline SQL or a stored procedure. One simple approach would be to pass the resultset into a DataTable and bind a GridView to that.
That does involve more work -- more code and more trips to the database. However, I think it is very useful when performing INSERTs that the web page is updated to display what actually got into the database. Sometimes, this is different from what the user thinks they entered, and they can see the problem immediately.
One question would be how to identify the data that has just been inserted. I can think of several ways to do that. One is to query for all records entered today by the person logged in (which is recorded in the CreatedBy and CreatedDate columns of the database tables). Sort the resultset in descending order of CreatedDate, so that the most recent entries appear at the top of the GridView. Another would be by assigning a batch number to the data entry and retrieving only the data in that batch.
If you really want to hang on to the data entry, you could put it into Session on the first page, and then retrieve it from Session for display on the second page.
Following along the lines of what DOK said, it's also a lot easier to validate data entered by your users in your business logic before you submit it to the database.
Secondly, users can change their minds about data on a webpage frequently. The data on the web could be in an partially-finished state or could have typos or errors in it. If someone else saw this data and believed that it needed to be completed, then you could end up with duplicated entries in the database that would then require reconciliation.
Honestly, your best bet is to use the Session object to hold temporary user data. The MSDN entry for the GridView RowEditing event contains some great source code for this approach. Whenever I have to use GridViews to handle data from the database, I mimic this.
In addition to handling problems with temporary data storage, you can compare the Session object to your database results to determine whether or not new rows have been inserted. This is somewhat costly as it involves overloading the Equals method (and GetHashCode as well, if you follow what Microsoft recommends) and using Equals to iterate over the two collections, comparing the properties of both objects, and determining which records are new based on records that don't exist in your Session object, but do exist in your database object.
It's also worth noting that this approach assumes that you don't delete data from your database, but set the status of a record in your database to "Deleted" -- if that's a boolean field or an sequence of codes you use to describe the state of rows in a table.
I am developing a website for a client (ASP.NET, T-SQL). It is a data-entry website allowing many of their users to login and manipulate records in the same database.
There are instructions (basically a list of string) throughout the form, telling the users what to do for each section; these instructions are themselves present in the database.
On each login, I store these instructions in the Session[] object per authenticated user. The instructions are identical for everyone.
I've looked at a solution which suggested storing a common session identifier in the database and then querying it to re-use that particular session but this seems very hacky. What is a best-practices solution to accomplish this? Is there a 'common' object available to all users?
Firstly, does it matter at this point? Yes, it's bad practice and inefficent, but if you're storing 20Kb of strings in memory and have a maximum of 100 users, that's 2,000Kb of data. Hardly a lot of memory "wasted". Even at 200Kb of strings, that's 20,000Kb of data. Again, not a lot. Is it worth your time, and the client waiting for you to solve it, right now?
If you decide it is then you could:
Store the strings in the Application object or a static class so that they're retrieved once and used many times.
Retrieve the strings on every page view. This may not be as performance damaging as it seems.
Use something like the Cache class in System.Web.Caching.
Make use of Output Caching.
Make use of Windows Server AppFabric "Velocity" memory cache.
Sounds to me like you're looking for the Application Cache. Like the Session, it is an in-memory cache of data. Unlike the session, it is shared among all users; each user doesn't get their own individual copy of the data. Also, when you add data elements to the cache, you can specify criteria which will automatically invalidate that data, and cause it to be reloaded/refreshed (useful when your seldom-changing data actually does change :).
Here's some articles which should give you everything you need to know about using the Application cache (and some other caching options within ASP.NET as well):
ASP.NET Caching Overview
Using the ASP.NET Application Cache to Make Your Applications Scream
Caching Data at Application Startup
.NET Data Caching
I would suggest using the application-level Cache object. It is available everywhere as part of HttpContext. You can populate it on App_Start.
You can put any kind of object into Cache, though obviously, the smaller the better.
Here are some examples of how to populate it using C#:
1) Add items to the cache as you would add items to a dictionary by specifying the item's key & value.
Example: add the current Value property of a text box to the cache.
Cache["txt1"] = txtName.value;
or
Cache["result"] = dataset;
2) The Insert method is overloaded, allowing you to define values for the parameters of the version you're using.
Example: add only an item key & value:
Cache.Insert("MyData1", connectionString);
3) The Add method has the same signature as the Insert method, but it returns an object representing the item you added.
Cache.Add("MyData1", connectionString);
To retrieve the from cache:
stringName = Cache["MyData"];
If the cached data is not a string, you may need to cast it to the proper data type.
result = (DataSet)Cache["result"];
One of the benefits of using the Cache object as opposed to the Application object is that the CLR will dump contents of Cache if the system is in danger of running out of memory.
I have data entry form like...
Data Entry Form http://img192.imageshack.us/img192/2478/inputform.jpg
There are some empty rows and some of them have values. User can Update existing values and can also fill value in empty rows.
I need to map these values in my DB table and some of them will be inserted as new rows into the database and existing record will be updated.
I need your suggestions, How can I accomplish this scenario with best approach.
Thanks
For each row, I would have a primary key (hidden), a dirty flag, and a new flag. In the grid, you would set the "dirty" flag to true when changes are made. When adding new rows in the UI, you would set the new flag as well as generate a primary key (this would be easiest if you used GUIDs for the key). Then, when you post this all back to the server, you would do inserts when the new flag is set and updates for those with the dirty flag.
Once the commit of the data has completed, you would simply clear the dirty and new flags.
Of course, if the data is shared by multiple contributors and can be edited concurrently, there's a bit more involved if you don't want someone overwriting another's edits.
I would look into using ADO.net DataSets and DataTables as a backing store in memory for your custom data grid. ADO.net allows you to bulk load a data set out of the database and track inserts, updates, and deletes against that data in memory. Once you are done, you can then bulk process the stored transactions back into the database.
The big benefit of using ADO.net is that all the prickly change tracking code is written for you already, and the library is deployed to every .net capable machine.
While it isn't in vogue right now, you can also send ADO.net data sets across the wire using XML serialization for altering and then send it back to be processed into the database.
Google around. There are literally thousands of books, tutorials, and blog posts on how to use ADO.net.