I'm trying to create a sort of global settings for a website and store this data on the database, however I keep thinking that may not be very efficient as these settings will have to be read on every request.
The type of settings like 'how many records to show per page', enable/disable things, I plan to store this on the database but don't want the overhead of having to call the database on every request to get the settings, specially when they don't change. Surely this is done all the time on CMS's, how do you think it should be done. I am thinking SqlCacheDependency but never set that up. Is there another way?
Also on the cards is the possibility to store those settings on web.config and create a GUI for it, the problem is that the administration of the site runs on it's own namespace and has it's own web.config, so the question here is if it's possible to manipulate a web.config outside the application namespace.
Thanks guys.
I would suggest to you to read the values from the DB in the Application_Start and store these values in the Application object. In this way you will not need to go to the DB to read the values every time. It will only read and store values once when the application starts.
void Application_Start(object sender, EventArgs e)
{
Application["name"] = ""; Value from DB
......................
......................
}
Note: It is not recommend to manipulate values in the web.config using UI, because once a user tries to modify any value from the UI, that changes the value in the web.config, all your User session will be terminated.
Another Note: Every time you change the information and update your DB, you will need to update the Application level object as well.
First, I wouldn't start worrying about counting database calls until you have an idea that your database calls are actually hurting performance. Modern databases are quick, especially for well-designed queries for scalar values. Lots of popular packages read config on every request and they seem to scale pretty well.
As for updating these values, you can pretty easily update a web.config file from another app presuming you've got the right permissions -- this will definitely require Full Trust which leaves out most hosting scenarios. Thing to remember is that, while VS treats the file special, it is just an XML file so the normal tricks for updating an XML file will apply. Because you are doing this from a different application, it will work. But Muhammad's warning about dumping user sessions would apply to the "victim" of the update. Which might be OK depending on what you are changing.
Related
I have two separate web applications:
The "admin" application where data is created and updated
The "public" application where data is displayed.
The information displayed on the "public" changes infrequently, so I want to cache it.
What I'm looking for is the "simplest possible thing" to update the cache on the public site when a change is made in the admin site.
To throw in some complexity, the application is running on Windows Azure. This rules out file and sql cache dependencies (at least the built in ones).
I am running both applications on a single web role instance.
I've considered using Memcached for this purpose. but since I'm not really after a distributed cache and that the performance is not as good as using a memory cache (System.Runtime.Caching) I want to try and avoid this.
I've also considered using NServiceBus (or the Azure equivalent) but again, this seems overkill just to send a notification to clear the cache.
What I'm thinking (maybe a little hacky, but simple):
Have a controller action on the public site that clears the in memory cache. I'm not bothered about clearing specific cached items, the data doesn't change enough for me to worry about that. When the "admin" application makes a cache, we make a httpwebrequest to the clear cache action on the public site.
Since the database is the only shared resource between the two applications, just adding a table with the datetime of the last update. The public site will make a query on every request and compare the database last update datetime to one that we will hold in memory. If it doesn't match then we clear the cache.
Any other recommendations or problems with the above options? The key thing here is simple and high performance.
1., where you have a controller action to clear the cache, won't work if you have more than one instance; otherwise, if you know you have one and only one instance, it should work just fine.
2., where you have a table that stores the last update time, would work fine for multiple instances but incurs the cost of a SQL database query per request -- and for a heavily loaded site this can be an issue.
Probably fastest and simplest is to use option 2 but store the last update time in table storage rather than a SQL database. Reads to table storage are very fast -- under the covers it's a simple HTTP GET.
Having a public controller that you can call to tell the site to clear its cache will work as long as you only have one instance of the main site. As soon as you add a second instance, as calls go through the load balancer, your one call will only go to one instance.
If you're not concerned about how soon the update makes it from the admin site to the main site, the best performing and easiest (but not the cheapest) solution is to use the Azure AppFabric Cache and then configure it to use a a local (in memory) cache with a short-ish time out (say 10 minutes).
The first time your client tries to access an item this would be what happens
Look for the item in local cache
It's not there, so look for the item in the distributed cache
It's not there either so load the item from persistent storage
Add the item to the cache with a long-ish time to live (48 hours is the default I think)
Return the item
Steps 1 and 2 are taken care of for you by the library, the other bits you need to write. Any subsequent calls in the next X minutes will return the item from the in memory cache. After X minutes it falls out of the local cache. The next call loads it from the distributed cache back into the local cache and you can carry on.
All your admin app needs to do is update the database and then remove the item from the distributed cache. The next time the item falls out of the local cache on the client, it will simply reload the data from the database.
If you like this idea but don't want the expense of using the caching service, you could do something very similar with your database idea. Keep the cached data in a static variable and just check for updates every x minutes rather than with every request.
In the end I used Azure Blobs as cache dependencies. I created a file change monitor to poll for changes to the files (full details at http://ben.onfabrik.com/posts/monitoring-files-in-azure-blob-storage).
When a change is made in the admin application I update the blob. When the file change monitor detects the change we clear the local cache.
I need to store settings that update from time to time about my website and was wondering what is the most efficient way to do this. Note that once these settings are changed, I need them to be accessible even after an IIS reboot, so a simple application variable is not the answer. I also know I can store application settings in my config file, but as you change them, the actual XML doesn't seem to update, so I would have to re-write the XML behind the scenes each time it changes.
As for writing to an file, such as an INI, my concern arises if as I am writing, another is trying to read. I have had IO locking errors before doing such a thing. Also I can do a database storage, but trying to keep database calls low.
Right now I am set on probably having an INI that I write to on change, on application load I pull from that so that I can always access the local variable. This would make IO issues pretty unlikely since I am only reading once. Just basically looking for some input on what is likely the most efficient way of doing this.
Thanks in advance,
Anthony F Greco
If you want to minimize database calls, I'd say put the values in the database, then cache them in your application. Depending on your needs, maybe expire the cache every 30 minutes, so when you change the DB value, it will be applied in no more than 30 minutes, and you will only make a DB call every 30 minutes (or when needed, like when your app restarts). Or you can use a SQL Cache Dependency, but I've never done that, so I don't know the pitfalls of that technique.
We moved all of our application settings from the web.config to the database as can be seen HERE.
Ours was done to get around the problems with promoting code from Dev to Test to QA to Prod, but it can be used for other reasons.
Ours is cached on application startup and then every 5 minutes it checks to see if a single counter in a table was updated. If so, it refreshes all of the settings.
I'd go with #Joe Enos
Except that I'd use an xml file or a .ini, as you'd like to limit your db calls.
A class in my ASP.NET website needs to access a database table multiple times for every request. The database table should rarely change, if ever. Maybe a couple times a month.
In the static constructor of the class, the table is fetched from the DB and cached in a static local variable. Whenever the class needs to access the table, then, it just uses the cached, static version.
My question concerns the lifespan of this cached, static version of the table.
I understand that it's fetched the first time the class is instantiated or a static method in the class is used. But how often does this occur on the web server? What if the table changes and we want to reset this static version of the table?
Basically, I'm wondering, is this table fetched once and then only refetched each time I restart IIS? What, with regard to the site and IIS, will trigger this static class to reset, causing the static table to be refetched?
Rather than a static variable on a class, why not make add this to the 'Application' collection? It's lifetime is well understood (the life of the website) and can easily be recycled by touching web.config. Populate it in your Application_Start method of global.asax.
I'd recommend using the ASP.NET cache itself, rather than having variables for each particular cache item (a single table right now, but I'm sure there's room for growth); this way you can specify expiration, among other things, such as dependencies.
You can get information on the cache here, and more specifically, using the cache here.
To answer your question about the life-cycle, or expectancy of a local variable, see this link, which should do a better job of explaining the innards than I.
Basically, I'm wondering, is this
table fetched once and then only
refetched each time I restart IIS?
Yes, you've got that spot on. Essentially, restarting IIS will cause your static variable to be "refreshed". If you use a static variable to store this kind of thing (which may not be the best solution, but I'm trying to directly answer your question without getting sidetracked), I would advise you build some code into your data layer so that your static variable is updated each time the database table in question is written to. This will mean that you dont need to bounce the server each time you update it.
Its also worth remembering that static variables are shared across all client requests, this can often lead to some unpredictable multi-threading errors.
In order to improve speed of chat application, I am remembering last message id in static variable (actually, Dictionary).
Howeever, it seems that every thread has own copy, because users do not get updated on production (single server environment).
private static Dictionary<long, MemoryChatRoom> _chatRooms = new Dictionary<long, MemoryChatRoom>();
No treadstaticattribute used...
What is fast way to share few ints across all application processes?
update
I know that web must be stateless. However, for every rule there is an exception. Currently all data stroed in ms sql, and in this particular case some piece of shared memory wil increase performance dramatically and allow to avoid sql requests for nothing.
I did not used static for years, so I even missed moment when it started to be multiple instances in same application.
So, question is what is simplest way to share memory objects between processes? For now, my workaround is remoting, but there is a lot of extra code and I am not 100% sure in stability of this approach.
I'm assuming you're new to web programming. One of the key differences in a web application to a regular console or Windows forms application is that it is stateless. This means that every page request is basically initialised from scratch. You're using the database to maintain state, but as you're discovering this is fairly slow. Fortunately you have other options.
If you want to remember something frequently accessed on a per-user basis (say, their username) then you could use session. I recommend reading up on session state here. Be careful, however, not to abuse the session object -- since each user has his or her own copy of session, it can easily use a lot of RAM and cause you more performance problems than your database ever was.
If you want to cache information that's relevant across all users of your apps, ASP.NET provides a framework for data caching. The simplest way to use this is like a dictionary, eg:
Cache["item"] = "Some cached data";
I recommend reading in detail about the various options for caching in ASP.NET here.
Overall, though, I recommend you do NOT bother with caching until you are more comfortable with web programming. As with any type of globally shared data, it can cause unpredictable issues which are difficult to diagnosed if misused.
So far, there is no easy way to comminucate between processes. (And maybe this is good based on isolation, scaling). For example, this is mentioned explicitely here: ASP.Net static objects
When you really need web application/service to remember some state in memory, and NOT IN DATABASE you have following options:
You can Max Processes count = 1. Require to move this piece of code to seperate web application. In case you make it separate subdomain you will have Cross Site Scripting issues when accesing this from JS.
Remoting/WCF - You can host critical data in remoting applcation, and access it from web application.
Store data in every process and syncronize changes via memcached. Memcached doesn't have actual data, because it took long tim eto transfer it. Only last changed date per each collection.
With #3 I am able to achieve more than 100 pages per second from single server.
I recently came across a ASP 1.1 web application that put a whole heap of stuff in the session variable - including all the DB data objects and even the DB connection object. It ends up being huge. When the web session times out (four hours after the user has finished using the application) sometimes their database transactions get rolled back. I'm assuming this is because the DB connection is not being closed properly when IIS kills the session.
Anyway, my question is what should be in the session variable? Clearly some things need to be in there. The user selects which plan they want to edit on the main screen, so the plan id goes into the session variable. Is it better to try and reduce the load on the DB by storing all the details about the user (and their manager etc.) and the plan they are editing in the session variable or should I try to minimise the stuff in the session variable and query the DB for everything I need in the Page_Load event?
This is pretty hard to answer because it's so application-specific, but here are a few guidelines I use:
Put as little as possible in the session.
User-specific selections that should only last during a given visit are a good choice
often, variables that need to be accessible to multiple pages throughout the user's visit to your site (to avoid passing them from page to page) are also good to put in the session.
From what little you've said about your application, I'd probably select your data from the db and try to find ways to minimize the impact of those queries instead of loading down the session.
Do not put database connection information in the session.
As far as caching, I'd avoid using the session for caching if possible -- you'll run into issues where someone else changes the data a user is using, plus you can't share the cached data between users. Use the ASP.NET Cache, or some other caching utility (like Memcached or Velocity).
As far as what should go in the session, anything that applies to all browser windows a user has open to your site (login, security settings, etc.) should be in the session. Things like what object is being viewed/edited should really be GET/POST variables passed around between the screens so a user can use multiple browser windows to work with your application (unless you'd like to prevent that).
DO NOT put UI objects in session.
beyond that, i'd say it varies. too much in session can slow you down if you aren't using the in process session because you are going to be serializing a lot + the speed of the provider. Cache and Session should be used sparingly and carefully. Don't just put in session because you can or is convenient. Sit down and analyze if it makes sense.
Ideally, the session in ASP should store the least amount of data that you can get away with. Storing a reference to any object that is holding system resources open (particularly a database connection) is a definite scalability killer. Also, storing uncommitted data in a session variable is just a bad idea in most cases. Overall it sounds like the current implementation is abusively using session objects to try and simulate a stateful application in a supposedly stateless environment.
Although it is much maligned, the ASP.NET model of managing state automatically through hidden fields should really eliminate the majority of the need to keep anything in session variables.
My rule of thumb is that the more scalable (in terms of users/hits) that the app needs to be, the less you can get away with using session state. There is, however, a trade-off. For web applications where the user is repeatedly accessing the same data and typically has a fairly long session per use of the site, some caching (if necessary in session objects) can actually help scalability by reducing the load on the DB server. The idea here is that it is much cheaper and less complex to farm the presentation layer than the back-end DB. Of course, with all things, this advice should be taken in moderation and doesn't apply in all situations, but for a fairly simple in-house CRUD app, it should serve you well.
A very similar question was asked regarding PHP sessions earlier. Basically, Sessions are a great place to store user-specific data that you need to access across several page loads. Sessions are NOT a great place to store database connection references; you'd be better to use some sort of connection pooling software or open/close your connection on each page load. As far as caching data in the session, this depends on how session data is being stored, how much security you need, and whether or not the data is specific to the user. A better bet would be to use something else for caching data.
storing navigation cues in sessions is tricky. The same user can have multiple windows open and then changes get propagated in a confusing manner. DB connections should definitely not be stored. ASP.NET maintains the connection pool for you, no need to resort to your own sorcery. If you need to cache stuff for short periods and the data set size is relatively small, look into ViewState as a possible option (at the cost of loading more bulk onto the page size)
A: Data that is only relative to one user. IE: a username, a user ID. At most an object representing a user. Sometimes URL-relative data (like where to take somebody) or an error message stack are useful to push into the session.
If you want to share stuff potentially between different users, use the Application store or the Cache. They're far superior.
Stephen,
Do you work for a company that starts with "I", that has a website that starts with "BC"? That sounds exactly like what I did when I first started developing in .net (and was young and stupid) -- I crammed everything I could think of in session and application. Needless to say, that was double-plus ungood.
In general, eschew session as much as possible. Certainly, non-serializable objects shouldn't be stored there (database connections and such), but even big, serializable objects shouldn't be either. You just don't want the overhead.
I would always keep very little information in session. Sessions use server memory resources which is expensive. Saving too many values in session increases the load on server and eventualy the performance of the site will go down. When you use load balance servers, usage of session can run into problems. So what I do is use minimal or no sessions, use cookies if the information is not very critical, use hidden fields more and database sessions.