Consider my dataTable contains 10,000 rows and i want to know the pitfall of storing datatable in a session variable... I want to use it until a new row has been added...
What type of session mode should i use?
Never do that, its not recommended.. It affects the performance if your server has low memory and busy processing times.
Before proceeding with this you need to consider
is your server got a available memory?
How busy your server is?
Whether the data you goin to put in session will be shared across among multiple user requests?
Why do you want to store that much data into an session.Probably you might be doing this for pagination (in a datagrid i assume), then you have to reconsider your design.
If you need the data through out your user session, then storing it in Session is good. But in your case you can go for either ViewState or Cache. You can store in ViewState is better.
ViewState["Table"] = ds.Tables[0];
to retrieve it,
DataSet ds=new DataSet();
ds = (DataSet)ViewState["Table"];
If a session variable is read before it has been assigned or if the current session times out, it will result in NullReferenceException as no null checking happens before reading from session variable.
There is no easy way to track the usage of session variables, their key names, their data types, the approximate memory being used per session, etc.
But do remember one thing putting DataSet inside the ViewState will increase the page loading.
**
The most efficient way to do what you
ask is to convert the dataset that was
returned by the DB into a Generic List
and then to use the session in order
to hold the List. List will spend less
memory then Dataset in session.
**
Firstly, do you really need that much data stored on a session level.
Could you possibly store a bit more on an application level (cache objects).
If you have a farm, that is more than one iis server, then you will have to use
either a in memory cache, or a sql cache.
With a large blob the in memory cache would be faster.
If its on a single computer in memory on that server would work.
I would seriously think about refactoring the problem.
Can you introduce paging or some other form of data window in order to reduce
the amount of data?
With dataset that big, I really dont think you should store it in session or viewstate. Storing in SQL state will have huge overhead of read and write too. Storing in memory session will significantly affect your worker process memory usage. Storing in ViewState is even worse as it will make page loading extremely slow due to page size and time taken to encode and decode the viewstate.
If you really have to do it, of all choices I will go for in memory session and pay the price. Otherwise I think you should consider refactoring your code and not having to cache datatable of a that size and pull out data in a set/page as you need to. Alternatively if you have layered architecture, have your application code cache it and when a request is made the application code can extract and return the record/subset to the UI when needed.
Related
This is my first post/question so please let me know if/how I can improve it. I found similar questions but nothing quite covered this.
When you store to InProc session you're just storing a reference to the data. So, if I have a public property foo, and I store it in Session("foo") = foo, then I haven't really taken up any additional memory (aside from the 32/64 bits used by the pointer)?
In my case, we are currently reloading foo on every page of our website, so if I were to instead store it in session, then it should be taking the same about of space, but not needing to reload on every page. I'm seeing a lot of people say not to store large objects in session, but if that large object already exists, what difference does it make to have a pointer to it? Of course I would remove the object from session the moment it was no longer needed.
The data we are trying to store is an object specific to the user's current work, but not user data. As an analogy, say the user was a car dealer, and he is looking at all the data for a particular customer. We have multiple pages for this customer, and we want to keep all the customer info loaded on each page, All the customer data is stored in a single xml data column in a SQL table, which we parse on every page.
We have tried binary serialization instead of parsing xml, so we could store with session in state server mode, but we found the performance to actually be worse.
We are running on a single web server.
First off, no. When you store something in the session state all the data required to store that object is consumed by the website process(s). Just because .NET treats variables like references doesn't mean it actually uses less memory than a no-GC language. It just means that copying that variable around is done efficiently without using reference operators or pointers.
Your question is a bit vague, but you have a few options for persisting data:
1) Send the data to the client as JSON and store it on the browser if it should be per-user and is needed more on the client side than the server side. You can then send pieces of the data with different requests if you need to (put it in hidden fields if you have to use ASPX web forms).
2) Store it in the session state if it is a small bit of per user data.
3) Store it in the ASP.NET cache if it is large and common to all users, see here (https://msdn.microsoft.com/en-us/library/6hbbsfk6.aspx).
4) If it is large and user-specific that is used primarily on the server then you have more of a performance problem. You should see if you can break out any user specific stuff from static stuff. If you do that and its still large then a database may not be a bad solution. If you are already using DB calls in your application then looking up this data on every request won't cause too much overhead and you won't have to regenerate it from scratch (You should only do this if the data takes a considerable time to generate as a DB call could be slower than just regenerating the data itself). I recommend writing some sort of middleware (HttpModule or OwinMiddleware) that uses whatever user Identity you use for auth to look up the data and then set it on the HttpContext.Current.Items collection. This way the data is usable for the entire request and you can add logic in the middleware to figure out when to set it.
I would think that having a large chunk of user-specific data would be a red flag as user data should just be a list of what the user can/can't do and what their preferences are.
If this is static data then its super simple. The application cache is what you want. The only complications would be if you have multiple servers that need synced data.
I have read a lot about storing datatable/set in session/viewstate and general consensus seems to be that its not a good idea as it slows down the webpage..but it has its advantages..
Now Iam making a website that allows users to create/manage/host quizzes..and I want to retrieve certain number of questions from database(value of questions will be defined) and store it in a datatable which is maintained in session...Max. no of questions should be 120..
So total data to be stored in session = 120 questions + options + correctanswer; along with other minor things like score of candidate and userdata
My question is: Considering maximum number of questions to be 120, will this much data seriously affect the performance of my page, if so then kindly help me by telling an alternative method...thx.
The view state is certainly not a good place to store this kind of data as the data set will be serialized into a (very long) string and sent to a client with each request. In general, I'd avoid view state whenever possible (in my opinion it has far more disadvantages than advantages).
Storing the data set in session should be fine as long as you use in-process session mode. In this case session data is maintained in memory and the only disadvantage of this approach (as I see it without going into specifics of your task) is that sometimes you might consider releasing the memory that is not needed at the moment. If you're using SQL Server to store session data, I'd choose another storage as serializing/deserializing your data set in this case will probably have an impact on the app performance.
That depends on where you want to put the limit on the application.
Storing the data in memory will limit the application to a certain number of simultaneous users.
Getting the data from the database each time will lower the number of requests that the application can handle per second.
As with any optimisation, you should only optimise when you know what the need is, so I would suggest that you just get the data from the database, and add caching when there actually is a need for it.
I have method in my BLL that interacts with the database and retrieves data based on the defined criteria.
The returned data is a collection of FAQ objects which is defined as follows:
FAQID,
FAQContent,
AnswerContent
I would like to cache the returned data to minimize the DB interaction.
Now, based on the user selected option, I have to return either of the below:
ShowAll: all data.
ShowAnsweredOnly: faqList.Where(Answercontent != null)
ShowUnansweredOnly: faqList.Where(AnswerContent != null)
My Question:
Should I only cache all data returned from DB (e.g. FAQ_ALL) and filter other faqList modes from cache (= interacting with DB just once and filter the data from the cache item for the other two modes)? Or should I have 3 cache items: FAQ_ALL, FAQ_ANSWERED and FAQ_UNANSWERED (=interacting with database for each mode [3 times]) and return the cache item for each mode?
I'd be pleased if anyone tells me about pros/cons of each approach.
Food for thought.
How many records are you caching, how big are the tables?
How much mid-tier resources can be reserved for caching?
How many of each type data exists?
How fast filtering on the client side will be?
How often does the data change?
how often is it changed by the same application instance?
how often is it changed by other applications or server side jobs?
What is your cache invalidation policy?
What happens if you return stale data?
Can you/Should you leverage active cache invalidation, like SqlDependency or LinqToCache?
If the dataset is large then filtering on the client side will be slow and you'll need to cache two separate results (no need for a third if ALL is the union of the other two). If the data changes often then caching will return stale items frequently w/o a proactive cache invalidation in place. Active cache invalidation is achievable in the mid-tier if you control all the updates paths and there is only one mid-tier instance application, but becomes near really hard if one of those prerequisites is not satisfied.
It basically depends how volatile the data is, how much of it there is, and how often it's accessed.
For example, if the answered data didn't change much then you'd be safe caching that for a while; but if the unanswered data changed a lot (and more often) then your caching needs might be different. If this was the case it's unlikely that caching it as one dataset will be the best option.
It's not all bad though - if the discrepancy isn't too huge then you might be ok cachnig the lot.
The other point to think about is how the data is related. If the FAQ items toggle between answered and unanswered then it'd make sense to cache the base data as one - otherwise the items would be split where you wanted it together.
Alternatively, work with the data in-memory and treat the database as an add-on...
What do I mean? Well, typically the user will hit "save" this will invoke code which saves to the DB; when the next user comes along they will invoke a call which gets the data out of the DB. In terms of design the DB is a first class citizen, everything has to go through it before anyone else gets a look in. The alternative is to base the design around data which is held in-memory (by the BLL) and then saved (perhaps asynchronously) to the DB. This removes the DB as a bottleneck but gives you a new set of problems - like what happens if the database connection goes down or the server dies with data only in-memory?
Pros and Cons
Getting all the data in one call might be faster (by making less calls).
Getting all the data at once if it's related makes sense.
Granularity: data that is related and has a similar "cachability" can be cached together, otherwise you might want to keep them in separate cache partitions.
Are there any other ViewState alternatives? I've heard a lot Like Session, holding the state of some of the page's controls' state and getting destroyed the moment the user leaves the page.
I know I'm describing ViewState itself, but I'm looking for a pattern of sorts or suggestions so I can avoid ViewState altogether.
An example of how I'm using it is where I'm storing the contents of my grid (A list of ViewModels) to ViewState. This helps to know which entries are dirty, which ones have been modified, their indexes, the currently selected objects, etc.
One of my colleagues has developed a way to store viewstate data in a file. So that, heavy viewstate data are not transmitted between client and server. Just a key (which is the viewstate data file) that represents the viewstate data file is held as a session variable. In our tests, we've found that saving viewstate in file decreased the server response time by decreasing the amount of viewstate (which was very huge then).
In this article under "Keeping View State on the Server", you can find out how that method can be implemented. You can even store viewstate data in a database table, which gives extra flexibilty if your application is on a web farm.
I don't think you are making a case to move away from ViewState.
If you are holding a large amount of data, you'll face issues when persisting it anywhere else. Session? it'll blow your memory consumption, or if its out of process you'll be moving all of that around each time the Session is loaded/written (once per request). You can of course, try to limit the issue by freeing the stored data as soon as possible / like TempData in asp.net MVC.
You can minimize the amount of info you need to store to check for modified records by introducing a timestamp / or record version. This way you can just check if a new version has been added, and show the user both what they tried to save and what someone else saved.
Another option is compressing your ViewState. It still adds bulk to the round trip, but generally it's minimal.
If you're using .Net 4, there are some useful new ViewState additions:
ASP.NET 4.0: more control on viewstate management
You have Session, and you have Cache.
Session is per user, Cache is global.
Do you really need to store all this in ViewState? why can you on submit (but you're very vague in your question so i'm making a few assumptions here) get all the old data from the DB, compare it with your new data, and update what is changed?
What is preferable, keeping a dataset in session or filling the dataset on each postback?
That would depend on many factors. It is usually best not to keep too many items in session memory if your session is inproc or on a state server because it is less scalable.
If your session resides on the database, it might be better to just requery and repopulate the dataset unless the query is costly to execute.
Don't use the session!!! If the user opens a second tab with a different request the session will be reused and the results will not be the ones that he/she expects. You can use a combination of ViewState and Session but still measure how much you can handle without any sort of caching before you resort to caching.
It depends how heavily you will do that and where your session data is stored on the server.
Keeping the datasets in session surely affects memory usage. All of the session variables are stored in memory, or in SQL server if you use SQL Server state. You can also offload the load to a different machine using the State Server mode.
The serialization/deserialization aspect. If you use insanely large datasets it could influence server seide performance badly.
On the other hand having very large viewstates on pages can be a real performance killer. So I would keep datasets in session or in the cache and look for an "outproc" way to store that session data.
Since I want as few database operations as possible, I will keep the data in session, but I'm not sure what would be the best way to do it. How do I handle concurrency and also since the data is shared when two users simultaneously use the page how can I make sure each of them have their own set of data in the session?
I usually keep it in session if it is not too big and/or the db is far and slow. If the data is big, in general, it's better to reload.
Sometimes I use the Cache, I load from Db the first time and put it in the cache. During postback I check the chache and if it is empty I reload.
So the server manage the cache by itself.
the trouble with the session is that if it's in proc it could disappear which isn't great, if it's state server then you have to serialize and if it's sql sate your're doing a round trip anyway!
if the result set is large do custom paging so that you are only returning a small subset of the total results.
then if you think more than one user will see this result set put each page in the cache as the user pages making sure that the cache is renewed when the data changes or after a while of it not being accessed.
if you don't want to make many round trips and you think you've got the memory then bung the whole dataset in the cache but be careful it doesn't melt the web server.
using the cache means users don't need their own set of data rather they use the global set.
as far as concurrency goes when you load up the insert/ edit page you need to populate it with fresh data and renew the cache after the add/update.
I'm a big believer in decoupling and i rarely, if ever, see the need to throw a full dataset out to the user interface.
You should really only pass objects to the UI which needs to be used so unless you're showing a big diagram or some sort of data structure which needs to display relationships between data it's not worth the cost.
Smaller subsets of data, when applicable, is far more efficient. Is your application actually using all features within a dataset on the UI? if not, then the best way to proceed would be to only pass the data out that you're displaying.
If you're binding data to a control and sorting/paging etc through it, you can implement a lot of the interfaces which enables the dataset to support this, in a lot smaller piece of code.
On that note, i'd keep data, which is largely static (e.g. it doesn't update that often) in the cache. So you need to look at how often the data is updated before you can really make a decision for this.
Lastly, i'll say this again, i see the need to utilise a dataset in the UI very, very rarely..it's a very heavy data object and the cost benefits of looking at your data requirements, versus ease of implementation, could far outweigh the need to cache a dataset.
IMHO, datasets are rather bad to use if you're worried about performance or memory utilisation.
Your answer doesn't hint at the use of the data. Is it reference data? Is the user actively updating it? Is more than one user meant to have update access at any one time?
Without any better information than you provided, go with the accepted wisdom that says keeping large amounts of data in session is a way to guarantee that your application will not scale and will require hefty resources to serve a handful of people.
There's are usually better ways to manage large datasets without resorting to loading all the data in-memory.
Failing that, if your application data needs are truly monsterous, then consider a heavy client with a web service back-end. It may be better suited than making a web page.
As other answers have noted, the answer "depends". However I will assume that you are talking about data that is shared between users. In that case you should not store the dataset in the user's session, but repopulate on postback.
In addition, you should populate a cache near the data access layer of your application to increase interactive performance and reduce load on the database. The design of your cache will again depend on the type of data (read-only vs. read/write vs. read-mostly).
This approach decouples the user session (a UI-layer concept) from data management, and provides the added benefit of supporting shared use of cached data (in cases where data is common across users).
Neither--don't "keep" anything! Display only what a user needs to see and build an efficient paging scheme to see rows backwards or fowards.
rp
Keep in session. Have option to repopulate if you need to, for example if the session memory is wiped.