First of all to give you a bit of background on the current environment. We have a number of ASP.NET applications, all of which use session for certain aspects. We are "Load Balanced" over multiple servers due to traffic levels, however, our load balancing is set to use "Sticky Sessions" as currently all web applications are set to use "InProc" for session state.
We are looking at being able to remove the "Sticky Sessions" configuration on our load balancer, as due to our traffic loads servers can and do get overloaded. We want to go with a more balanced approach, but must be able to use session.
I know that SqlServer for session state will work, but for reasons beyond our control, we cannot use SqlServer to store our state. In researching it seems that StateServer is our best bet. We have an additional server, with loads of memory sitting around. This server could be our StateServer for the entire Web Cluster. We just want to know the following things.
1.) Besides any potential serialization issues with the switch from InProc to StateServer, are there any major known issues with losing session objects or generating errors with the above listed environment?
2.) Aside from the single point of failure, and slighly slower performance are there any other gotchas that we need to be aware of with using StateServer.
3.) Are there any metrics that show the performance differences between the three types of state storage?
Here is a decent FAQ on asp.net state: http://www.eggheadcafe.com/articles/20021016.asp
From that Article, here is some information on StateServer:
In a web farm, make sure you have the same MachineKey in all your web servers. See KB 313091 on how to do it.
Also, make sure your objects are serializable. See KB 312112 for details.
For session state to be maintained across different web servers in the web farm, the Application Path of the website (For example \LM\W3SVC\2) in the IIS Metabase should be identical in all the web servers in the web farm. See KB 325056 for details
I have only used sql and in-proc. But these 3 that apply when using sql server apply as well:
Avoid storing too much information in the session, as it affects both in serialization and data transmitted over the network.
Make sure you don't have anything that depends on the Session_onEnd. This is just not available for out of process sessions.
Turn off session on pages that doesn't uses it. This don't make a difference for in-process session, but for out of process it will save you a lot.
Make sure your server etag ids are synchronized across the web farm otherwise caching at client browsers will be upset.
Have you reviewed your code in detail to make sure everything can be serialized out of process and across a LAN efficiently?
Are you solving the main performance problem within your system? I ask because the database is the typical source of contention.
My main motivation for moving away from sticky sessions was operational flexibility i.e. cycle down a problematic server or to deploy a software upgrade. So having implemented a central session state service make sure you take full advantage from an operational stand point.
In my experience we've found out that native state server or even using SQL Server for sessions is a very scary scenario as both have issues (mainly performance). By the way, we are also using sticky sessions.
I think you can explore other products for this to achive the absolute best. A free option would be Velocity but it is still not released.
And another comprehensive but proven product will be (Very expensive actually) NCache. THis will even help in your serilizations with less cost, If you use their API's it will be even better results.
Take a look and see which looks best for you.
About SQL Server, you server will die very soon if you have enough number of hits coming in (I belive you have some hits already which yielded you to do Web Farm or you do it just for the sake of redundancy)
Bottom line: We are evaluating Velocity because NCAchce is really expensive. However advantages are huge.
We are using StateServer for a very small web farm with only two nodes for a few hundred users.
I'm not responsible for its operation but I remember only two issues in two years where the service had to be restarted because it crashed.
I would like to another one more point to the accepted answer:
Make sure the version of framework dlls is the same.
In my case the System.Web dll versions were different as a few windows updates were skipped on one of the servers of the farm.
Related
Since we have moved to azure, we have numerous session lost issues only on production.
We have InProc, cookie based, sticky session, large timeout, no high traffic and no high memory/process usage.
We use HAProxy as loadbalancer.
I have done basic research and none of the following seems to be the cause:
session timeout
application pool settings/recycling
memory size and usage thresholds
no eaten exceptions
there is no changes to file system to cause a restart
I'm particularly more suspicious about how loadbalancer/ssl and application work together and if http headers are fine, but I don't know any tools to really monitor that.
I'm assigned to find a solution at the same time I have no privilege to access the machines.
Logs(Log4Net) are all stored in database but doesn't help to give a clear understanding of what is going on the system and cannot follow a user session using them.
I'm allowed to find the problem by adding required logs to code or to develop some kind of monitoring module or to use profiling/debugging tools.
Only once a month there will be a production deployment so I'm trying to use the opportunity as best as possible.
Question:
Is there any useful monitoring/profiling tool that can give me a clear view of what is happening in the system by aggregating information I may need? for example following a user/session between requests from time of login until session drop plus information about headers and other system application parameters.
if there is not such a tool out there, please give me your ideas to write one?
This is a common issue in load balanced environment. As mentioned in this answer for a similar question,
InProc mode, which stores session state in memory on the Web server. Which means that session data is maintained inside your web server on a given VM and is not shared outside of the VM. So when you have multiple server for load balancing, the session state isn't shared with each other. To solve this, you must store your session state external to the web server.
Use Redis, or SQL Database, or something else.
I am just getting in to the more intricate parts of web development. This may not be in the best place. However, when is it best to get load balancing for a web project? I understand that it depends on good design/bad design as to how many users you can get to visit a site without it REALLY effecting the performance. However, I am planning to code a new project that could potentially have a lot of users and I wondered if I should be thinking off the bat about load balancing. Opinions welcome; thanks in advance!
I should not also that the project most likely will be asp.net (webforms or mvc not yet decided) with backend of mongodb or pgsql(again still deciding).
Load balancing can also be a form of high availability. What if your web server goes down? It can take a long time to replace it.
Generally, when you need to think about throughput you are already rich because you have an enormous amount of users.
Stackoverflow is serving 10m unique users a month with a few servers (6 or so). Think about how many requests per day you had if you were constantly generating 10 HTTP responses per second for 8 hot hours: 10*3600*8=288000 page impressions per day. You won't have that many users soon.
And if you do, you optimize your app to 20 requests per second and CPU core which means you get 80 requests per second on a commodity server. That is a lot.
Adding a load balancer later is usually easy. LBs can tag each user with a cookie so they get pinned to one particular target. You app will not notice the difference. Usually.
Is this for an e-commerce site? If so, then the real question to ask is "for every hour that the site is down, how much money are you losing?" If that number is substantial, then I would make load balancing a priority.
One of the more-important architecture decisions that I have seen affect this, is the use of session variables. You need to be able to provide a seamless experience if your user ends-up on different servers during their visit. Session variables won't transfer from server to server, so I would avoid using them.
I support a solution like this at work. We run four (used to be eight) .NET e-commerce websites on three Windows 2k8 servers (backed by two primary/secondary SQL Server 2008 databases), taking somewhere around 1300 (combined) orders per day. Each site is load-balanced, and kept "in the farm" by a keep-alive. The nice thing about this, is that we can take one server down for maintenance without the users really noticing anything. When we bring it back, we re-enable our replication service and our changes get pushed out to the other two servers fairly quickly.
So yes, I would recommend giving a solution like that some thought.
The parameters here that may affect the one the other and slow down the performance are.
Bandwidth
Processing
Synchronize
Have to do with how many user you have, together with the media you won to serve.
So if you have to serve a lot of video/files to deliver, you need many servers to deliver it. Let say that you do not have, what is the next think that need to check, the users and the processing.
From my experience what is slow down the processing is the locking of the session. So one big step to speed up the processing is to make a total custom session handling and your page will no lock the one the other and you can handle with out issue too many users.
Now for next step let say that you have a database that keep all the data, to gain from a load balance and many computers the trick is to make local cache of what you going to show.
So the idea is to actually avoid too much locking that make the users wait the one the other, and the second idea is to have a local cache on each different computer that is made dynamic from the main database data.
ref:
Web app blocked while processing another web app on sharing same session
Replacing ASP.Net's session entirely
call aspx page to return an image randomly slow
Always online
One more parameter is that you can make a solution that can handle the case of one server for all, and all for one :) style, where you can actually use more servers for backup reason. So if one server go off for any reason (eg for update and restart), the the rest can still work and serve.
As you said, it depends if/when load balancing should be introduced. It depends on performance and how many users you want to serve. LB also improves reliability of your app - it will not stop when one system goes crashing down. If you can see your project growing to be really big and serve lots of users I would sugest to design your application to be able to be upgraded to LB, so do not do anything non-standard. Try to steer away of home-made solutions and always follow good practice. If later on you really need LB it should not be required to change your app.
UPDATE
You may need to think ahead but not at a cost of complicating your application too much. Do not go paranoid and prepare everything to work lightning fast 'just in case'. For example, do not worry about sessions - session management can be easily moved to SQL Server at any time and this is the way to go with LB. Caching will also help if you hit some bottlenecks in the future but you do not need to implement it straight away - good design (stable interfaces), separation and decoupling will allow for the cache to be added later on. So again - stick to good practices, do not close doors but also do not open all of them straight away.
You may find this article interesting.
I am moving a site from a single server to a server farm consisting of three web servers behind a load balancer. It seems easy enough to handle session management - just make sessions "sticky" at the Load Balancer (we evaluated SQL-based session management but have decided to continue using InProc session management for efficiency).
However, we also use a sizable configuration object that we keep in the Application space (e.g. Application[ObjName]). Since the config object is loaded from memory, we have no problem until someone makes a change to the configuration. At that point, the application on the hosting server will have the change and the database will have the change. However, the other two servers won't have the change. We've debated having a "once a minute" polling rule (e.g. on new sessions), keeping information in the session instead (not very efficient), etc. All have serious drawbacks. I am wondering what other people do. Is it possible to keep the Application space on SQL Server but the Session space inproc? Any help or insight about how to handle this would be appreciated!
Application[] is always going to be local memory based, so no matter what you've going to have some code changes to make. So put it somewhere else, like a distributed cache, AppFabric, NCache, memcached.net etc. When someone makes a change to the configuration update the cache, when you need to read the settings read from the cache. Propogation/Sync is taken care of by the cache itself.
We currently decided to use NCache as we have got 4 web servers for our web farms. This third party caching tool can work perfectly with load balancer and is easy to configure(just the Express version of it is free. For Professional and Enterprise version, only Developer Machine is out of charge). It is also really fast and stable. You must setup the NChache on each server and set the load balancer to work with all of them. Hope it helps.
I keep hearing that it's bad practise to store large object collections / anything in the session. Often during conversation it's quickly followed by: 'Just turn sessions off'
So what is the general problem with sessions? I use them a fair bit and since they 'real' session is stored behind a strongly typed container I don't really see the issue.
There is nothing wrong with session - you just need to be mindful of its limitations. To say "just turn off session" is throwing the baby out with the bathwater.
There is a huge difference between storing BIG objects and small objects in a session
The session will stay alive on a server untill it expiers, and that means those big objects pollute your available memory. If you do that with a server under load, or a server that runs many application pools, then this can cause trouble.
You dont need cookies to have a session, since ASP cal also encode that information in the urls. Also you can configure the session store to run out of process, or even to store the information inside a SQL Server (reducing the memory load on the server, and enabeling sessions across a farm)
So basically: Objects are ok - Big objects not
Here's my take -- sessions are not bad but sometimes they are overused. It can also be harder to understand a web application's flow when it relies on a lot of sessions so of course you should be careful not to get carried away.
However, you should feel free to use them anytime you need to store temporary data to be made accessible across multiple pages. In no other situation should they be used. But that situation is one for which sessions were specifically designed.
Now, if you're worried about memory consumption on the server, that's not necessarily a reason to avoid sessions. But it may be more of a reason to avoid the InProc session provider. In fact I'm not a fan of InProc sessions as they tend to expire prematurely after a certain number of recompiles in your application.
What I actually prefer and nearly always use are SQL Server sessions. They'll be slightly slower, but the benefits are numerous. They'll persist even if the server is rebooted and that makes them a very reliable choice. And of course since they're stored in the SQL file system instead of in memory, they won't make such a big hit on memory.
This article on MSDN talks about the various session providers and also explains how to configure SQL to handle your sessions. If you don't have SQL, just know that even the free SQL Server Express 2008 can be configured as your session provider.
I had thought that it largely depends on the traffic to your web site. If you are running something like amazon.com, trying to store the user's shopping cart in a session would take huge amounts of IIS allocated memory, bringing down your web server. For smaller web sites, session variables are fine to use in moderation.
Storing large objects in Session is bad, yes, but "large" is relative.
Basically, storing an object in session will keep it in memory until the session expires, so if you have a site with a high user count all storing mega-objects in their session, you'll kill your server pretty quickly.
With that being said, an argument could be made for the idea that if you have objects that are 5k+ in memory and have enough users to actually cap out a server then you can probably afford more hardware anyway.
There are also topics like server clustering and session integrity between boxes in the cluster. Some frameworks handle this, I don't know if .NET does or not.
There are two things to be careful of:
Memory consumption: if you store large data objects in session and you have many user you may well run out of memory or at the very least triggering many early recycling of your application
This is only a problem if you have multiple web servers (web farm): the session has to be stored externally (not in process) in a SQL server or a windows service so that it is accessible from different machines. This can be quite slow at times.
Session requires the user to have cookies turned on
If you're working in a web farm, you'll run into trouble.
I guess these reasons don't have anything to do with storing large objects in session, just in using sessions at all.
2 major issues come to mind...
1) Persistence of sessions across servers when you start scaling your website
2) Memory usage explosion from storing UI objects in session state
The more serious issue is the tendency to store objects in session. When you store something as innocuous as a Label from a page on your page, you get LOTS of unwanted object attributes as well. You probably just wanted the text of that label stored in your session, but along with it, you get references to the page itself...and all of a sudden, you have a massive usage of memory to store the page, its view state, and lots of unwanted attributes in memory on your server.
Check out this link about storing UI elements in session
You may want to check out this question as well.
This is an old thread although.
But I have an experience for a session problem. I would like to share it.
There is a simple flow.
One .aspx validate a client, and read a bill-html from a file (for this client), then save this html(about 2MB) in a session variable.
This .aspx will auto redirect to next .aspx, the next .aspx retrieves this html from session. Then show it to the client.
It works fine in most cases. But some clients encountered a problem: The bill he saw is not his bill, but others.
We used sniffers tools to intercept the network package.
And we saw a strange situation:
Our IIS has definitely sent the SessionID(eg: 1111111) to the client, But when the client redirects to next page and tries to access session. The SessionID(eg: 11112222) that this client brings is different.
We think that the browser of that client does not accept the SessionID.
And finally, we abandon the use of Session, and solved this problem.
I have an ASP .NET 2.0 web application which is a online survey system. At times it has huge number of users and the application go slow.
I wanted to do load balancing for the web application which runs in a single server now.
Will anyone suggest me...
In what all scenarios i should consider load balancing to my application?
what type of applications need load balancing?
what is the pros and cons of load balancing?
what is the guidelines for devoloping applications which targets load balancing,
At max how many number of concurrent users can access web application without load balancing without affecting performance much?
In my case application is already devoloped. What all the areas i should make changes to prepare it for load balancing?
Thanks in advance.
First you need to ensure that you know where the bottleneck to performance is.
If you can focus first on getting the total round trip time for each user's page load down then you will be able to handle more users.
In a case where you are bottlenecked on database calls you could setup more servers for load balancing your web application and get very little benefit.
Is there a bandwidth issue with your webserver? Are requests for images, css and javascript files slowing down just as much as web application page_load requests?
Ensure that you aren't storing too much data in session state. If you are storing lists or other large objects in memory you have to remember that you will be multiplying that memory usage by the number of users causing things to get out of control pretty quickly if you have 10,000 users with an active session. In some of these cases it may be preferable to move state information out to cookies that are stored by the user.
From my experience, the software load balancing options are limited, inconsistent and inflexible.
Of course, when developing the application, you need to ensure that your application can scale out for a web farm scenario - i.e, things like ensuring you are using a distributed cache provider, a state server etc.
The hardware based solutions will provide multiple methods of distributing loads and are very consistent. Consider options like using NLB/F5 Big-IP load balancers.
Have a look at this post from Scott as well - Old, but useful. http://www.hanselman.com/blog/LoadBalancingAndASPNET.aspx
Thanks, Anoop
In what all scenarios i should consider load balancing to my application?
In any scenario when your current server can no longer handle the load by itself. There are two kinds of scaling, vertical (buy a better server) or horizontal (add more servers). Load balancing is a form of horizontal scaling, and typically gives you more bang for your buck, but is also much harder to set up.
What type of applications need load balancing?
Pretty much the same answer here as with a.
What is the pros and cons of load balancing?
Pro is that it lets you scale. Cons are that it introduces unique issues.
What is the guidelines for devoloping applications which targets load balancing?
The big one is session. Session state by default is stored in proc, which means that it only exists on the one server. If the user has the potential of being bounced across different servers, they will lose their session. I would recommend reading up on it here (the way you probably want to go is either dont use session, or use state server)
At max how many number of concurrent users can access web application without load balncing without affecting performance much?
Completely depends on your server hardware and your application requirements. I believe there are 5 asp worker processes per core, so that is one consideration right off the bat. There is also RAM usage to consider.