Apologies if there is an answer already out here but I've looked at over 2 dozen threads and can't find the specific answer.
So, for our ASP.NET (2.0) application, our infrastructure team set up a load balancer machine that has two IIS 7.5 servers.
We have a network file server where the single copy of the application files reside. I know very little about the inner workings of load-balancing and even IIS in general.
My question is regarding sessions. I guess I'm wondering if the 'balancing' part is based on sessions or on individual page requests.
For example, when a user first logs in to the site, he's authenticated (forms), but then while he navigates around from page to page--does IIS 7.5 automatically "lock him in" to the particular server that first logged him in and authenticated him, or could his page requests alternate from one server to the next?
If the requests do indeed alternate, what problems might I face? I've read a bit about duplicating the MachineKey, but we have done nothing in web.config regarding MachineKey--it does not exist there at all.
I will add that we are not experiencing any issues (that we know of anyway) regarding authentication, session objects, etc. - the site is working very well, the question is more academic, and I just want to make sure I'm not missing something that may bite me down the road.
Thanks,
Jim
while he navigates around from page to page--does IIS 7.5 automatically "lock him in" to the particular server that first logged him in and authenticated him
That depends on the configuration of the load balancer and is beyond the scope of a single IIS. Since you haven't provided any information on what actual balancer you use, I can only provide a general information - regardless of the balancer type (hardware, software), it can be configured for so called "sticky sessions". In such mode, you are guaranteed that once a browser establishes connection to your cluster, it will always hit the same server. There are two example techniques - in first, the balancer just creates a virtual mapping from source IP addresses to cluster node numbers (which means that multiple requests from the same IP hit the same server), in second - the balancer attaches an additional HTTP cookie/header that allows it to recognize the same client and direct it to the same node.
Note that the term "session" has nothing to do with the server side "session" where you have a per-user container. Session here means "client side session", a single browser on a single operating system and a series of request-replies from it to your server.
If the requests do indeed alternate, what problems might I face
Multiple issues. First, encryption, if relies on machine key, will not work. This means that even forms cookies would be rejected by cluster nodes other than the one that issued the cookie. A solution is to have the same machine key on all nodes.
Another common issue would be the inproc session provider - any data stored in the memory of one application server will not "magically" appear on other cluster nodes, thus, making the session data unavailable. A solution is to configure the session to be stored in a separate process, for example in a sql server database.
I will add that we are not experiencing any issues (that we know of anyway) regarding authentication, session objects
Sounds like a positive coincidence or the infrastructure team has already configured sticky sessions. The latter sounds possible, the configuration is usually obvious and easy.
Related
Since we have moved to azure, we have numerous session lost issues only on production.
We have InProc, cookie based, sticky session, large timeout, no high traffic and no high memory/process usage.
We use HAProxy as loadbalancer.
I have done basic research and none of the following seems to be the cause:
session timeout
application pool settings/recycling
memory size and usage thresholds
no eaten exceptions
there is no changes to file system to cause a restart
I'm particularly more suspicious about how loadbalancer/ssl and application work together and if http headers are fine, but I don't know any tools to really monitor that.
I'm assigned to find a solution at the same time I have no privilege to access the machines.
Logs(Log4Net) are all stored in database but doesn't help to give a clear understanding of what is going on the system and cannot follow a user session using them.
I'm allowed to find the problem by adding required logs to code or to develop some kind of monitoring module or to use profiling/debugging tools.
Only once a month there will be a production deployment so I'm trying to use the opportunity as best as possible.
Question:
Is there any useful monitoring/profiling tool that can give me a clear view of what is happening in the system by aggregating information I may need? for example following a user/session between requests from time of login until session drop plus information about headers and other system application parameters.
if there is not such a tool out there, please give me your ideas to write one?
This is a common issue in load balanced environment. As mentioned in this answer for a similar question,
InProc mode, which stores session state in memory on the Web server. Which means that session data is maintained inside your web server on a given VM and is not shared outside of the VM. So when you have multiple server for load balancing, the session state isn't shared with each other. To solve this, you must store your session state external to the web server.
Use Redis, or SQL Database, or something else.
I have a web application that appears to be randomly dropping ASP.NET sessions. This application is hosted across four servers along with a load balancer. I have looked at the Session State settings in IIS7 for the site's virtual directory. Each server has this set to "In Process". From some reading I have done, it looks like this means the session data is only held on a single server, and if the load balancer switches the server on that user, the session will get dropped. Is this correct?
Articles that I have read also state that if using "In Process" when hosting on multiple servers, you should be using something called "Sticky Sessions", but don't go much into detail about it. How I can check to see if "Sticky Sessions" are being used? Is this an IIS7 setting, something unique to the load balancer, or something totally different?
This is just my theory, so I am also wondering if there are some other indicators I should look into before I can draw the conclusion or eliminate the possibility that the load balancer/IIS settings are the issue.
I've checked into the Event Viewer and I don't think this is happening due to any recycling of AppPools or IIS itself.
Yes, you are right, when another server is hit in one of next requests, there is no data.
Sticky sessions are handled by the balancer. It could stick clients by their ip or issue a stick cookie. Depends on the actual balancer and now it is configured. In NLB this was called "session affinity".
Another common workaround would be to configure your session to a sql server mode but sounds like you are aware of this possibility and you don't want to go that way.
Here is the scenario:
We have 3 web servers A, B, C.
We want to release a new version of the application without taking the application down
(e.g. not using the "Down for maintenance page").
Server A goes live with latest code.
Server B gets taken off-line. Users on Server B get routed to A and C.
Page1.aspx was updated with new control. Anyone that came from Server B to Server A while
on this page will get a viewstate error when they perform an action on this page. This is what we want to prevent.
How do some of you resolve this issue?
Here are some thoughts we had (whether it's possible or not using our load balancer, I don't know... I am not familiar with load balancer configuration [it's an F5]):
The more naive approach:
Take down servers A and B and update. C retains the old code. All traffic will be directed to C, and that's ok since it's the old code. When A and B go live with the update, if possible tell the load balancer to only keep people with active sessions on C and all new sessions get initiated on A and B. The problem with this approach is that in theory sessions can stick around for a long time if the user keeps using the application.
The less naive approach:
Similar to the naive approach, except (if possible) we tell the load balancer about "safe" pages, which are pages that were not changed. When the user eventually ends up on a "safe" page, he or she gets routed to server A or B. In theory the user may never land on one of these pages, but this approach is a little less risky (but requires more work).
I assume that your load balancer is directing individual users back to the same server in the web farm during normal operations, which is why you do not normally experience this issue, but only when you start redirecting users between servers.
If that assumption is correct then it is likely the issue is a inconsistent machinekey across the server farm.
ViewState is hashed against the machine key of the server to prevent tampering by the user on the client side. The machine key is generated automatically by IIS, and will change every time the server restarts or is reset, as well as being unique to each server.
In order to ensure that you don't hit viewstate validation issues when users move between servers there are two possible courses of action.
Disable the anti-tampering protection on the individual page or globally in the pages element of the web.config file using the enableViewStateMac attribute with a false value. I mention this purely for the sake of completeness - you should never do this on a production website.
Manually generate a machine key and share that same value across each application (you could use the same key for all your applications, but it is sensible to use one key per application to maximise security), on each of your servers. To do this you need to generate keys (do not use any you may see in demos on the internet, this defeats the purpose of the unique machine key), this can be done programatically or in IIS manager (see http://www.codeproject.com/Articles/221889/How-to-Generate-Machine-Key-in-IIS7). Use the same machine key when deploying the website to all of your servers.
I can't answer on the best practice for upgrading applications that require 100% uptime.
I was wondering whether it would be possible to change the sqlConnectionString used for SessionState in ASP.net based upon the domain an application is running on?
A scenario; We have 20 sites running from one application all talking to different databases depending which domain (site) they are browsing from.
When browsing www.domain1.com the application talks to the database 'db1'. The site www.domain2.com on the other hand talks to the database 'db2' etc, thus selecting the relevant content and also spreading the load to each database rather than using one master database to handle all connections for the sites.
An issue that has arisen though - for this setup we use SqlServer mode for the SessionState so all users to all sites sessions are stored in 1 aspstate database, now as the sites get busier / number of sites increase this database comes under increasing strain to handle all the session requests for all the sites and we are starting to get some timeout errors where the connections to this database are bottlenecking.
We can seperate out the sites to from their own application and set up different applications with the same code but within each application set a different Session database in each Web.Config and thus lightening the load. This task would be quite time consuming though and would result in more management in the long term. SO.. I would love to know if it's possible to modify within the code the sqlConnectionString used for SessionState, based upon a domain, before the session object is created? Can we inherit from System.Web.HttpApplication and use the Application_AcquireRequestState event to create the required setup of the HttpSessionState object?
Hopefully this makes sense and that someone can provide some pointers and prove to me that this isn't a pipe dream!
Cheers,
Steve
I think you are missing a big point--putting things in separate databases on the same server isn't going to help things at all if the bottleneck is sql server--it is either SQL running out of headroom or the network running out of bandwidth. I'd try and figure out which one it was before doing anything.
Your issue isn't so much that the connections to the database are bottlenecking, its that you are overwhelming the network connection to the database with data from all of the sessions.
By default, the Sql Server state provider simply serializes your data and ships it to the database. This is VERY inefficient and takes a LONG time to transfer on a fast network.
We solved this problem by going to a custom provider, like DOTSS that compresses session content before shipping it to the database. The compression rates we see are 80%-90% and the compression time is less than 10ms.
You can implement a custom session state provider. See MSDN for details. I've never done it, but with a little luck you can wrap the SqlServer session state module and redirect it based on the domain
First of all, I don't see there is advantage of "I would love to know if it's possible to modify within the code the sqlConnectionString used for SessionState, based upon a domain, before the session object is created" compared to set this in web.config.
Secondly, I think you need change that connection string setting in App_Start, so all the request will use that changed settings.Application_AcquireRequestState probably too late for this.
Why not split up the sites into sperate web applications and use hostheader to differentiate between the web sites. That way you could easily configure which session database you want your web application to use since each web application would have a seperate web.config file.
You could partition your session across different databases by implementing IPartitionResolver, and using a different partition for each domain.
Here's an example showing how to implement a custom partition resolver. (The example partitions by session ID, but it would be trivial to change it to partition by domain instead.)
We have several dozen development sites whose database connections are handled via the project's main Web.Config.
There is a separate configuration section corresponding to each URL on our intranet (e.g. http://development11, http://development12). We have SQL instances with a similar naming convention (DEVDB1\SQL1, DEVDB1\SQL2).
Based on the URL configured on the intranet IIS server, the app grabs the appropriate config. For testing we can easily modify the user, the database server or individual databases utilized for a particular site.
I am looking for information on how to create an ASP.NET web farm - that is, how to make an ASP.NET application (initially designed to work on a single web server) work on 2, 3, 10, etc. servers?
We created a web application which works fine when, say, there are 500 users at the same time. But now we need to make it work for 10 000 users (working with the web app at the same time).
So we need to set up 20 web servers and make something so that 10 000 users could work with the web app by typing "www.MyWebApp.ru" in their web browsers, though their requests would be handled by 20 web-servers, without their knowing that.
1) Is there special standard software to create an ASP.NET web farm?
2) Or should we create a web farm ourselves, by transferring requests between different web servers manually (using ASP.NET / C#)?
I found very little information on ASP.NET web farms and scalability on the web: in most cases, articles on scalability tell how to optimize and ASP.NET app and make it run faster. But I found no example of a "Hello world"-like ASP.NET web app running on 2 web servers.
Would be great if someone could post a link to an article or, better, tell about one's own experience in ASP.NET "web farming" and addressing scalability issues.
Thank you,
Mikhail.
1) Is there special standard software
to create an ASP.NET web farm?
No.
2) Or should we create a web farm
ourselves, by transferring requests
between different web servers manually
(using ASP.NET / C#)?
No.
To build a web farm, you will need some form of load balancing. For up to 8 servers or so, you can use Network Load Balancing (NLB), which is built in to Windows. For more than 8 servers, you should use a hardware load balancer.
However, load balancing is really just the tip of the iceberg. There are many other issues that you should address, including things like:
State management (cookies, ViewState, session state, etc)
Caching and cache invalidation
Database loading (managing round-trips, partitioning, disk subsystem, etc)
Application pool management (WSRM, pool resets, partitioning)
Deployment
Monitoring
In case it might be helpful, I cover many of these issues in my book: Ultra-Fast ASP.NET: Build Ultra-Fast and Ultra-Scalable web sites using ASP.NET and SQL Server.
I'd say you should configure an NLB cluster (Network Load Balancing), which basically splits all requests between cluster nodes (And as an added benefit detects if things are down and stops sending them requests). There's features built into windows for this, but they don't compare to a hardware device for performance or scalability. If you're using Windows 2008 it really is simple to set one up. If you do this make sure you have a shared machine key or you'll start getting exceptions for viewstate being invalid (When 1 server submits the form and it posts to the other and they're using different keys to encode the data).
You can also use DNS round-robin but at 20 servers presumably in 1 datacenter I wouldn't see a point to going to such crazy lengths. If you've got multiple data centers though this is definitely worth considering (As NLB won't really work well between data centers).
You'll also want to be sure if a user swaps servers they don't loose their session. The simplest way would be to use a Session State database (Configurable in the web.config, or you can do it server-wide in IIS's configs). If you don't use sessions though just turn them off in the Pages directive of the web.config and call it a day. You could also use a session state server, but I don't have any experience with this.
It may also be worth considering spending some time optimizing the code or adding caching directives to static content - it can be very cost-effective even if you only trim the need for a few of those servers.
Hope that helps.
If you keep your server stateless, it is easy with a good router that implements some round-Robbin protocol (that send each call to the single published server ip to a different web server).
if it is not stateless (like - if a login is required, or ssl) than you need to keep each session to the same server.
Here is some info about MS Application Request Routing - you will get everything there:
IIS Load balancing
I would not recommend #2. You will do much better off with a load balancer.
Pay attention to session state management. Unless you configure the load balancer to keep each user on the same web server, you will have to use the session state server or database.
Also, check your code's usage of Application and Cache variables. These will be different on every web server. If those values are static, you may not have a problem. But if they can change, you can end up with different values on each web server.
There used to be a problem with ViewState in 1.x, as explained here. I'm not sure if this problem still exists.
Then, there are some changes that you need to make to the Machine Key in web.config, as explained here.