I have an ASP.NET application that is running on two load balanced servers. Everything is working fine except for one group of customers. All of these customers are coming from the same company. Randomly, an unhandled NullReferenceException error is thrown. It happens at random times in random places. It seems as if the session is just totally gone. since this is only happening for a specific group of users I have to assume that is has something to do with thier environment. I have seen users coming in with IE6, IE7, IE8 and FF and it the error occurs in all cases.
I am not 100% sure how to troubleshoot this. Does anyone have any ideas?
EDIT: Session is set to "InProc"
<sessionState mode="InProc" cookieless="false" timeout="20" />
InProc session isn't shared between servers, so it sounds like this group of users is moving from one server to another and the others aren't. Maybe your load balancer is trying to achieve sticky sessions using something like IP address or whatever and this organisation is blocking that information.
I got in contact with the user that was having the porblem. I asked him to open a browser and go to whatsmyip.org and tell me what it says is his IP address. Then I asked him to refresh the screen a few times. Well, wouldn't you know it, the IP address changed. It kept switching between two different IP addresses. This was not the IP address of his machine but of two different proxies. Each request could come form one or the other apparently.
Our load balancer (something called Zeus - I am not a network guy) was set to estabilsh session affinity (a.k.a. sticky connections) using IP addresses. We changed the settings so that the load balancer would drop a cookie and use that to maintain the session and everything works correctly now.
If you're using SQL to store session state, check that all the servers in the farm are looking at the same SQL database - I've been caught out by this one before and it took quite a while to work it out!
Edit:
Actually you might need to set it to StateServer as you're running in a web farm.
See this about Session-State Modes from MSDN.
If your load balancing is based on directing every hit to the least-busy server, then InProc isn't going to work. you would need to use StateServer or SQLServer modes.
Imagine the first hit from a client is directed to server A - that starts a new session on server A. The second hit from the same client could go to server B, supplying the session cookie from server A which server B doesn't recognise.
If you have 'sticky' (or client affinity) load balancing, where the first hit is allocated to the least-busy server, but then subsequent hits from the same session are directed to the same server, then InProc should still work.
Related
We have recently implemented a new ASP.NET site to our webservers to replace our old Classic ASP site(Both severs are Windows 2008 R2 Using IIS 7.5). They are hosted on a Load Balancer.
This one .NET webform application is used for approximately 30 clients (each with their own URL. client1.mysite.biz, client2.mysite.biz etc...)
Our original plan was deploy our new application into 3 "WebSites" each with their own app pools and BIND the clients to the relevant Website.
When binding we bound to both Http and Https for the URL (we have certificates for each of the sites)
INITIAL PROBLEM:
We noticed that after we bound more than half the sites and tested, we were suddenly being greeted with "Service Unavailable. Service is Temporarily Unavailable" (NO NUMBER just the words) every time. We unbound everything and tried again (meticulously testing each time we bound a site). Each time after binding a certain number of sites the same thing happened.
We ran out of down time and went to Plan B. We put the whole thing in the "Default Website" as a virtual directory (No bindings) (This is how the Classic ASP site was setup)
OUR PROBLEM NOW:
Occasionally we get the same dreaded white screen with "Service Unavailable. Service is Temporarily Unavailable" (NO NUMBER just the words).
It seems to happen randomly (not load or time dependent as far as we can tell). If using AJAX it simply is caught in the "Error" portion of the AJAX code but I believe it is the same problem. The error occurs INSTANTLY when it does happen. If the user attempts to repeat the action that caused the problem everything is fine (they are not logged out and they proceed on their way).
However this is happening MULTIPLE times a day and it's across ALL of our sites (not just this new one).
One more item of great importance. This appears to be happening to ALL of our sites (Virtual Directories and custom WebSites on BOTH of our web servers). That seems to rule out a "bad" server (both are in the cloud did I mention?) and it also "seems" to rule out App Pool settings but what do I know?
About our IIS servers: We have multiple application pools running multiple different instances of websites (different code). Some are testing sites. Some are using classic ASP and others and using ASP.NET.
What we've tried: We scoured the web looking for answers and have edited our machine.config file to increase all manner of things such as "Threads, Max-Connections etc...". We've edited our App Pool settings by increasing our Queue Length and turning on ALL the logs.
Anyone seen anything like this before? My theory is it has something to do with the bindings and the frequency of the error is increased for each binding I initiate but that is difficult to test when it happens on my production servers only.
We have finally solved this problem. As mentioned previously, we noticed that the IIS logs contained a sc-win32-status 64 error when we experienced the Service Unavailable problem in the browser when (and only when) our site was using the Load Balancer.
To help look into this further, we did a network capture of the traffic on the Load Balancer while testing. We reproduced the random Service Unavailable problem, saw the associated win32-status 64 error in the IIS logs, and identified the specific packet of traffic on the network capture for this event.
Using Wireshark, we followed the TCP stream and noticed that the TCP connection was reset by the Load Balancer immediately after this packet. We reproduced the problem three times and every time there was a TCP reset immediately afterwards.
Walking backwards through the TCP stream, we noticed in all three instances a packet for HTTP/1.1 200 (accplication/octet-stream) and prior to that a request to download a document (ie. .pdf or .xlsx or .docx) from one of our sites. The server that contains all our documents is not a web server and does not have the IIS role active. The document server does not have a way to define the content/media type for the document that is being downloaded. Hence the generic (application/octet-stream) packet in the network capture. The Load Balancer treated the request for a document as potentially malicious and decided to reset the TCP connection if another request is made. To fix the problem, we added a content type library function to our application using this post as a guide. Sorted!
In Summary:
A document was requested from our document server via our web
application
The document was sent back to the user with a generic content type =
application/octet-stream
The Load Balancer flagged this activity to be potentially malicious
Another request within this TCP connection was made
The Load Balancer reset the TCP connection
This results in a Service Unavailable
Lesson Learned:
Always define your content/media types if you are serving content from a non web server or a web server running an IIS version less than 7 (Heaven forbid).
A UC Certificate was originally meant for Microsoft Exchange, but it can also be used to cover multiple domains. We use one and it covers about 60+ domains (actually 4 or 5 domains with lots of subdomains). We also apply the certificate to a load balancer and two web servers and we have multiple sites. So far as I can tell the certificates operate as expected. you can view it from any of the 60+ domains. One odd thing about our setup is that in the IIS UI, you can't bind the same certificate to more than one site so we had to use the appcmd command line interface to bind multiple sites to the same certificate.
After looking more closely at our IIS logs it appears that there is indeed something that coincides with this behavior. We get an error of 200 0 64 which is the sc-win32-status 64: "the specified network name is no longer available".
Now our 2 IIS servers are hosted in the cloud on Sungard, and we are using a load balancer that they setup for us. It was our theory that the load balancer was "losing" the proper session id of the user when this 64 error occurs and has no idea where it was supposed to be.
We ran some controlled tests. One group we took OFF the load balancer and sent them directly to one of the servers and another group used the load balancer but made sure to connect to the same server. Both teams conducted the tests of trying to reproduce the error (which is to say we clicked a popup on the site over and over).
The results were interesting. The group that was NOT on the load balancer NEVER received the "Service Unavailable" error! BUT the logs indicated they were getting 64 errors 45 times. The group that WAS on the load balancer was able to produce the "Service Unavailable" message twice and the logs confirmed that there were exactly 2 instances of the 64 error that coincided to the exact moment that the errors were observed.
So what does this mean?
1.) Load balancer has some settings "Sticky Sessions?" that aren't keeping the sessions in right (but we can't find the right settings. It's not even our load balancer it's SunGard's). Anyone have any advice on these settings for ASP.NET?
2.) 64 errors are a part of web life? We gave more cpu power to one of our Virtual IIS servers and received less 64 errors. This is all I can come up with. We've sunk too much time and money trying to solve this, but it appears that I have an option at least of taking people off the load balancer and just routing them to one or the other server and in addition I can at least beef up the server to handle more traffic and reduce the 64 errors.
My webapp is deployed in a cluster of multiple JBoss instances. There is an admin page in the webapp to perform certain Jboss instance-specific operations.
The problem is that requests are sent to a load balancer instead of directly hitting specific individual instance.
Is there any way to direct request to a specific instance? Or at least when the admin page is up, all subsequent requests (Ajax) will stick to the original instance that serves the page at the beginning.
I don't think HttpSession is going to help here. I need to target specific instance and not maintaining the state of individual client.
Thanks.
You were looking for how to configure for Sticky sessions.
Send all requests in a user session consistently to the same backend server known as persistence or stickiness. A significant downside to this technique is its lack of automatic failover: if a backend server goes down, its per-session information becomes inaccessible, and any sessions depending on it are lost. The same problem is usually relevant to central database servers; even if web servers are "stateless" and not "sticky".
Assignment to a particular server might be based on a username, client IP address, or by random assignment. While there are advantages and disadvantages to the approaches.
I would suggest to please go through below article in configuring JBoss under a cluster rather going in deep understanding unless and until you would want to know in deep.
http://docs.jboss.org/jbossas/docs/Clustering_Guide/beta422/html/clustering-http-nodes.html
https://community.jboss.org/wiki/HTTPLoadbalancer
I want to be able to deploy a new version of my asp.net/mvc website without loosing client session state or causing any downtime. The way I'm thinking of accomplishing this is by creating a Windows Network Load Balancing server so that clients can reach it via a single url such as https://mysite.org/. It would then redirect traffic to one of two other sites (A.mysite.org or B.mysite.org). I'll set the NLB's affinity to Single, and disable site B so that all sessions are are directed to site A. When I need to deploy a new version of the website, I'll deploy to site B, enable site B, and disable site A. So, everybody that was on site A can stay there (using version 1) until they log off. All new sessions will connect to site B and run version 2. The next time I deploy, I'll do the reverse.
I've never used NLB. Is this appropriate? Is there a simpler, easier way?
How does NLB know when a request from client X already has a session on A or B? Ie. when they log off the website, and try to login again, will the nlb send them to the same site they were on before?
There are quite a few considerations here
Firstly, rather than juggling the affinity on your NLB, you will probably be better storing your ASP.NET Sessions in StateServer or SQL based Session management to allow web clients (or web service clients) to access your site without 'sticky' affinity. Once you've set up the StateServer or created the SQL Session DB, it should be a simple change to your app's web config.
NLB itself works great for keeping your site up while you upgrade your site. You will typically drainstop a server in the cluster before reinstalling your app to it, test it, and then bring it back into the NLB cluster, before repeating the process with the next server etc.
AFAIK, NLB Single Affinity works at TCP/IP level and is does not interrogate ASP.NET sessions. Basically any connection from the same client IP to the same server IP:Port combination will be directed to the same server. Also AFAIK, both servers will be sharing the NLB IP (In addition to any existing IP's they have).
Since it seems your site uses SSL, it seems that unless you have affinity, that the SSL session keys will need to be renegotiated on each request, which could have performance implications.
I am using internet IE 7, IE 8
My application in running in dns fail over environment with primary and back up server. As the primary server is down, failover changes to the secondary server after 2 - 3 minutes.
But the problem is, the current opened page in IE is still sending requests to the primary server due to dns caching, which stores the IP of the primary server for default 30 minutes) and hanged.
This problem can be solved if we can clear or edit the dns cache with C# ASP.Net coding.
Thanks in advance for replying.
You cannot access a client machine's DNS Cache from your ASP.NET server or anything in the browser. It would be a huge security hole to either of those environments to do so.
The better approach if you're looking for DNS failover is to talk to your network administrator. Ask him/her to set the TTL for your DNS records to a number smaller than your failover time. This will increase the frequency with which the client machines refresh their caches (for your site only) and shorten their downtime in the event of a failover.
The negligible drawback is that it can increase (ever so slightly) their wait time for the site because they have to do DNS requests more often.
There is an undocumented api called DnsFlushResolverCache in dnsapi.dll, see this link for an example for how to use it from C# (not ASP.NET) .
I have the similar issue from JAva applet. JVM has bug which doesn't honor the System property to disable DNS caching. Any workarounds? Changing the java.security file works. But I am looking for a better solution.
ASP.NET Login control is very slow making the initial connection to AD when authenticating to a different domain than the domain the web server is a member of. Problem occurs for the IIS server and when using with the Visual Studio's built in web server.
It takes about 30 seconds the first time when attempting to use the control to connect against another domain. There is no trust relationship bewteen the web server's domain and the other domains (attempted connecting to several different domains). Subsequent connections execute quickly until the connection times out.
Using Systernals Process Monitor to troubleshoot, there are two OpenQuery operations right before the delay to "C:\WINDOWS\asembly\GAC_MSIL\System.DirectoryServices\2.0.0.0_b03f5f7f11d50a3a\Netapi32.dll with a result NAME NOT FOUND" and right after the 30 second delay the TCP Send and TCP Recieves indicate communication begins with the AD server.
Things we have tried:
Impersonating an administrator on the web server in the web.config;
Granting permissions to the CryptoKeys to the NetworkService and ASPNET;
Specifying by IP instead of DNS name;
Multiple variations of specifying the name and ldap server with domains and OU's;
Local host entries;
Looked for ports being blocked (SYN_SENT) with netstat -an.
Nslookup resolves all the domains and systems involved correectly.
TraceRt shows the Correct routes
Any Idea or hints are greately appreicated.
we finaly fixed it with
connectionProtection="None"
To turn off all attmepts to connetc on the secure port and only use 389.
We also had to specify the Connection username like "ad/bob" instead of just "bob" after the connetionProtection change.
Thanks,
Eric-
I had the same issue. Following this advice seems to have sorted it.
https://elgg.leeds.ac.uk/webteam/weblog/15385.html
We had a custom user running as the app pool user, giving the user read access to this directory 'C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys' Sorts the issue.
If you have the .NET 2.0 SP1 installed
just add the following to the
machine.config (of course all sites
will not check verisign any longer)
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
http://forums.asp.net/t/968778.aspx