I am seeing a very strange problem on one of my production boxes. We have an application hosted in IIS 6 on a single machine with an apache web server in front of it. My application is using ASP.NET Membership for authentication and relies on session state. I am seeing a problem right now where some server requests will hit a null exception when trying to access any session variables, but if the request is retried, the page hits no exceptions and behaves properly.
I believe this has something to do with the Session ID cookie either getting corrupted or lost on the request, but I have no idea what could cause that. The reason I believe this is because it seems like ASP.NET is not seeing the cookie and creating a new session, which would explain why the variables are null. When it gets the exception, it cannot write the new SessionID cookie back to the client, so the client still maintains the original SessionID. Then when the next request is sent, the original SessionID cookie is used, which now ASP.NET finds and is able to retrieve the session state. This is pure speculation, but seems to fit the symptoms.
Also this site is using no other cookies other then the ones required by ASP.NET Membership and ASP.NET Session, so I am well under the cookie limit for IE. The site has been functioning properly for about 8 months and this issue just popped up recently. I have tried IIS resets and actually rebooting the machine, but nothing has seemed to help the problem.
Updates:
Here are some clarifications that were asked for.
1.)Our Apache server is the only thing exposed to the internet. All requests occur over HTTPS to this box. The Apache box then forwards all request past over HTTP to our application server. This is being done for security reasons. We have looked to see if Apache might have been the issue, but there seems to be no error in the Apache logs.
2.)The null exception is occurring when trying to access an object stored in session that the application expects to be there as opposed to the exception happening with the actual session object itself.
We have found the root of the issue. It looks like the IIS meta-base got corrupted on our app server. The best way to fix this issue is to do a clean install of IIS, but because of business constraints, this is not an option for us. So another solution is to actually just create a new App Pool for the application to run under. According to some people with more IIS expertise than myself, this will fix the problem in the short term, but it is very likely that the same thing will happen to this App Pool. So we are required to create new App Pools if this starts occurring again.
Related
From time to time, some requests on my website starts to hang on the RequestAcquireState state of the Session module. When that spiral begins all requests timeout and we need to restart the IIS on the affected server.
I investigated it a lot and the only conclusion I got is that somehow a deadlock is happening while the application tries to access user data stored in Session.
The only option I can think of to fix this issue is to either reduce or stop using Sessions in my application. This is definetely part of the plan, but it will take a while before we can complete that.
We run 6 machines with IIS 7.5, out of proc StateServer and server affinity on in our Load Balance.
Any hints on how to workaround this issue or fix it at all without having to remove Sessions entirely?
Lock mechanism exist on both provider and session module (IIS Session Module). You can develop custom session module, but you still need provider without locking or You can develop custom provider without locking but you still need IIS session module and it is not so simple to implement at that level.
The Solution is UnlockedStateProvider [aka Unlocked]
Follow the white rabbit :P (Check the demo project, it explains everything.)
The answer is Hotfix Rollup 2828841 for .NET Framework 4.5 , here all the explanation:
http://forums.asp.net/t/1888889.aspx/2/10?Question+regarding+a+possible+bug+within+NET+4+5
and here the download link
It works for me on IIS 7.5 Windows Server 2008 rs x64 , asp.net web forms application with lot of ajax request.
I just found out today that if you have a long running request (or in my case, an infinite loop), then all subsequent requests will be locked, because by default ASP.NET locks on session.
So if you have users with requests in RequestAcquireState, then check if there is a request in ExecuteRequestHandler that is locking the session, and thus preventing other requests from starting.
There is a discussion here on how to prevent locking on session.
(Basically, create most of your pages as Session-Read-Only, and modify session as rarely as you can.)
Is it possible those users have another long running request and the requests you see piling up are actually secondary requests? By default, ASP.NET will lock Session until a request is complete. If a second request comes in before the first one is complete, it will have to wait. If you are using MVC, you can change this behavior by adding an attribute to your controller.
[SessionState(SessionStateBehavior.ReadOnly)]
This makes Session read-only, removing the locking behavior allowing subsequent requests to be processed.
I am currently debugging some issue about this.
We have a ASP.NET web application and I am debugging on Cassini. When I tried to use IE and send out the request to the server, some time (e.g. in about 20minutes) is needed to process and then send out the response.
In case of multi-tab IE, I tried to send out the requests in different tab at about the same time to the same server but the response is handled only after the one of the response is sent out.
If a new instance of IE is started and the requests are sent out in these different instances, the server can process and send out the response almost simultaneously. After doing some research I found that IIS express may solve my problem, but I cannot. Anyone has experienced similar problem or have I missed out some really important things to check with first?
Thank you for your help.
This is primarily due to ASP.net's session state variable and the fact that only one request at a time may have R/W access to a particular session (as determined by the SessionID cookie).
Any additional requests requiring any form of session access (since Read/Write is the default) will be blocked until the previous request has been completed.
Based on the following links:
http://johnculviner.com/asp-net-concurrent-ajax-requests-and-session-state-blocking/
https://msdn.microsoft.com/en-us/library/ms178581.aspx?f=255&MSPPError=-2147217396
I think that you miss the point that the session is lock all request leaving only one per time to run.
Read about that and why:
Replacing ASP.Net's session entirely
Also : Web app blocked while processing another web app on sharing same session
The reason is that Sessions in ASP.NET are not thread safe. Therefore ASP.NET serializes access to requests from the same session.
If you have a multi-tab IE then your tabs share one session. The first request is executed right off and the other ones are queued. If you have different instances then each of them creates a new session and therefore the request are executed in parallel.
Ran into this issue yesterday on one of our sites. First of all the site is hosted in a web farm environment and for the time being I have added a static machineKey to the web.config on both nodes (2 node environment). This has solved the issue and everything is running fine now.
This raised the following question:
Why is it that all our other sites that run on this environment does not require this (machineKey in the web.config).
I checked event logs to make sure that we are not having the same issue on other sites and everything looks fine. I also confirmed that the app pool is not recycling too often and everything was normal with regards to app pool settings.
The only explanation I can come up with is that the site is rendered by one node and subsequent post backs go to another node - which would leave me to believe that the problem lies with the load balancer. Our infrastructure guys tells me that everything is as it should be with regards to the load balancer and the scenario that I am proposing will not happen.
Am I missing the obvious here or are there anything else that I can consider?
Thanks in advance
Basically, yes, you're right - you generally see this in a web farm environment when "Sticky Sessions" aren't properly configured in the load balancer, and the users postback is sent to a different server.
To be fair to your network guys, it's possible that most requests are being sent to one server, but that this application is tipping the usage such that requests are often sent to another server - but you should be seeing that across all sites, unless the traffic patterns are completely different.
The other possible cause is that either your page is taking too long to load, and the users are posting back before the page has completely finished loading - I'd managed to get one of my sites doing that with a couple of remote advert calls buried halfway through the page load, or the users are waiting too long between page render and postback and the session on the loadbalancer is timing out so it thinks it's a new request.
If you are working with a web-farm environment, machine key values, if specified in the web.config need to be synced. In addition, you will want to make sure that the machine key values in the machine.config file are also synced between the two.
I'm having a problem with a web app I'm managing. Users starting receiving the following error occasionally:
Validation of viewstate MAC failed. If
this application is hosted by a Web
Farm or cluster, ensure that
configuration specifies
the same validationKey and validation
algorithm. AutoGenerate cannot be used
in a cluster.
The problem is that it's not a cluster - it's a single Windows 2003 server. After digging around, it appears that adding a machineKey section and some extra attributes to the Pages directive in my web.config resolves this error:
<machineKey validationKey='MACHINE KEY SNIPPED'
decryptionKey='DECRYPTION KEY SNIPPED'
validation='SHA1'/>
<pages validateRequest="true" enableEventValidation="false">
After changing these two things in my web.config, the error goes away, but now I have a new problem - Instead of an error that my viewstate is invalid, the app just "Forgets" who my user is, and sends them back to the login page. Now, the users are browsing through the application, and then they're unexpectedly sent to the login page, even after they've already been logged in for a few minutes. While I can't force this to happen, it usually happens within visiting 10-12 different pages, so pretty frequently.
I'd love a resolution to this - does anybody know what else might be causing the viewstate error on a single server, or what I can do to ensure that it's validated properly?
It sounds as though the worker process is recycling itself (assuming you're storing session state in-process). Picking a fixed key means that the viewstate is still valid when the process comes back up, but you've lost the session state. You could try storing the session state in a database, but I'd be more concerned to fix the underlying problem. Does your application suddenly allocate vast amounts of memory, or anything like that? Is there anything suspicious in the event log?
It turned out that this began happening when I added additional worked processes to the app pool that was running our application. Because the session state was being stored InProc (and not in a state service or a SQL Server), it was losing track of who the user was when it switched them between working processes. For now, dropping the number of processes on our server back to one corrected the problem, since raising it didn't seem to have any improvement in the first place.
1st post on stackoverflow, hope to have great feedback :)
I'm currently trying to load balance our web site. We have set up a 2 cluster NLB on windows server 2003 with IIS 6.
While testing the setup, I found that sometimes, our session is lost. A day and a half later, here's the result:
Yes, our machine.config both have the same encryption/decryption key.
Yes, the id in iis metabase.xml are the same for both machine. Actually, the entire file are the same, except for "AdminACL".
Both web application are set with "StateServer" and both pointing at the same machine.
From that point, searching on google gives less information and possible solutions.
From what I know, there's no particular pattern that cause this problem. It just happen once in a while.
While trying to find the problem, I've seen that a request sent the asp session id cookie to the server, but the server didn't map it to the user session.
So the request number x was sent from the client, with the cookie, session was mapped, and everything went smoothly.
The request number x+1 was sent from the client, with the cookie, but session was not found.
Both request were made on the same machine in the NLB.
Here's a snippet of the asp trace.axd:
1st request:
Request Details
Session Id: j2ffvy45updpc52uhw1mbg55 Request Type: GET
Time of Request: 11/26/2008 2:58:06 PM Status Code: 200
Request Encoding: Unicode (UTF-8) Response Encoding: Unicode (UTF-8)
Request Cookies Collection
Name Value Size
ASP.NET_SessionId j2ffvy45updpc52uhw1mbg55 42
AID 22 9
Response Cookies Collection
Name Value Size
Headers Collection
Name Value
Cookie ASP.NET_SessionId=j2ffvy45updpc52uhw1mbg55; AID=22
2nd Request:
Request Details
Session Id: Request Type: POST
Time of Request: 11/26/2008 2:58:08 PM Status Code:
Request Encoding: Unicode (UTF-8) Response Encoding:
Request Cookies Collection
Name Value Size
Response Cookies Collection
Name Value Size
Headers Collection
Name Value
Cookie ASP.NET_SessionId=j2ffvy45updpc52uhw1mbg55; AID=22
As you can see in the 2nd request, the cookie is sent from the client, but asp seems to never add the cookies in it's "Request Cookies Collection". I think that's why it doesn't find the session.
So why the cookie is not mapped to the session? Is that the problem? Is the problem elsewhere?
Feel free to ask any clarifications.
Thank you all for your feedback.
JF
I finally found the answer to my problem. It's origin are within the application code (like 99% of a programmer's 3rd party tools 'bugs'). I decided to post it anyway in case someone is in a similar scenario.
This code was part of WebServiceRequester class. The web service requester class was instanciated when session was created and it is saved in session. During creation, we initalizate the member 'm_webServiceURL', and this member is saved in session after. At which value was this member initialize was depending on a setting on the local machine.
The important part is the following:
WebServiceRequester class contains a WebService objects.
WebService objects can't be saved in session, they are not serializable in asp. The property had the [NonSerialized] attribute on it. So everytime we accessed the 'WebService' property of the object for the first during a page life cycle, we had to create a new one, and assigning ot it the url 'm_webServiceURL' which was saved in session.
So you see, new webservice object, on possibly a different machine, meaning a different setting on each machine.
so here's what happened:
box 29 was set to access Web Service at localhost
box 30 was set to access Web Service as 192.168.253.29.
Technically, they are both set on the same machine. But here's a scenario:
login on box 29. m_webServiceURL is set to localhost in session.
[some request on box 29 here]
NLB balancing bring us on box 30.
box 30 loads it's session, create a new webservice obect with localhost as the web service address.
box 30 made the request to the wrong web service leading to a Session Expired exception.
One of the problem during the debug, was that the local communication were not recorded with the network monitor.
What lead me on the trace, was that we never had an exception logged on the box 29 log trace, as it should have.
Thanks for you suggestions everyone, it was really appreciated.
Have a good day.
JF
Not strictly an answer to your question, but have you tried it using a sql server based session store? (Search on MSDN for the permanent script rather than the temp script that's provided with asp.net)
I've heard "bad things" about the executable session service, and consequently have not used it. Never had any problems web farming with the sql server based solution though.
Sorry it's not strictly an answer to your problem, but it should either (a) fix it, or (b) narrow it down significantly.
Well, if you're using visual studio, you could at least test it with the MSDE (the cut down version of SQL Server that comes with Visual Studio)...
It might help rule out state server problems...
Using the database approach has its own issues. I think you should be able to use your preferred approach.
Perhaps this session troubleshooting article would help?
Or "Troubleshooting Session Related Issues in ASP.NET"
Or "Troubleshooting Expired ASP.NET Session State and Your Options"
I'll be lame and re-iterate the proposal of MS SQL Server. Install SQL Server Express which is completeley free including for commercial use and it has only these 3 drawbacks which shouldn't be a problem for you in this stage :
Max 4GB size database
Max 1 CPU Core used
Max 1GB RAM used
A few points to take into consideration:
What's the load on your website? State Server has the tendency to crash when facing a large number of concurrent hits. We're only using it in scenarios where we have a really small number of users (in the 10's, mostly backend systems). Whenever we tried using it in production for sites serving 1000's of users daily, it would crash leading to loss of session data.
On one of the production environments we manage, we're using MSSQL 2005 Express to manage the sessions, the site has 10K+ users a day and 200K+ pages a day. This is a recommended approach in case session is a must and tightly coupled into your application.
If you're about to user MSSQL Express as your state DB, remember that it doesn't come with SQL Server Agent meaning there's no tasks scheduler running in the background and cleaning your expired sessions. I'd recommend finding a scheduler and running the clean expired sessions stored procedure periodically.
Good luck
Instead of messing around with SQL, send your tests directly into one of your IIS nodes to see if you still get the same issue. I'm sure if your only doing a small number of tests StateServer won't be the issue.
Try setting the domain name of the asp.net_sessionid through code to ".yourdomain.com".
By default the ASP.net_SessionID cookie domain name is set to the full application path. So, this may be one of the reason why the cookie is not travelling.
E.g.
Request.Cookies["ASP.NET_SessionId"].Domain = ".yourdomain.com".
Remember the first "." is important in the domain name.
You could do this in the HttpModule in the AcquireRequestState event.