Our (reasonably busy) intranet web site has recently been sometimes taking 2-3 minutes to log in users on the production server, and we haven't been able to find why.
The web site is a "top-level" web site (mapped to an IP address) containing an additional 10 virtual directories. The web site contains both classic ASP and ASP.NET pages, and is running .NET 4.0 (version 4.0.30319.235; the latest, I believe) and the entire site is HTTPS (SSL). It uses ASP.NET Forms authentication with an LDAP provider (the corporation's LDAP in the same domain). It's a very straightforward authentication, with an LDAP connection string and provider configuration in the root web.config, and an "asp:login" control, using the defined LDAP provider, on the login page. The web site is configured with wildcard mapping in IIS to allow the classic ASP pages to be authenticated with the same ASP.NET login process. Sessions are the default "Inproc", not using SQLServer or a state service.
Intermittently, starting about 3 weeks ago, it has been taking 2-3 minutes for users to log in to the web site. There are some pages in the site that don't require authentication and those still perform fine, and after loggin in, authenticated pages also work fine without any delays. It's only the login process itself that's slow.
The exact same website configuration and code running on the development server and workstations never encounters the same delays in authentication.
The login code has not changed in well over a year, and the site has been running .NET 4.0 for probably a year; the servers were updated to the latest .NET version in October-2010. The wildcard mapping was also set up well over a year ago. The login slowness only started about 3 weeks ago. The web server department is not aware of any changes that were made around that time to the web servers and/or the network.
While the long delay occurs on the login page, I don't think the delay is actually in the authentication process; it seems to be in some sort of setup before that. I have added the setting of a Session variable with the current date/time to the beginning (in the LogginIn event) and end of the login code, and there is usually less than a second between these times; however, the clock time between clicking the "Login" button on the login page and the setting of the first of these Session variables is a couple of minutes (consistently around 2 minutes and 14 seconds). I have tried this with page buffering enabled and disabled with no difference in the times.
The same configuration and code is set up in these various environments:
(OK) Development server: Windows Server 2003 SP2 64-bit (32-bit IIS) in SSL mode
(OK) Development server: Windows Server 2008 R2 64-bit (32-bit IIS) in SSL mode
(OK) dev. workstations: Windows 7 64-bit (32-bit IIS), NOT running SSL
(Sometimes slow) Staging web server: Windows Server 2003 SP2 32-bit in SSL mode
(Sometimes slow) Production web cluster: Windows Server 2003 SP2 32-bit in SSL mode
That is, in the development servers and workstaions, I cannot seem to cause the same slowness no matter what I try - resetting IIS, recycling the app pool, updating web.config, etc.
The development servers are using self-signed certificates for SSL; the staging and production servers are using "official" (Verisign) certificates. The production web servers are also used for other web sites (and the other web sites are all still at .NET 2.0), but none of those other sites are using LDAP authentication. All machines (production and development) are up-to-date with Windows Updates.
I can reproduce the problem on the staging or production servers by forcing an application restart (such as by updating web.config) or asking for an IIS Reset, or (on the staging server) waiting for more than 20 minutes with no activity (I think, for the application pool default lifetime to end).
We have checked the following:
The "machineKey" section in every web.config in the entire site is identical
The application pool for the site is set to .NET 4.0, and no other sites are using the same application pool
The application pool identity is "NETWORK SERVICE"
We have tried the following in production with no change in the occasional login delay:
Adding connectionProtection="None" to the LPAP provider configuration
Adding an "applicationName" to the LDAP provider configuration
Adding a port number (the SSL/secure port number) to the LDAP connection string
We have looked through Windows event logs and the IIS logs on the staging and production servers, without finding an obvious connection to the issue. The only possibly-related error that does sometimes occur is the following (that's been logged sometimes about a second before the login succeeds, not at the time that the "Login" button was clicked), however, this may also be because a previously-logged in session has timed out:
Event code: 4006
Event message: Membership credential verification failed.
Application information:
Application domain: /LM/W3SVC/ [...]
Trust level: Full
Application Virtual Path: /
Process information:
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Can anyone suggest any other troubleshooting ideas or potential configuration issues that should be checked or changed?
You can take a memory dump when the problem is happening, and then you use windbg do the analysis.
How to take memory dump:
http://support.microsoft.com/kb/286350
This blog has lots information about how to analysis memory dump
http://blogs.msdn.com/b/tess/
Related
We have an asp.net webforms site we are migrating to IIS 10, server 2022 datacenter Azure Edition VM, from an IIS 8.5 Server 2012 R2 Azure VM.
When testing the site on the new platform, the issue surfaces after a user on AT&T mobile submits credentials at the log in page, is authenticated and redirected to the protected content area. At this point the site hangs, sometimes the protected content page will partially load and/or the connection times out with a reset. Other times it will not load the protected content area at all. Prior to LogIn authentication, that same User can access other non-protected content pages over https as normal.
The site performs as expected when accessing from any other network. Everything also works if a VPN is used to access the site over the same AT&T mobile data connection.
When the protected content area hangs. If we keep the browser open and enable the VPN afterwards using the same Mobile connection the content will load normally without having to reauthenticate.
There are no issues on the original server 2012 R2 running IIS 8.5 even using AT&T mobile.
Other info:
Forms Authentication - Target framework 4.7.2
Confirmed the user authenticates on the server with their login creds and is issued auth cookie over SSL. Auth cookie appears in the response header on client machine. AppAuth Cookie size 3.5kb
Once the page hangs, the user can no longer access any prior page from the unprotected content area until they delete the site cookies.
IIS logs show the Post from the log in and the 302 redirect.
Wire Shark shows a successful handshake on TSL1.2 on the client machine. Sometimes see a RST, ACK in Wireshark after several data packets.
HttpErr Log sometimes indicates Timer_EntityBody.
Confirmed issue exists on multiple devices and connections using AT&T Mobile data. From both a browser and as a mobile hot spot connected to a computer.
No additional Deny filters are configured in the Azure NSG.
Other Actions taken to troubleshoot:
Disabled TLS 1.3 over TCP in IIS Bindings.
Disabled HTTP/2 in IIS Bindings
IIS Rebooted after changes
Re-provisioned a different public IP and NSG for the VM in Azure.
Set Timer_MinBytesPerSecond = 0
What are we missing?
Is there a different setting in IIS 10 vs IIS 8.5 that could cause this issue on certain networks?
I have a .Net application hosted in IIS 10 running on Windows Service 2019. Sometimes the website stops responding (usually when running E2E tests). Even after restarting Website/Application Pool/IIS/Machine it doesn't work.
Looking in Event Viewer I see errors like these:
Forms authentication failed for the request. Reason: The ticket supplied has expired
Failed to stop a listening channel for protocol 'http' at allotted time from worker process serving application pool
A process serving application pool exceeded time limits during shut down
In HTTPERR files I see a lot of messages containing Connection_Abandoned_By_ReqQueue and Connection_Dropped
In inetpub log files I can't see any relevant, just the url requests.
To add more information, we have signalr installed and sometimes errors appear in the events, errors with messages like:
The user identity cannot change during an active SignalR connection.
Any idea what might be causing this?
While using Microsoft Azure Services I have setup my UAT environment as follows:
2 - VMs with same configuration running on Window Server 2012 -> For
MVC (FrontEnd)
2 - VMs with same configuration running on Window
Server 2012 -> For API (Backend)
2 separate NLBs for both of these machines to distribute the loads evenly between the designated VMs
The API one works fine no issue on that but on the MVC whenever the request of single user is redirected to machine 2 from machine 1 it always results in Ant forgery Exception and I am forced to sign out and sign in again. This happens in alternate case as well when the request is redirected from VM2 to VM1.
Things I have tried and still no success:
I have tried setting up SharedSession for the machines to keys from a common shared file which I placed on the primary server (VM1)
In the above I tried placing the SharedSession key on both the VMs in case it get's an issue but still no resolution
I have tried setting up user generated machine keys on IIS with custom settings and it still won't work
Is it possible to have an IIS service autostart with no access to the ApplicationHost.config file?
I have shared hosting on GoDaddy, with a few IIS services running. One of these services contains a method that starts a timer and updates some data on my server every 15 minutes. This is necessary for my Windows Phone app to function correctly.
However, GoDaddy seems to restart their servers once or twice every weekday at random times, which stops the service (and the necessary timer) from running until I call the method from a client application. I have to check pretty frequently to see if things are running and then start the service manually.
I'd like the service to start automatically whenever GoDaddy restarts the server, but since it's shared hosting I don't have access to ApplicationHost.config, and there doesn't seem to be an option for it in GoDaddy's IIS settings.
Anyone have any ideas?
Thanks!
IIS will recycle/terminate an idle Web app/w3p.exe every 20 minutes by default, for many good reasons. IIS also have a lot settings to restart Web apps that consume to much resource or generate too many errors. GoDaddy would restart servers regularly to clean up environment, since the servers might host a lot web applications with poor quality.
Basically you can do
You have a 24x7 machine, and you write a program or use cUrl to send
client requests to your Web app every 1, 5, or 10 minutes to wake
up.
If you don't have a 24x7 machine, you may try some function in
Amazon AWS to do wake up/warm up calls. I don't remember the
respective service name in Amazon.
Or you contact Godaddy's technical support about this issue.
I developed a e-commerce project. My project copies running on Server 2003 / IIS 6 with .Net Framework 4.0 .
Today, i have Server 2008 and IIS 7.5. Project is running normally but when i redirected https page, session is broken.
How do i fix it?
IIS creates different application pools for each http and https bindings. so session state will be different in each binding (protocol) so you can not access session from one protocol in other one, unless you set cookie in client machine or set cookie name for your session.
the simple solution is to set domain wide cookie in http page and retrieve it in https page!