On one of our production servers, occasionally requests get stuck in the RequestAquireState while in the session module. As it is an MVC request, it does not timeout, so we sometimes get requests that run in the background for several hours.
We are using the standard asp.net session module on .net4 and IIS 7.5 We are using InProc.
Why would it get stuck?
Had the same problem when running with Asp.net State Server, restarting the service resolved the problem.
Related
I have a .Net application hosted in IIS 10 running on Windows Service 2019. Sometimes the website stops responding (usually when running E2E tests). Even after restarting Website/Application Pool/IIS/Machine it doesn't work.
Looking in Event Viewer I see errors like these:
Forms authentication failed for the request. Reason: The ticket supplied has expired
Failed to stop a listening channel for protocol 'http' at allotted time from worker process serving application pool
A process serving application pool exceeded time limits during shut down
In HTTPERR files I see a lot of messages containing Connection_Abandoned_By_ReqQueue and Connection_Dropped
In inetpub log files I can't see any relevant, just the url requests.
To add more information, we have signalr installed and sometimes errors appear in the events, errors with messages like:
The user identity cannot change during an active SignalR connection.
Any idea what might be causing this?
I create a new asp net core website on IIS and worked properly, but after publish the load balancer a new issue appears which is after login to the system, the SignalR connection closed immediately and logout from website due to session timeout.
Noting that no clear error appears.
Any advice please.
In load balanced environments SignalR needs to be back-planed, this puts what would otherwise have been stored in your webservers memory into a database. Without back-planing when a SignalR reply comes in via the load balancer and does not end up at the server which originally asked for the reply from the servers POV the reply isn't in response to anything since it has nothing in memory.
Back-planing is really easy. High level overview here: https://learn.microsoft.com/en-us/aspnet/signalr/overview/performance/scaleout-in-signalr
Database specific instructions...
Azure: https://learn.microsoft.com/en-us/aspnet/signalr/overview/performance/scaleout-with-windows-azure-service-bus
Redis: https://learn.microsoft.com/en-us/aspnet/signalr/overview/performance/scaleout-with-redis
SQL Server: https://learn.microsoft.com/en-us/aspnet/signalr/overview/performance/scaleout-with-sql-server
I have an asp.net application running on IIS 7.5 , Windows Server 2008 R2 – using an application pool in classic mode, framework version 4.
Sometimes I am running into the following problem:
The application can work for a few days, but all of a sudden I receive an http error 503 – server unavailable.
When I look at the application pool it seems to be running (I see it started), but it actually FREEZES – every request to it is responded with 503.
At the worker processes list (in the IIS manager) I see a lot of requests unhandled.
It's important to mention that other asp.net application running under other application pools are working just fine which means the IIS is working fine and the problem is only in this specific application pool.
When I researched the http error logs I saw the following error logs in the Windows\System32\LogFiles\HTTPERR folder:
In regular settings when everything works fine I noticed records of
"Timer_ConnectionIdle" (a normal thing from what I have read).
At a certain times I notice an appearance of "Client_Reset" records.
15 minutes after the "Client_Reset" errors started->records of "QueueFull appear".
In order to work with the application I am doing an iisreset (I guess a recycle for the pool will be enough also).
I will be happy to receive any help or suggestions.
EDIT:
It's important to mention that nothing related gets written to the IIS logs, or the System and Application logs. This error occurs before.
Without more information about your problem, most quick fix will be to Configure Recycling Settings for an Application Pool. Since your problem is about request queues, you can choose option After reaching a number of requests.
When I encounter HTTP 500 errors, I find that enabling IIS Tracing is very helpful. Below is a link that describes the process of enabling tracing and then reviewing the trace. The first section of the site describes how to install IIS, so you probably want to skip to the section labeled "Enable Failed-Request Tracing."
Troubleshooting Failed Requests Using Tracing in IIS 7
Edit:
Since you're getting a QueueFull error, you may want to monitor the request queues. The easiest way to do this is using Perfmon. On the server with IIS, open Performance Monitor and add the appropriate counters under "HTTP Service Request Queue." In your case, "Current Request Queue" for the ailing Application Pool would likely be of value.
We've implemented a "background service" in our Asp.Net web app that receives messages from MSMQ at random intervals without requiring an HTTP request to start up the application. We auto start this web app using a serviceAutoStartProvider.
Everyhing works when IIS initially starts up, the server is rebooted and so on, we receive messages just fine. BUT if we just stop the site in IIS (not touching the application or app pool), the application stops receiving MSMQ messages. And when we start the web site again, the serviceAutoStartProvider is not called again, so our app does not start listening to MSMQ messages again!
If we issue a HTTP request against the web app after the IIS site has been stopped and started again, it starts listening to MSMQ messages again.
Shouldn't our "background service" web app continue to listen to MSMQ messages even if the IIS site is stopped? It won't get any requests, but I think it should continue to run.
What exactly happens in an Asp.Net application/app pool when the IIS site is stopped? Any events fired that we can hook up to? The app pool claims to be "started" in IIS manager, but code is not running in it.
Why isn't our serviceAutoStartProvider called when the site is started again? I believe it is "by design", since the application isn't really stopped. But the applications isn't running, either, has to be waken up by an actual HTTP request.
When the IIS Web App shuts down (eg. due to no new HTTP(S) requests for the timeout time) the .NET app domain (within the app pool worker process) completely closes and unloads. This includes all background threads, including those used by the .NET thread pool.
A Web App can be configured with a longer (or no) timeout, then background worker threads could continue to process work.
But better would be to either run such workers in a specialist service process managed completely separately.
Or, even better, use IIS application hosting with WCF to create the MSMQ listener. I understand in this case the integration of Windows Process Activation Services with IIS would restart the Web App if a new message arrived after it had been shutdown.
I would host the MSMQ listenner in a windows service. Why couple it to IIS?
UPDATE
Actually what I mean is why couple the MSMQ and ASPNET in the same app pool?
You can now use "Application Initialization" feature of IIS8 (and IIS7.5), more information including version availability and usage documentation can be found at:
http://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-80-application-initialization
This replaces "Application Warm-Up Module" which is no longer supported, and provides us with proper control over component/service initialization in an "always running" scenario.
Our (reasonably busy) intranet web site has recently been sometimes taking 2-3 minutes to log in users on the production server, and we haven't been able to find why.
The web site is a "top-level" web site (mapped to an IP address) containing an additional 10 virtual directories. The web site contains both classic ASP and ASP.NET pages, and is running .NET 4.0 (version 4.0.30319.235; the latest, I believe) and the entire site is HTTPS (SSL). It uses ASP.NET Forms authentication with an LDAP provider (the corporation's LDAP in the same domain). It's a very straightforward authentication, with an LDAP connection string and provider configuration in the root web.config, and an "asp:login" control, using the defined LDAP provider, on the login page. The web site is configured with wildcard mapping in IIS to allow the classic ASP pages to be authenticated with the same ASP.NET login process. Sessions are the default "Inproc", not using SQLServer or a state service.
Intermittently, starting about 3 weeks ago, it has been taking 2-3 minutes for users to log in to the web site. There are some pages in the site that don't require authentication and those still perform fine, and after loggin in, authenticated pages also work fine without any delays. It's only the login process itself that's slow.
The exact same website configuration and code running on the development server and workstations never encounters the same delays in authentication.
The login code has not changed in well over a year, and the site has been running .NET 4.0 for probably a year; the servers were updated to the latest .NET version in October-2010. The wildcard mapping was also set up well over a year ago. The login slowness only started about 3 weeks ago. The web server department is not aware of any changes that were made around that time to the web servers and/or the network.
While the long delay occurs on the login page, I don't think the delay is actually in the authentication process; it seems to be in some sort of setup before that. I have added the setting of a Session variable with the current date/time to the beginning (in the LogginIn event) and end of the login code, and there is usually less than a second between these times; however, the clock time between clicking the "Login" button on the login page and the setting of the first of these Session variables is a couple of minutes (consistently around 2 minutes and 14 seconds). I have tried this with page buffering enabled and disabled with no difference in the times.
The same configuration and code is set up in these various environments:
(OK) Development server: Windows Server 2003 SP2 64-bit (32-bit IIS) in SSL mode
(OK) Development server: Windows Server 2008 R2 64-bit (32-bit IIS) in SSL mode
(OK) dev. workstations: Windows 7 64-bit (32-bit IIS), NOT running SSL
(Sometimes slow) Staging web server: Windows Server 2003 SP2 32-bit in SSL mode
(Sometimes slow) Production web cluster: Windows Server 2003 SP2 32-bit in SSL mode
That is, in the development servers and workstaions, I cannot seem to cause the same slowness no matter what I try - resetting IIS, recycling the app pool, updating web.config, etc.
The development servers are using self-signed certificates for SSL; the staging and production servers are using "official" (Verisign) certificates. The production web servers are also used for other web sites (and the other web sites are all still at .NET 2.0), but none of those other sites are using LDAP authentication. All machines (production and development) are up-to-date with Windows Updates.
I can reproduce the problem on the staging or production servers by forcing an application restart (such as by updating web.config) or asking for an IIS Reset, or (on the staging server) waiting for more than 20 minutes with no activity (I think, for the application pool default lifetime to end).
We have checked the following:
The "machineKey" section in every web.config in the entire site is identical
The application pool for the site is set to .NET 4.0, and no other sites are using the same application pool
The application pool identity is "NETWORK SERVICE"
We have tried the following in production with no change in the occasional login delay:
Adding connectionProtection="None" to the LPAP provider configuration
Adding an "applicationName" to the LDAP provider configuration
Adding a port number (the SSL/secure port number) to the LDAP connection string
We have looked through Windows event logs and the IIS logs on the staging and production servers, without finding an obvious connection to the issue. The only possibly-related error that does sometimes occur is the following (that's been logged sometimes about a second before the login succeeds, not at the time that the "Login" button was clicked), however, this may also be because a previously-logged in session has timed out:
Event code: 4006
Event message: Membership credential verification failed.
Application information:
Application domain: /LM/W3SVC/ [...]
Trust level: Full
Application Virtual Path: /
Process information:
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Can anyone suggest any other troubleshooting ideas or potential configuration issues that should be checked or changed?
You can take a memory dump when the problem is happening, and then you use windbg do the analysis.
How to take memory dump:
http://support.microsoft.com/kb/286350
This blog has lots information about how to analysis memory dump
http://blogs.msdn.com/b/tess/