Application "freezes" and requests queue is full - asp.net

I have an asp.net application running on IIS 7.5 , Windows Server 2008 R2 – using an application pool in classic mode, framework version 4.
Sometimes I am running into the following problem:
The application can work for a few days, but all of a sudden I receive an http error 503 – server unavailable.
When I look at the application pool it seems to be running (I see it started), but it actually FREEZES – every request to it is responded with 503.
At the worker processes list (in the IIS manager) I see a lot of requests unhandled.
It's important to mention that other asp.net application running under other application pools are working just fine which means the IIS is working fine and the problem is only in this specific application pool.
When I researched the http error logs I saw the following error logs in the Windows\System32\LogFiles\HTTPERR folder:
In regular settings when everything works fine I noticed records of
"Timer_ConnectionIdle" (a normal thing from what I have read).
At a certain times I notice an appearance of "Client_Reset" records.
15 minutes after the "Client_Reset" errors started->records of "QueueFull appear".
In order to work with the application I am doing an iisreset (I guess a recycle for the pool will be enough also).
I will be happy to receive any help or suggestions.
EDIT:
It's important to mention that nothing related gets written to the IIS logs, or the System and Application logs. This error occurs before.

Without more information about your problem, most quick fix will be to Configure Recycling Settings for an Application Pool. Since your problem is about request queues, you can choose option After reaching a number of requests.

When I encounter HTTP 500 errors, I find that enabling IIS Tracing is very helpful. Below is a link that describes the process of enabling tracing and then reviewing the trace. The first section of the site describes how to install IIS, so you probably want to skip to the section labeled "Enable Failed-Request Tracing."
Troubleshooting Failed Requests Using Tracing in IIS 7
Edit:
Since you're getting a QueueFull error, you may want to monitor the request queues. The easiest way to do this is using Perfmon. On the server with IIS, open Performance Monitor and add the appropriate counters under "HTTP Service Request Queue." In your case, "Current Request Queue" for the ailing Application Pool would likely be of value.

Related

IIS .Net Website hangs even after restart IIS/Machine

I have a .Net application hosted in IIS 10 running on Windows Service 2019. Sometimes the website stops responding (usually when running E2E tests). Even after restarting Website/Application Pool/IIS/Machine it doesn't work.
Looking in Event Viewer I see errors like these:
Forms authentication failed for the request. Reason: The ticket supplied has expired
Failed to stop a listening channel for protocol 'http' at allotted time from worker process serving application pool
A process serving application pool exceeded time limits during shut down
In HTTPERR files I see a lot of messages containing Connection_Abandoned_By_ReqQueue and Connection_Dropped
In inetpub log files I can't see any relevant, just the url requests.
To add more information, we have signalr installed and sometimes errors appear in the events, errors with messages like:
The user identity cannot change during an active SignalR connection.
Any idea what might be causing this?

Specific IIS user not working with TLS 1.2

We have run into a problem with IIS, TLS 1.2 and domain users. I searched SO and other forums, but all possibly related topics didn't lead me to a solution.
Please don't judge the configuration, it wasn't invented by me, I just need to solve this problem.
What happens is the following:
We have an old web application, that opens an executable with Process.Start and that executable calls an external webservice. This used to work fine with TLS 1.0, but in the near future, the external webservice demands TLS 1.2.
So now we are trying to make this work, and we are almost there: we upgraded the executable's .Net Framework version to 4.7.2 and enabled TLS 1.2 on the Windows Server 2008 R2. The web app's .Net Framework version is set to 4.6.1. It seems to me that this should be everything there is to it.
And indeed, when we run the executable stand alone (not called by the web app) from the server, so owned by the domain user logged on to the server (with RDP), everything works as expected; we receive the proper answer from the web service.
Also, when we call the executable by the web app and in IIS the application pool identity is set to a build in account: ApplicationPoolIdentity, everything works as expected as well.
But, when we set the application pool identity to a dedicated domain account (so a different one than the one that executed the executable earlier), the trouble begins. Connecting the web service fails with the following exception:
System.ServiceModel.EndpointNotFoundException: There was no endpoint
listening at https://<some url>/<some webservice name>.asmx that could
accept the message. This is often caused by an incorrect address or
SOAP action. See InnerException, if present, for more details. --->
System.Net.WebException: Unable to connect to the remote server
---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a
period of time, or established connection failed because connected
host has failed to respond ...
Now the question is of course, what could be causing this?
I like to believe that the failing domain account is configured correctly, but it seems it is not. Or could it be something else, that I don't even know the existence of...
EDIT:
I managed to narrow it down to a permissions issue: when the dedicated domain account runs the application stand alone, it works as it should. When the dedicated account runs it from within the IIS context (started by the web app), it doesn't work, but when the dedicated account is given admin rights, it also works as expected.
That leaves me to the question: what additional permissions does IIS need to allow this setup? Maybe in combination with TLS 1.2 thingies.
Any help would be greatly appreciated.

HTTP Error 503. The service is unavailable

I'm struggling to setup the environment in IIS8, I searched a lot but couldn't find a right solution.
I checked the error logs, but no idea.
C:\Windows\System32\LogFiles\HTTPERR
2013-10-09 09:28:39 192.168.43.205 60172 192.168.43.205 80 HTTP/1.1
GET / 503 2 AppOffline qa.hti.local
2013-10-09 09:28:39 192.168.43.205 60192 192.168.43.205 80 HTTP/1.1
GET /favicon.ico 503 2 AppOffline qa.hti.local
Then in Event Viewer:
WARNINGS:
A listener channel for protocol 'http' in worker process '11188'
serving application pool 'qa.hti.local' reported a listener channel
failure. The data field contains the error number.
A listener channel for protocol 'http' in worker process '7492'
serving application pool 'qa.hti.local' reported a listener channel
failure. The data field contains the error number.
A listener channel for protocol 'http' in worker process '9088'
serving application pool 'qa.hti.local' reported a listener channel
failure. The data field contains the error number.
A listener channel for protocol 'http' in worker process '9964'
serving application pool 'qa.hti.local' reported a listener channel
failure. The data field contains the error number.
A listener channel for protocol 'http' in worker process '7716'
serving application pool 'qa.hti.local' reported a listener channel
failure. The data field contains the error number.
I don't understand what the warning means.
ERROR: Application pool 'qa.hti.local' is being automatically disabled due to a series of failures in the process(es) serving that
application pool.
Note: I learned that consecutive 5 failures leads to APP Pool crash, and this can increased. I also tried increasing this but no success.
Please share your thoughts.
I came across this question as I was experiencing a similar issue and searching for a solution.
My problem specifically had to do with our IIS shared configuration. We had enabled a feature in IIS on one of the servers (Http Redirect) that was not installed on any of the others so the server 'features' were out of sync with all the servers.
I was able to resolve the issue by uninstalling the new addition on the first server so it was back to matching the others. An IIS reset later and the AppPools were no longer going down and all was back to normal.
So if you are using IIS Shared Configuration and the IIS is creating 'Service Unavailable' errors and the AppPools are going down, this can be a symptom of the system configuration being out of synch which is corrupting the shared configuration. Hopefully this post will help someone find the solution faster than I was able to.
I had a similar problem, and it was due to another error (2282 IIS-W3SVC-WP) that the pool stopped itself. My problem was that de module owssvr.dll could not be loaded due to a configuration problem.
The solution for me was to set the setting "Enable 32-bit applications" from True to false.
I was deploying a solution on Sharepoint 2013 on a Windows Server 2012 with Visual studio 2013.
I know this was supposedly a very specific problem, but I want to help all those with the same problem.
Propably you do not have the permissions to read the directory.
The directory (and the files) need to have read-access for either windows-group "USERS" or windows-grou "IIS_IUSRS" and also for your apppoolidentity.
This occurred for me too on Windows 10 1803 after an update.
Earlier in the event log there were errors to do with missing DLLS, specifically iiswsock.dll and compstat.dll.
After turning on the following Windows features (under IIS > WWWServices > Performance Features and AppDev Features):
Dynamic and Static Content Compression, and
WebSocket Protocol Windows features
those errors disappeared after an IIS restart.
503 2
Is that 2 the substatus code? If so, you might want to make sure your site is not being hit with excessively (5000+) # of requests.
http://support.microsoft.com/kb/943891
The data field contains the error number.
what's the error number?
This also can be caused by you're software vendor not realizing that they installed the 32bit version of the application pool apps on your 2008 R2 server. After a little troubleshooting my server because they wanted me to reinstall IIS i checked the windows app logs and emailed them the x86 architecture error for their app.
This can occur if a piece of IIS is missing (e.g. one of the many IIS modules has not been installed on the new server).
The thing to do is to carefully compare the IIS config of the source and target, and add the missing pieces to the target.
In a recent IIS8.5->8.5 migration, I had this issue. Went through the whole stack. The last piece was that the Web Cert auth piece of IIS was missing.
To install:
powershell
import-module servermanager
add-windowsfeature Web-Cert-Auth
I reinstalled server and copied applicationHost.config to new server, but not installed corresponding modules, and got this error. I fixed that by check modules in IIS manager and ensure modules are installed.
Update: In Windows Logs-> Application, there're some info about which module is missing.
In case this helps anyone else: We run multiple https sites on an IIS 10 server. One of the steps in configuring a new site is to give the new application pool permission to the system certificate. In the registry, under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\MY, grant the application pool on the local machine full control. That fixed this issue for us.
I had the same problem and solved it by adding the domain account the application pool was using as an identity to the local group of Administrators on a web server. Perhaps it would also do the trick to grant access to the application directory for the application pool identity account, as #Lisa-Berlin stated above.

Net.Msmq Binding Not Picking Up Messages in Queue as soon as the application pool gets recycled

"when the application pool is "alive" the service picks up the messages correctly, but as soon as the application pool gets recycled (because of timeout or any other reason), the service stops picking up the messages, that just sit
in the queue until the service starts again by browsing to the service webpage"
Have you find a solution, to activate the service without manualy browse the service.
The solution is to configure auto-start. Then IIS will start your service immediately without waiting for the first request.
But first you need to add AppFabric to your ISS, then you need be sure you have the "Start Mode" option in your pool advance settings.
Note: In my windows 7 IIS7 didnt work, but in my windows server 2012 R2 IIS8 Works perfectly
Also you can check this similar question:
MSMQ WCF hosted in IIS

Requests stuck in RequestAcquireState in IIS worker process

On one of our production servers, occasionally requests get stuck in the RequestAquireState while in the session module. As it is an MVC request, it does not timeout, so we sometimes get requests that run in the background for several hours.
We are using the standard asp.net session module on .net4 and IIS 7.5 We are using InProc.
Why would it get stuck?
Had the same problem when running with Asp.net State Server, restarting the service resolved the problem.

Resources