IIS application pool crashing unexpectedly - asp.net

We are facing a very tricky issue for almost one week now and we are still unable to solve it.
The issue is that we have an application pool on IIS who is crashing and recycling several times a day.
We monitored what's happening on event viewer but the error there was not enough detailed.
Example of the 2 errors (2 seconds between them) from windows event viewer :
Error 1 :
Event Id : 1000
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/20/ROOT
Process ID: 14300
Exception: System.ObjectDisposedException
Message: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
StackTrace: at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FtpClient.FtpSocketStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.FtpClient.FtpSocketStream.WriteLine(Encoding encoding, String buf)
at System.Net.FtpClient.FtpClient.Execute(String command)
at System.Net.FtpClient.FtpClient.Disconnect()
at System.Net.FtpClient.FtpClient.Dispose()
at System.Net.FtpClient.FtpClient.Finalize()
Error 2:
Event Id : 1325
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18666, time stamp: 0x58f32841
Exception code: 0xe0434352
Fault offset: 0x00015608
Faulting process id: 0x37dc
Faulting application start time: 0x01d35b56102b1ddf
Faulting application path: C:\windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\windows\SYSTEM32\KERNELBASE.dll
Report Id: b9f70358-c76a-11e7-8111-0cc47a0c4e39
Faulting package full name:
Faulting package-relative application ID:
At this moment, we were able to find which error is making app pool crashing. It concerns accessing disposed object from FtpClient library. But we were still not able to find what in our code is doing such this call.
Then, we decided to install DebugDiag (https://www.microsoft.com/en-us/download/details.aspx?id=49924). A very powerful tool by the way.
We launched DebugDiag for monitoring app pool crashes and we were able to generate a very detailed report (in .mhtml format).
Here is the full call stack :
.NET Call Stack
[[HelperMethodFrame]]
System_ni!System.Net.Sockets.NetworkStream.Write(Byte[], Int32, Int32)+673bde
System.Net.FtpClient.FtpSocketStream.Write(Byte[], Int32, Int32)+33
System.Net.FtpClient.FtpSocketStream.WriteLine(System.Text.Encoding, System.String)+44
System.Net.FtpClient.FtpClient.Execute(System.String)+1c6
System.Net.FtpClient.FtpClient.Disconnect()+57
System.Net.FtpClient.FtpClient.Dispose()+46
System.Net.FtpClient.FtpClient.Finalize()+10
[[DebuggerU2MCatchHandlerFrame]]
[[ContextTransitionFrame]]
[[GCFrame]]
[[DebuggerU2MCatchHandlerFrame]]
Full Call Stack
KERNELBASE!RaiseException+48
clr!RaiseTheExceptionInternalOnly+27c
clr!IL_Throw+141
[[HelperMethodFrame]]
System_ni!System.Net.Sockets.NetworkStream.Write(Byte[], Int32, Int32)+673bde
System.Net.FtpClient.FtpSocketStream.Write(Byte[], Int32, Int32)+33
System.Net.FtpClient.FtpSocketStream.Write(Byte[], Int32, Int32)+33
System.Net.FtpClient.FtpSocketStream.WriteLine(System.Text.Encoding, System.String)+44
System.Net.FtpClient.FtpSocketStream.WriteLine(System.Text.Encoding, System.String)+44
System.Net.FtpClient.FtpClient.Execute(System.String)+1c6
System.Net.FtpClient.FtpClient.Disconnect()+57
System.Net.FtpClient.FtpClient.Dispose()+46
System.Net.FtpClient.FtpClient.Finalize()+10
clr!MethodTable::FastBox+b0
clr!MethodTable::CallFinalizer+139
clr!CallFinalizer+a6
clr!FinalizerThread::FinalizeAllObjects+a6
clr!FinalizerThread::FinalizeAllObjects_Wrapper+14
clr!Thread::DoExtraWorkForFinalizer+1b1
clr!Thread::DoExtraWorkForFinalizer+234
clr!Thread::DoExtraWorkForFinalizer+5f8
[[DebuggerU2MCatchHandlerFrame]]
clr!Thread::DoExtraWorkForFinalizer+137
clr!Thread::DoADCallBack+30f
[[ContextTransitionFrame]]
clr!Thread::DoExtraWorkForFinalizer+19f
clr!FinalizerThread::DoOneFinalization+129
[[GCFrame]]
clr!FinalizerThread::FinalizeAllObjects+a6
clr!FinalizerThread::FinalizerThreadWorker+ed
clr!Thread::DoExtraWorkForFinalizer+1b1
clr!Thread::DoExtraWorkForFinalizer+234
clr!Thread::DoExtraWorkForFinalizer+5f8
[[DebuggerU2MCatchHandlerFrame]]
clr!ManagedThreadBase::FinalizerBase+33
clr!FinalizerThread::FinalizerThreadStart+d4
clr!Thread::intermediateThreadProc+55
kernel32!BaseThreadInitThunk+24
ntdll!__RtlUserThreadStart+2f
ntdll!_RtlUserThreadStart+1b
We were able to find the exception that made the pool crashing but we are still unable to find which part of our code is doing the call.
At the bottom of the call stack, you can noticed that the first event is : ntdll!_RtlUserThreadStart+1b which I really don't know ..
So I would be glad to have some help to solve this 2 questions :
How to find in my code what is creating the bug ?
How can such error making my application pool crashed and recycled ?
Environment :
- IIS 8.5
- Application pool CLR version 4
- Windows Server 2012 R2

The message Cannot access a disposed object means that the garbage collector has disposed your object, which you then try to use.
Are you calling the FtpClient with using, are you doing a .close()?
The error logs aren't actually useful, it would be better if you posted your code instead.
Also, put you code inside try...catch so that you can capture the error rather than leaving it unhandled. Sometimes file transfers fail so you need to be doing something like the following :
try {
using(Stream s = ftpClient.OpenRead()) {
// perform your transfer
}
}
catch(Exception) {
// Typical exceptions here are IOException, SocketException, or a FtpCommandException
}

Related

IIS Application pool stop every 15 days without any configuration

On a Windows instance a specific application pool stops every 15 days and we have to manually restart that Under the event logs we are getting dotnet clr.dll error given below.
Faulting application name: w3wp.exe, version: 10.0.17763.1, time
stamp: 0xcfdb13d8 Faulting module name: clr.dll, version: 4.7.3610.0,
time stamp: 0x5e13fa89 Exception code: 0xc00000fd Fault offset:
0x000000000012ebdc Faulting process id: 0x2678 Faulting application
start time: 0x01d7069cd23d6f15 Faulting application path:
c:\windows\system32\inetsrv\w3wp.exe Faulting module path:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Report Id:
b44401f0-97b2-4b8c-8fba-703227faec12 Faulting package full name:
Faulting package-relative application ID:
What should be the process for debugging this issue to get the root cause?
I am new to server management and processes so please bear with me.
Thanks in advance.
It looks like you have a StackOverflow Exception, which is caused by unbounded recursion (a function repeatedly calling itself, etc). You can track the problem down using DebugDiag and WinDbg.
DebugDiag can be configured to generate a crash dump when the StackOverflowException occurs.
Download at Debug Diagnostic Tool.
About how to use the Debug Diagnostics tool to troubleshoot a process that has stopped responding in IIS you can refer to this link: How to use the Debug Diagnostics tool.
Debugging tools for Windows is part of the Windows SDK and can be downloaded at Download the Windows Driver Kit (WDK). To use WinDbg, you'll need to get the symbols files. Download the symbol files and put them in a local folder. Getting Started with WinDbg.
In the results, it should be clear what the problem is.

ASP.NET Core 2, IIS AppPool crashes and stops

I've just updated a web app to ASP.NET Core 2.0 running on IIS 7.5 on Windows Server 2008 R2. Since upgrading the app randomly crashes and leaves it with a 503 Service Unavailable response. It can run for several hours before crashing.
Before crashing I get a couple of warnings in event viewer and then a error.
Example of warning:
A process serving application pool '<My app pool>' terminated unexpectedly. The process id was '14896'. The process exit code was '0x0'.
Another example of warning:
A process serving application pool '<My app pool>' suffered a fatal communication error with the Windows Process Activation Service. The process id was '18252'. The data field contains the error number.
Example of error:
Application pool '<My app pool>' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
I guess the crash/error comes from rapid failure feature in IIS, but I can't pinpoint the cause of the original warnings.
I have a few runtime errors like the ones below too, but I'm sure how related they are:
{
"#t": "2017-11-20T15:24:24.3190000Z",
"#m": "Request timed out.",
"#i": "0307d0fb",
"#l": "Error",
"#x": "Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request timed out.\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompletion.ThrowFailed()\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.GetResult(ReadResult& result)\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult()\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadableBufferAwaitable.GetResult()\r\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.<ConsumeAsync>d__24.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()",
"SourceContext": "xxxx.Extensions.GlobalExceptionFilter",
"ActionId": "97118189-7c25-427b-82c7-10b3d885a146",
"ActionName": "xxxx.Controllers.FileController.Upload (xxxx)",
"RequestId": "0HL9G4JA0VTIJ:00000073",
"RequestPath": "/api/file"
}
Event viewer also reports errors like these for the app pool in question:
An unhandled win32 exception occurred in w3wp.exe [18244]. Just-In-Time debugging this exception failed with the following error: Debugger could not be started because no user is logged on.
Check the documentation index for 'Just-in-time debugging, errors' for more information.
And...
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: aspnetcore.dll, version: 7.1.1982.0, time stamp: 0x594ab904
Exception code: 0xc0000005
Fault offset: 0x000000000000fe3b
Faulting process id: 0x1ef0
Faulting application start time: 0x01d3622d8eb0e8c0
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\inetsrv\aspnetcore.dll
Report Id: ce37e890-ce20-11e7-9f93-00155dc82c03
I could really use some help trying to find the source of these crashes.
Microsoft has updated the IIS module for ASP.NET Core now and that solves the problem.

ActiveX component can't create object error. WebsiteError:8004e025

When I am trying to hit my application url, I am getting "ActiveX component can't create object error. WebsiteError:8004e025" error in the browser. This application is hosted in IIS and operating system is Windows Server 2003 32-bit.
When I checked event viewer logs, I got following error:
Event Type: Error
Event Source: COM+
Event Category: (98)
Event ID: 4840
Date: 06/06/2016
Time: 13:24:45
User: N/A
Computer: APW02508WCAPP02
Description:
The IObjectControl::Activate() method failed. The CLSID of the object is:
{CAFBD1A2-73BD-438B-841C-A9C88BD4C248}
Server Application ID: {74CF179D-508C-43FE-AE29-B3480386BDE3}
Server Application Instance ID:{E7855B79-7A03-4E7A-8BA4-37666501917F}
**Error Code** = **0x800a01ad**
COM+ Services Internals Information:
File: d:\nt\com\complus\src\comsvcs\jit\jit.cpp, Line: 475
Comsvcs.dll file version: ENU 2001.12.4720.3959 shp
I tried registering Comsvcs.dll and it got registered, but still I am getting same error. I know that this is not a code issue as this application works on other windows server 2003 machine. Only on this particular machine, I am getting this error. Can anyone please tell me how to resolve this?

How to get more description from Event Viewer error on web server

I have a web application build in asp.net 4.5 which intermittently logs the user out.
I was able to get from the Event Viewer the following error, which I think is the error that's causing the user to be logged out:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8
Faulting module name: nlssorting.dll, version: 4.0.30319.17929, time stamp: 0x4ffa55e0
Exception code: 0xc00000fd
Fault offset: 0x00001cda
Faulting process id: 0x358c
Faulting application start time: 0x01cf11fdded81940
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll
Report Id: 4c2ecce0-7df2-11e3-9850-bc764e080abb
I couldn't get much information from the error description, so I would like to see if there is a way to get more information about it, such as what type of exception was thrown by the web application in order to break the process or (ideally) which dll or method threw that error.
Thanks in advance!

IIS7 App pool crashing

Any ideas?
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: MSVCR100_CLR0400.dll, version: 10.0.30319.1, time stamp: 0x4ba2211c
Exception code: 0xc00000fd
Fault offset: 0x0000000000057f91
Faulting process id: 0x1e40
Faulting application start time: 0x01ccbfff3d24cf1a
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\MSVCR100_CLR0400.dll
Report Id: 8b4b0cd0-2bf2-11e1-9370-005056820010
Nothing has changed code wise. Tried diadebug 1.2 which we have gone through the exceptions.
Sorry not much to go on. If anyone knows best way to trap / capture IIS error ?
The faulting module is C:\Windows\system32\MSVCR100_CLR0400.dll, part of VC++ runtime. As IIS itself is not dependent on VC++ runtime, it is obvious that your application leads to the crash. Dump analysis may help, but if you are not capable of reading the dumps, you may open a support case via http://support.microsoft.com and consult Microsoft support guys.
DIY dump analysis is tough, but you can start if you insist,
http://blogs.msdn.com/b/lexli/archive/2009/06/29/diy-dump-analysis-possible.aspx
Not much information to work with there, but 0xc00000fd is a stack overflow exception. The most common cause is overly recursive function/method calls.

Resources