I've just updated a web app to ASP.NET Core 2.0 running on IIS 7.5 on Windows Server 2008 R2. Since upgrading the app randomly crashes and leaves it with a 503 Service Unavailable response. It can run for several hours before crashing.
Before crashing I get a couple of warnings in event viewer and then a error.
Example of warning:
A process serving application pool '<My app pool>' terminated unexpectedly. The process id was '14896'. The process exit code was '0x0'.
Another example of warning:
A process serving application pool '<My app pool>' suffered a fatal communication error with the Windows Process Activation Service. The process id was '18252'. The data field contains the error number.
Example of error:
Application pool '<My app pool>' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
I guess the crash/error comes from rapid failure feature in IIS, but I can't pinpoint the cause of the original warnings.
I have a few runtime errors like the ones below too, but I'm sure how related they are:
{
"#t": "2017-11-20T15:24:24.3190000Z",
"#m": "Request timed out.",
"#i": "0307d0fb",
"#l": "Error",
"#x": "Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request timed out.\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompletion.ThrowFailed()\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.GetResult(ReadResult& result)\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult()\r\n at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadableBufferAwaitable.GetResult()\r\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.<ConsumeAsync>d__24.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()",
"SourceContext": "xxxx.Extensions.GlobalExceptionFilter",
"ActionId": "97118189-7c25-427b-82c7-10b3d885a146",
"ActionName": "xxxx.Controllers.FileController.Upload (xxxx)",
"RequestId": "0HL9G4JA0VTIJ:00000073",
"RequestPath": "/api/file"
}
Event viewer also reports errors like these for the app pool in question:
An unhandled win32 exception occurred in w3wp.exe [18244]. Just-In-Time debugging this exception failed with the following error: Debugger could not be started because no user is logged on.
Check the documentation index for 'Just-in-time debugging, errors' for more information.
And...
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: aspnetcore.dll, version: 7.1.1982.0, time stamp: 0x594ab904
Exception code: 0xc0000005
Fault offset: 0x000000000000fe3b
Faulting process id: 0x1ef0
Faulting application start time: 0x01d3622d8eb0e8c0
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\inetsrv\aspnetcore.dll
Report Id: ce37e890-ce20-11e7-9f93-00155dc82c03
I could really use some help trying to find the source of these crashes.
Microsoft has updated the IIS module for ASP.NET Core now and that solves the problem.
Related
On a Windows instance a specific application pool stops every 15 days and we have to manually restart that Under the event logs we are getting dotnet clr.dll error given below.
Faulting application name: w3wp.exe, version: 10.0.17763.1, time
stamp: 0xcfdb13d8 Faulting module name: clr.dll, version: 4.7.3610.0,
time stamp: 0x5e13fa89 Exception code: 0xc00000fd Fault offset:
0x000000000012ebdc Faulting process id: 0x2678 Faulting application
start time: 0x01d7069cd23d6f15 Faulting application path:
c:\windows\system32\inetsrv\w3wp.exe Faulting module path:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Report Id:
b44401f0-97b2-4b8c-8fba-703227faec12 Faulting package full name:
Faulting package-relative application ID:
What should be the process for debugging this issue to get the root cause?
I am new to server management and processes so please bear with me.
Thanks in advance.
It looks like you have a StackOverflow Exception, which is caused by unbounded recursion (a function repeatedly calling itself, etc). You can track the problem down using DebugDiag and WinDbg.
DebugDiag can be configured to generate a crash dump when the StackOverflowException occurs.
Download at Debug Diagnostic Tool.
About how to use the Debug Diagnostics tool to troubleshoot a process that has stopped responding in IIS you can refer to this link: How to use the Debug Diagnostics tool.
Debugging tools for Windows is part of the Windows SDK and can be downloaded at Download the Windows Driver Kit (WDK). To use WinDbg, you'll need to get the symbols files. Download the symbol files and put them in a local folder. Getting Started with WinDbg.
In the results, it should be clear what the problem is.
We are facing a very tricky issue for almost one week now and we are still unable to solve it.
The issue is that we have an application pool on IIS who is crashing and recycling several times a day.
We monitored what's happening on event viewer but the error there was not enough detailed.
Example of the 2 errors (2 seconds between them) from windows event viewer :
Error 1 :
Event Id : 1000
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/20/ROOT
Process ID: 14300
Exception: System.ObjectDisposedException
Message: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
StackTrace: at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FtpClient.FtpSocketStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.FtpClient.FtpSocketStream.WriteLine(Encoding encoding, String buf)
at System.Net.FtpClient.FtpClient.Execute(String command)
at System.Net.FtpClient.FtpClient.Disconnect()
at System.Net.FtpClient.FtpClient.Dispose()
at System.Net.FtpClient.FtpClient.Finalize()
Error 2:
Event Id : 1325
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18666, time stamp: 0x58f32841
Exception code: 0xe0434352
Fault offset: 0x00015608
Faulting process id: 0x37dc
Faulting application start time: 0x01d35b56102b1ddf
Faulting application path: C:\windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\windows\SYSTEM32\KERNELBASE.dll
Report Id: b9f70358-c76a-11e7-8111-0cc47a0c4e39
Faulting package full name:
Faulting package-relative application ID:
At this moment, we were able to find which error is making app pool crashing. It concerns accessing disposed object from FtpClient library. But we were still not able to find what in our code is doing such this call.
Then, we decided to install DebugDiag (https://www.microsoft.com/en-us/download/details.aspx?id=49924). A very powerful tool by the way.
We launched DebugDiag for monitoring app pool crashes and we were able to generate a very detailed report (in .mhtml format).
Here is the full call stack :
.NET Call Stack
[[HelperMethodFrame]]
System_ni!System.Net.Sockets.NetworkStream.Write(Byte[], Int32, Int32)+673bde
System.Net.FtpClient.FtpSocketStream.Write(Byte[], Int32, Int32)+33
System.Net.FtpClient.FtpSocketStream.WriteLine(System.Text.Encoding, System.String)+44
System.Net.FtpClient.FtpClient.Execute(System.String)+1c6
System.Net.FtpClient.FtpClient.Disconnect()+57
System.Net.FtpClient.FtpClient.Dispose()+46
System.Net.FtpClient.FtpClient.Finalize()+10
[[DebuggerU2MCatchHandlerFrame]]
[[ContextTransitionFrame]]
[[GCFrame]]
[[DebuggerU2MCatchHandlerFrame]]
Full Call Stack
KERNELBASE!RaiseException+48
clr!RaiseTheExceptionInternalOnly+27c
clr!IL_Throw+141
[[HelperMethodFrame]]
System_ni!System.Net.Sockets.NetworkStream.Write(Byte[], Int32, Int32)+673bde
System.Net.FtpClient.FtpSocketStream.Write(Byte[], Int32, Int32)+33
System.Net.FtpClient.FtpSocketStream.Write(Byte[], Int32, Int32)+33
System.Net.FtpClient.FtpSocketStream.WriteLine(System.Text.Encoding, System.String)+44
System.Net.FtpClient.FtpSocketStream.WriteLine(System.Text.Encoding, System.String)+44
System.Net.FtpClient.FtpClient.Execute(System.String)+1c6
System.Net.FtpClient.FtpClient.Disconnect()+57
System.Net.FtpClient.FtpClient.Dispose()+46
System.Net.FtpClient.FtpClient.Finalize()+10
clr!MethodTable::FastBox+b0
clr!MethodTable::CallFinalizer+139
clr!CallFinalizer+a6
clr!FinalizerThread::FinalizeAllObjects+a6
clr!FinalizerThread::FinalizeAllObjects_Wrapper+14
clr!Thread::DoExtraWorkForFinalizer+1b1
clr!Thread::DoExtraWorkForFinalizer+234
clr!Thread::DoExtraWorkForFinalizer+5f8
[[DebuggerU2MCatchHandlerFrame]]
clr!Thread::DoExtraWorkForFinalizer+137
clr!Thread::DoADCallBack+30f
[[ContextTransitionFrame]]
clr!Thread::DoExtraWorkForFinalizer+19f
clr!FinalizerThread::DoOneFinalization+129
[[GCFrame]]
clr!FinalizerThread::FinalizeAllObjects+a6
clr!FinalizerThread::FinalizerThreadWorker+ed
clr!Thread::DoExtraWorkForFinalizer+1b1
clr!Thread::DoExtraWorkForFinalizer+234
clr!Thread::DoExtraWorkForFinalizer+5f8
[[DebuggerU2MCatchHandlerFrame]]
clr!ManagedThreadBase::FinalizerBase+33
clr!FinalizerThread::FinalizerThreadStart+d4
clr!Thread::intermediateThreadProc+55
kernel32!BaseThreadInitThunk+24
ntdll!__RtlUserThreadStart+2f
ntdll!_RtlUserThreadStart+1b
We were able to find the exception that made the pool crashing but we are still unable to find which part of our code is doing the call.
At the bottom of the call stack, you can noticed that the first event is : ntdll!_RtlUserThreadStart+1b which I really don't know ..
So I would be glad to have some help to solve this 2 questions :
How to find in my code what is creating the bug ?
How can such error making my application pool crashed and recycled ?
Environment :
- IIS 8.5
- Application pool CLR version 4
- Windows Server 2012 R2
The message Cannot access a disposed object means that the garbage collector has disposed your object, which you then try to use.
Are you calling the FtpClient with using, are you doing a .close()?
The error logs aren't actually useful, it would be better if you posted your code instead.
Also, put you code inside try...catch so that you can capture the error rather than leaving it unhandled. Sometimes file transfers fail so you need to be doing something like the following :
try {
using(Stream s = ftpClient.OpenRead()) {
// perform your transfer
}
}
catch(Exception) {
// Typical exceptions here are IOException, SocketException, or a FtpCommandException
}
I have recently set up a Windows 2016 Server. I set up my .net application on the server. However, the moment I access the application, the w3wp crashes and shuts down the application pool by itself.
Event Viewer gives out 2 errors.
1) Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FF8FEDDE899 (00007FF8FEC60000) with exit code 80131506.
2) Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a
Faulting module name: clr.dll, version: 4.6.1637.0, time stamp: 0x586fe714
Exception code: 0xc0000005
Fault offset: 0x000000000017e899
Faulting process id: 0x1bec
Faulting application start time: 0x01d29cca82c75315
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 99612f7d-0679-40b3-aa26-d44ca19c44b8
Faulting package full name:
Faulting package-relative application ID:
Event ID = 1000
the application is built on .net framework 4.6.1. Since the server is Win 2016 it is shipped with framework 4.6.2. I have installed a test application with .net 4.6.2 framework and an application with 4.6.1 framework (Visual Studio) and they run fine. But my main application gives out the above errors.
I have executed svc /SCANNOW which has found some corrupt files and successfully repaired them. But this has not helped. IIS version of the server is 10.0.14393.0.
Any help would be appreciated. Thanks
There could be a hotfix for this RAnand - check out the link to Microsoft support link
I'm having migration issues with a site that used to run on 2003 server 32bit to a windows 2008 r2 server 64 bit. and ASP version 1.1. Iv'e taken all the steps needed (I hope) to ensure it functions correctly. Such as,
downloaded ASP.NET V1.1 onto the server
selected the app pool to run this ASP version and set it to classic mode
ensured that CGI is Allowed on the .NET version
application is running in 32bit mode
But I'm getting the "This Page cannot be displayed" message when navigating to it with the browser. I noticed this error in the event logs -
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00db8960
Faulting process id: 0xcb4
Faulting application start time: 0x01d0b40cf800bfca
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: unknown
Report Id: 35c785b2-2000-11e5-be06-005056b20078
Anyone have ideas on how to rectify this issue?
Thank you.
In IIS, find the ApplicationPool that is executing your web application
Advanced settings
Set "Enable 32 bit application" to true
IISReset
I have a web application build in asp.net 4.5 which intermittently logs the user out.
I was able to get from the Event Viewer the following error, which I think is the error that's causing the user to be logged out:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8
Faulting module name: nlssorting.dll, version: 4.0.30319.17929, time stamp: 0x4ffa55e0
Exception code: 0xc00000fd
Fault offset: 0x00001cda
Faulting process id: 0x358c
Faulting application start time: 0x01cf11fdded81940
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll
Report Id: 4c2ecce0-7df2-11e3-9850-bc764e080abb
I couldn't get much information from the error description, so I would like to see if there is a way to get more information about it, such as what type of exception was thrown by the web application in order to break the process or (ideally) which dll or method threw that error.
Thanks in advance!