Local PDF generation works well, but the same application stoped working on server. Event log says:
Faulting application wkhtmltopdf.exe, version 0.12.2.1, time stamp 0x54bd0265, faulting module MSVCP120.dll, version 6.0.6002.18881, time stamp 0x51da3e00, exception code 0xc0000135, fault offset 0x0006f52f, process id 0x44b8, application start time 0x01d07ce4c087128f.
and
Exception information:
Exception type: IOException
Exception message: The pipe has been ended.
at System.IO.__Error.WinIOError(Int32 errorCode, String
maybeFullPath)
at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
at NReco.PdfGenerator.HtmlToPdfConverter.GeneratePdfInternal(String htmlFilePath, Byte[] inputBytes, String coverHtml, String outputPdfFilePath, Stream outputStream)
I've spent 4 hours wrestling with this today, so I thought i'd post my solution to help anyone else suffering in thr future.
wkhtmltopdf was updated and worked on my local machine.
Pushed the code to the server and nothing.
I eventually tracked this down to needing to install the 32 bit version of C++Runtime from here https://www.microsoft.com/en-GB/download/details.aspx?id=40784
In my case, due to some 3rd party libraries I cannot remove yet, I was forced to run the website in 32 bit mode. As my server is 64 bit, the 32 bit versions were not present. (As my PC is a Dev Machine, I assume that I got all the relevent libraries from MS, hence why I had no local issue.)
Solution is to check accesibility of dll's msvcp120.dll and msvcr120.dll. Local copy of libraries into app bin dir solve my problem.
Related
Jscript Unexpected exceptions
After Windows Server patch Vulnerability (CVE-2019-1367) released in 23. September
Windows Server 2019 (KB4522015) https://support.microsoft.com/en-us/help/4522015/windows-10-update-kb4522015
Windows Server 2016 (KB4522010)
Windows Server 2008 R2 for x64-based Systems Service Pack 1 (KB4522007)
Windows Server 2012 (KB4522007)
Windows Server 2012 R2 (KB4522007)
Updated 07.10.2019 Also "Preview of Monthly Rollup" and "Monthly Rollup" packages are affected and doesn't fix the specific Jscript Workflow issues
Windows Server 2019: KB4516077, KB4524148
Windows Server 2016: KB4516061, KB4524152
Windows Server 2012 R2: KB4516041, KB4524156
in the classic ASP application on several workflow cases are occurring jscript Unexpected errors in server side:
Active Server Pages error 'ASP 0115'
A trappable error (C0000005) occurred in an external object. The script cannot continue running
Active Server Pages error 'ASP 0240'
A ScriptEngine threw exception 'C0000005' in 'IActiveScript::Close()' from 'CActiveScriptEngine::FinalRelease()'.
Patch
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer, aka
'Scripting Engine Memory Corruption Vulnerability'. This CVE ID is
unique from CVE-2019-1221.
https://www.cvedetails.com/cve/CVE-2019-1367/
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer. The
vulnerability could corrupt memory in such a way that an attacker
could run arbitrary code in the context of the current user. An
attacker who successfully exploited the vulnerability could gain the
same user rights as the current user. In a web-based attack scenario,
an attacker could host a specially crafted website that is designed to
exploit the vulnerability through Internet Explorer and then convince
a user to view the website, for example, by sending an email. The
security update addresses the vulnerability by modifying how the
scripting engine handles objects in memory.
https://blog.qualys.com/laws-of-vulnerabilities/2019/09/24/microsoft-releases-out-of-band-security-updates
Patch is said to address issue in Memory management. Doesn't specify what exactly changes, what are the new limitations. But seems that it causes some side-effect failure cases.
Error nature
The errors cannot be handled by regular try-catch approach
The errors causes workflow interruptions
The exception seemingly happens only one time entering the specific workflow, and on repeated web-request for the same routine the code succeeds (Until App-pool restarted).
Sometimes exceptions is entering the workflow first, second or third time.
The exception only happens if IIS ASP Debugging Properties - Enable Server-side Debugging is set to False
Background
Verified that the issue is present on all tested Server instances with the patch. Also isolated the patch by checking the State before and after applying the patch (Server 2012 R2, Server 2016, Windows 10 - 1809)
From Classic ASP Server cannot handle the issue, with try-catch,
General error is returned - Script Error Message
Or if turned off (ASP - Send Errors To Browser) ASP error codes with page where they happen
Event Viewer also registers those errors, but without additional information
Global.asa doesn't offer global error handling, ASP Server object Server.GetLastError() doesn't catch the exception
Explored exceptions with
DebugDiag
Sysinternals Process Monitor
IIS - Failed Request Tracing
Environment
App-Pool: Classic pipeline mode, Enable 32-Bit Applications: True
Application: ASP
ClientL IE 11 Enterprise mode, with ActiveX enabled
Application pool identity is Impersonated in Web request calls
Issues, identified
1 In w3wp__V...__First chance exception 0XC0000005.dmp the assembly
instruction at msvcrt!memcpy+198 ###
in C:\Windows\System32\msvcrt.dll from Microsoft Corporation has caused an access
violation exception (0xC0000005) when trying to read from memory location 0x0000000a on thread 33
Instruction Address
Source
[0x7532a2d8] msvcrt!memcpy+198
[0x6ac17deb] jscript!AString::CopyToBuffer+4b
[0x6ac10524] jscript!AString::ConvertToBSTR+1bb74
[0x6abdf6b7] jscript!PrepareInvoke+277
[0x6abf52df] jscript!InvokeDispatch+8f
[0x6abe2f03] jscript!VAR::InvokeByDispID+523
[0x6abdbde0] jscript!NameTbl::InvokeInternal+270
[0x6abe2b17] jscript!VAR::InvokeByDispID+137
[0x6abe6083] jscript!CScriptRuntime::Run+2db3
...
Followed by - Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b7c2d77] jscript!VarStack::ScavengeRoots+27
[0x6b7c2b89] jscript!GcContext::CollectCore+79
[0x6b7c2af4] jscript!GcContext::Collect+1b
[0x6b7bca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b7a604a] jscript!CSession::Close+18a
[0x6b7a32d9] jscript!COleScript::CloseInternal+13b
[0x6b7a2d36] jscript!COleScript::Close+16
[0x6b8a71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
Not identified the exact line that causes the issue, the FailedRequestTrace last record is assigning string variable from Application Scope xml object attribute. (CurrentStatement return attrib.text)
Similar case - access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b907e09] jscript!AString::CopyToBuffer+69
[0x6b900524] jscript!AString::ConvertToBSTR+1bb74
[0x6b8e49a7] jscript!VAR::ConvertASTRtoBSTR+13
[0x6b8c49e8] jscript!VAR::GetValue+58
[0x6b8e0f34] jscript!ConvertToString+58
[0x6b922fbf] jscript!JsString+4f
[0x6b8d92e6] jscript!NatFncObj::Call+e6
...
Followed by - access violation exception (0xC0000005) when trying to read from memory location 0x004e0049
[0x6b8e2d77] jscript!VarStack::ScavengeRoots+27
[0x6b8e2b89] jscript!GcContext::CollectCore+79
[0x6b8e2af4] jscript!GcContext::Collect+1b
[0x6b8dca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b8c604a] jscript!CSession::Close+18a
[0x6b8c32d9] jscript!COleScript::CloseInternal+13b
[0x6b8c2d36] jscript!COleScript::Close+16
[0x6bfb71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
2 In w3wp__...__Second_Chance_Exception_C0000005.dmp the assembly instruction at asp!CResponseBuffer::Write+3a
in \?\C:\Windows\System32\inetsrv\asp.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000014 on thread 32
[0x6f042e88] asp!CResponseBuffer::Write+3a
[0x6f0452ea] asp!CResponse::WriteSz+4c
[0x6f02dd3b] asp!CErrInfo::LogErrortoBrowser+ff
[0x6f02d4c9] asp!CErrInfo::LogErrortoBrowserWrapper+d7
[0x6f02d047] asp!CErrInfo::LogError+e8
[0x6f02e241] asp!HandleError+116
[0x6f02f009] asp!HandleErrorMissingFilename+df
[0x6f04941b] asp!CActiveScriptEngine::Call+bb
[0x6f030eff] asp!CallScriptFunctionOfEngine+4d
[0x6f02f99f] asp!ExecuteRequest+173
[0x6f02f828] asp!Execute+23d
[0x6f035c6f] asp!CHitObj::ViperAsyncCallback+467
[0x6f05df53] asp!CViperAsyncRequest::OnCall+73
[0x6eefd325] comsvcs!CSTAActivityWork::STAActivityWorkHelper+45
[0x77098346] combase!EnterForCallback+16e [onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2] onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2
[0x7709816d] combase!SwitchForCallback+206 [onecore\com\combase\dcomrem\crossctx.cxx # 1694] onecore\com\combase\dcomrem\crossctx.cxx # 1694
[0x7709bae4] combase!PerformCallback+bc [onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16] onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16
[0x7709b7f9] combase!CObjectContext::InternalContextCallback+119 [onecore\com\combase\dcomrem\context.cxx # 4421 + 1a] onecore\com\combase\dcomrem\context.cxx # 4421 + 1a
[0x77198e66] combase!CObjectContext::DoCallback+26 [onecore\com\combase\dcomrem\context.cxx # 4254] onecore\com\combase\dcomrem\context.cxx # 4254
[0x6eefd015] comsvcs!CSTAActivityWork::DoWork+175
[0x6eeff0e0] comsvcs!CSTAThread::DoWork+26
[0x6eeff599] comsvcs!CSTAThread::ProcessQueueWork+48
[0x6eeff8dd] comsvcs!CSTAThread::WorkerLoop+13d
[0x76577e71] msvcrt!_callthreadstartex+25
[0x76577f31] msvcrt!_threadstartex+61
[0x765f0419] kernel32!BaseThreadInitThunk+19
[0x77d5662d] ntdll!__RtlUserThreadStart+2f
[0x77d565fd] ntdll!_RtlUserThreadStart+1b
...
Most likely comes from writing to log file
ioo_fso = Server.CreateObject("Scripting.FileSystemObject");
...
loo_file = loo_fso.OpenTextFile(ls_filename,8,true);
...
try {
loo_file.WriteLine("[" + str + "]")} catch (ee) {}
Proces-Monitor shows "SHARING VIOLATION" log records for w3wp.exe on accessing the log file
3 Also experienced ASP 0115 right after custom Server custom component creation
var pbkdf2;
try {
pbkdf2 = Server.CreateObject("Pbkdf2");
pbkdf2.hashPassword(ls_newpassword, 100000);
} catch (e) {
addToLogg("Login:CreateObject failed for Pbkdf2, " + e.description);
}
from FailedReqLogFiles logs, but not identified yet in DebugDiag
Questions
I know ASP Jscript is an old, out-phasing technology, but there should be plenty of Enterprise solution still out there, so could be someone else also runs into these issues.
I would expect the Jscript to fall in regular way, so that the error situation could be handled
Has someone else come upon similar situation?
What are the new restrictions on jscript code?
Are there ways to handle these fails in server-side, prior to returning response to client?
Maybe there is some ASP/jscript environment settings, memory management settings, Windows privileges, permissions that can potentially solve the issues?
Solution
As hinted by #Max (below) the last Microsoft KBs fixes the Jscript Workflow issues.
The only comment in KB that reflects jscript is
Addresses an issue with applications and printer drivers that utilize
the Windows JavaScript engine (jscript.dll) for processing print jobs.
So apparently the fix in common jscript handling
The summary of KBs that fix the issue
Server 2019 1809: KB4519338
Server 2019 1903: KB4517389
Server 2016: KB4519998
Server 2012 R2 : KB4520005
Server 2012: KB4520007
Server 2008 R2: KB4519976
Server 2008 (with SP2): KB4520002
Not required to uninstall previous KB updates.
See that previous monthly Rollup (October 3) is no longer present in Window Updates after installing the new one.
While I didn't manage to isolate the main "First chance exception 0xC0000005" from the Workflows:
After applying the October 8 KB the issue 0xC0000005 is no longer there in the previously failing Workflows
The isolated second chance exceptions are also not reproduced, see ntdl Access Violation case, Server Object Creation case
My isolated Regex samples with test data from the failing workflow didn't reproduce the Exception, so I cannot confirm that the regex use directly caused the issues
However VBScript proved not to be affected, as pointed out by #Lee
This appears to be an unfortunate issue in the basket, while fixing memory corruption creating a several ones.
We also experienced these same errors related to CVE-2019-1367 and classic ASP. We narrowed the scope of the errors to a few places we were using JScript instead of VBScript to do JSON conversions, then we narrowed it further to were we were using regex. We worked around the errors by rewriting the functionality that was in JScript code in VBScript.
I found this article that refers to CVE-2019-13670 with a very similar number and very similar wording: Google Chrome could allow a remote attacker to execute arbitrary code on the system, caused by a V8 memory corruption in regex..
CVE-2019-1367 is specific to Internet Explorer and updated C\Windows\system32\JScript.dll. From that, I'm guessing IE's javascript engine and classic ASP JScript engine are both handled by JScript.dll? Wild guess. CVE-2019-13670 is specific to Chrome (which I assume doesn't use JScript.dll), but it mentioned regex and we found our problems to be specific to regex usage in JScript.
My group is also experiencing these issues. Our entire legacy system is written in ASP
with JScript. The KB4522007 update was installed on 2019-09-25, at which point we noticed the errors. In addition to the error mentioned in the original post, we are experiencing other errors as well:
A trappable error (C0000005) occurred in an external object. The script cannot continue running.
The function returned |.
Function expected
Variable uses an Automation type not supported in JScript
Object doesn't support this property or method
These are all errors happening in unmodified files, and were never observed prior to the update. The errors are periodic, and not systematically reproducible...
Removing the KB4522007 updated caused the errors to go away.
it seems the latest updates from Microsoft fixes the problem.
ASP.NET WebAPI2 application on .NET 4.6.2, hosted on IIS on Windows Server 2016. From time to time, there is a lot (hundreds) of requests stuck for hours (despite the fact I have request timeout 60s set) with no CPU usage. So, I took the memory dump of w3wp process, along with sos.dll, clr.dll and mscordacwks.dll and all my project's dlls and pdbs from bin directory from server and used WinDbg as described in many blogs and tutorials. But, in all of them, they are able to directly see CLR stack by calling ~*e !clrstack. I can see CLR stacktrace for some Redis and ApplicationInsights workers, but for all other managed threads I can see only:
OS Thread Id: 0x1124 (3)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
!dumpstack for any of these gives just this:
0:181> !dumpstack
OS Thread Id: 0x1754 (181)
Current frame: ntdll!NtWaitForSingleObject+0x14
Child-SP RetAddr Caller, Callee
000000b942c7f6a0 00007fff33d63acf KERNELBASE!WaitForSingleObjectEx+0x8f, calling ntdll!NtWaitForSingleObject
000000b942c7f740 00007fff253377a6 clr!CLRSemaphore::Wait+0x8a, calling kernel32!WaitForSingleObjectEx
000000b942c7f7b0 00007fff25335331 clr!GCCoop::GCCoop+0xe, calling clr!GetThread
000000b942c7f800 00007fff25337916 clr!ThreadpoolMgr::UnfairSemaphore::Wait+0xf1, calling clr!CLRSemaphore::Wait
000000b942c7f840 00007fff253378b1 clr!ThreadpoolMgr::WorkerThreadStart+0x2d1, calling clr!ThreadpoolMgr::UnfairSemaphore::Wait
000000b942c7f8e0 00007fff253d952f clr!Thread::intermediateThreadProc+0x86
000000b942c7f9e0 00007fff253d950f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
000000b942c7fa20 00007fff37568364 kernel32!BaseThreadInitThunk+0x14, calling ntdll!LdrpDispatchUserCallTarget
000000b942c7fa50 00007fff3773e821 ntdll!RtlUserThreadStart+0x21, calling ntdll!LdrpDispatchUserCallTarget
So I have no idea, where to look for bug in my code.
(here is the full result:
https://gist.github.com/rouen-sk/eff11844557521de367fa9182cb94a82
and here is the results of !threads:
https://gist.github.com/rouen-sk/b61cba97a4d8300c08d6a8808c4bff6e)
What can I do? Google search for GetFrameContext failed gives nothing helpful.
As mentioned, this is not trivial, however you can find a case study of similar problem here: https://blogs.msdn.microsoft.com/rodneyviana/2015/03/27/the-case-of-the-non-responsive-mvc-web-application/
In a nutshell:
Download NetExt. It is the zip file here:
https://github.com/rodneyviana/netext/tree/master/Binaries
Open your dump and load NetExt
Run !windex to index the heap
Run !whttp -order -running to see a list of running requests
If the requests contains thread number you can go to the thread to see what is happening
If the requests contains --- instead of thread number, they are waiting a thread and this is a sign that some throttling is happening
If it is a WCF service, run !wservice to see the services
Run !wruntime to see runtime information
Run !wapppool to see Application Pool information
Run !wdae to list all errors
... And so it goes. When you do this again and again you will be able to spot issues easily
I spend a lot of time to search answer but I didn't found it. We have console application (exe) with C1 Report Component for automated printing task. This application is executed from cmd file and cmd file is called by cmdexec from SQL Server 2014 every minute. It's work very fine but there is exceptiton GDI+ Initialize - System.Drawing.SafeNativeMethods+Gdip.Initialize aproximetelly montly. There is no chance to solve this so we must restart server.
I tried to create cmd file with echo command. It was worked fine and I got output from echo. So I created application with Console.Writeline("Hello world") only. It was worked fine. Then I add reference to System.Drawing. Nothing happend - ouput was produced. Then I created System.Drawing.Bitmap object and I got GDI+ Initialize exception.
Job on SQL server run as SQLRUNSERVICES and it is in Administrator group. But when I log as SQLRUNSERVICES and launch our application in command line, application works fine without exception. When I log as another user there is no problem with this. Only with SQLRUNSERVICES throught SQL Proxy.
I suppose there must be something wrong with windows session. This application starts, prints some reports and finishes. So I suppose when application finishes memory will be free and GDI object will be free so. I check if we have call dispose method for each object working with GDI.
Can I have some chance to see GDI objects used by session? Or what could cause this exception? I cannot use task manager because when application is not running there is nothing to see.
Thank you for replies. It is very stresfull problem for us because it is going repeatedly every month after restart server.
We created testing application to print most quickly and we got this exception for one day. We call Dispose method for C1Report ofcourse. We will try to use using section. But it looks like C1 error. Test application printed about 65000 count of document and then crashed by GDI+ Initialization Exception. There is stack:
System.TypeInitializationException: The type initializer for 'Gdip' threw an exception. ---> System.Runtime.InteropServices.ExternalException: A generic error occurred in GDI+.
at System.Drawing.SafeNativeMethods.Gdip.Initialize()
at System.Drawing.SafeNativeMethods.Gdip..cctor()
--- End of inner exception stack trace ---
at System.Drawing.SafeNativeMethods.Gdip.GdipCreateBitmapFromScan0(Int32 width, Int32 height, Int32 stride, Int32 format, HandleRef scan0, IntPtr& bitmap)
at System.Drawing.Bitmap..ctor(Int32 width, Int32 height, PixelFormat format)
at System.Drawing.Bitmap..ctor(Int32 width, Int32 height)
at C1.C1Preview.GraphicsHolder.FromBitmap()
at C1.C1Preview.GraphicsHolder.FromScreen()
at C1.C1Preview.C1PrintDocument.SelectBestMeasurementDevice(Boolean showWarning, MeasurementDeviceEnum& measurementDevice, String& measurementPrinterName, GraphicsHolder& measurementGraphicsHolder)
at C1.C1Preview.C1PrintDocument.#fFe()
at C1.C1Preview.C1PrintDocument.SetCreationDevice(MeasurementDeviceEnum creationDevice, String creationPrinterName)
at C1.C1Report.Layout..ctor(C1Report report)
at C1.C1Report.C1Report.Initialize()
at C1.C1Report.C1Report..ctor()
I have a website which has been up and running absolutely fine for about 8 months now. It's running .net 4 intergrated mode.
Recently i've started to get some "random" w3wp.exe crashes, and after 5 of them, IIS rapid fail protection kicks in and I have to manually login to the server and start the application pool again.
Here's what the event viewer says for the Error:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: nlssorting.dll, version: 4.0.30319.296, time stamp: 0x504835c7
Exception code: 0xc00000fd
Fault offset: 0x000000000000191f
Faulting process id: 0x1998
Faulting application start time: 0x01ce6e6b9b80c949
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
Report Id: d9cf3164-da5e-11e2-8cc5-f46d0440f6d5
Straight after the crashes, i get an "Information" log in the event viewer which at the bottom gives me the location of a .wer file.
This is what the .wer files contains:
Version=1
EventType=APPCRASH
EventTime=130162847687759734
ReportType=2
Consent=1
ReportIdentifier=d7c5e520-da5e-11e2-8cc5-f46d0440f6d5
IntegratorReportIdentifier=d7c5e51f-da5e-11e2-8cc5-f46d0440f6d5
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=w3wp.exe
Sig[1].Name=Application Version
Sig[1].Value=7.5.7601.17514
Sig[2].Name=Application Timestamp
Sig[2].Value=4ce7afa2
Sig[3].Name=Fault Module Name
Sig[3].Value=nlssorting.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=4.0.30319.296
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=504835c7
Sig[6].Name=Exception Code
Sig[6].Value=c00000fd
Sig[7].Name=Exception Offset
Sig[7].Value=000000000000197d
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.1296.17
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=6141
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=61419d6dee6cf74b8ac2b00b4c3b3373
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=c19b
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=c19b8acf029a3088171b1f5f3dd9dc77
UI[2]=c:\windows\system32\inetsrv\w3wp.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=IIS Worker Process stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[5]=C:\Windows\system32\msvcrt.dll
LoadedModule[6]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[7]=C:\Windows\system32\RPCRT4.dll
LoadedModule[8]=C:\Windows\system32\pcwum.DLL
LoadedModule[9]=C:\Windows\system32\USER32.dll
LoadedModule[10]=C:\Windows\system32\GDI32.dll
LoadedModule[11]=C:\Windows\system32\LPK.dll
LoadedModule[12]=C:\Windows\system32\USP10.dll
LoadedModule[13]=C:\Windows\system32\ole32.dll
LoadedModule[14]=c:\windows\system32\inetsrv\IISUTIL.dll
LoadedModule[15]=C:\Windows\system32\IMM32.DLL
LoadedModule[16]=C:\Windows\system32\MSCTF.dll
LoadedModule[17]=C:\Windows\system32\CRYPTBASE.dll
LoadedModule[18]=C:\Windows\system32\ntmarta.dll
LoadedModule[19]=C:\Windows\system32\WLDAP32.dll
LoadedModule[20]=c:\windows\system32\inetsrv\w3wphost.dll
LoadedModule[21]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll
LoadedModule[23]=C:\Windows\system32\CRYPT32.dll
LoadedModule[24]=C:\Windows\system32\MSASN1.dll
LoadedModule[25]=C:\Windows\system32\XmlLite.dll
LoadedModule[26]=C:\Windows\system32\ktmw32.dll
LoadedModule[27]=c:\windows\system32\inetsrv\IISRES.DLL
LoadedModule[28]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[29]=C:\Windows\system32\rsaenh.dll
LoadedModule[30]=C:\Windows\system32\mscoree.dll
LoadedModule[31]=C:\Windows\system32\CLBCatQ.DLL
LoadedModule[32]=C:\Windows\system32\mlang.dll
LoadedModule[33]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine4.dll
LoadedModule[34]=C:\Windows\system32\MSVCR100_CLR0400.dll
LoadedModule[35]=C:\Windows\system32\USERENV.dll
LoadedModule[36]=C:\Windows\system32\profapi.dll
LoadedModule[37]=C:\Windows\system32\PSAPI.DLL
LoadedModule[38]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[39]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[40]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[41]=C:\Windows\system32\inetsrv\iiscore.dll
LoadedModule[42]=c:\windows\system32\inetsrv\W3TP.dll
LoadedModule[43]=c:\windows\system32\inetsrv\w3dt.dll
LoadedModule[44]=C:\Windows\system32\HTTPAPI.dll
LoadedModule[45]=C:\Windows\system32\slc.dll
LoadedModule[46]=C:\Windows\system32\WS2_32.dll
LoadedModule[47]=C:\Windows\system32\NSI.dll
LoadedModule[48]=C:\Windows\system32\Normaliz.dll
LoadedModule[49]=C:\Windows\system32\faultrep.dll
LoadedModule[50]=C:\Windows\system32\Secur32.dll
LoadedModule[51]=C:\Windows\system32\SSPICLI.DLL
LoadedModule[52]=C:\Windows\system32\NLAapi.dll
LoadedModule[53]=C:\Windows\system32\napinsp.dll
LoadedModule[54]=C:\Windows\System32\mswsock.dll
LoadedModule[55]=C:\Windows\system32\DNSAPI.dll
LoadedModule[56]=C:\Windows\System32\winrnr.dll
LoadedModule[57]=C:\Windows\System32\wshtcpip.dll
LoadedModule[58]=C:\Windows\System32\wship6.dll
LoadedModule[59]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[60]=C:\Windows\system32\WINNSI.DLL
LoadedModule[61]=C:\Windows\system32\rasadhlp.dll
LoadedModule[62]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[63]=C:\Windows\System32\inetsrv\cachuri.dll
LoadedModule[64]=C:\Windows\System32\inetsrv\cachfile.dll
LoadedModule[65]=C:\Windows\System32\inetsrv\cachtokn.dll
LoadedModule[66]=C:\Windows\System32\inetsrv\cachhttp.dll
LoadedModule[67]=C:\Windows\System32\inetsrv\compdyn.dll
LoadedModule[68]=C:\Windows\System32\inetsrv\compstat.dll
LoadedModule[69]=C:\Windows\System32\inetsrv\defdoc.dll
LoadedModule[70]=C:\Windows\System32\inetsrv\protsup.dll
LoadedModule[71]=C:\Windows\System32\inetsrv\redirect.dll
LoadedModule[72]=C:\Windows\System32\inetsrv\static.dll
LoadedModule[73]=C:\Windows\System32\inetsrv\authanon.dll
LoadedModule[74]=C:\Windows\System32\inetsrv\authbas.dll
LoadedModule[75]=C:\Windows\System32\inetsrv\authsspi.dll
LoadedModule[76]=C:\Windows\system32\NETAPI32.dll
LoadedModule[77]=C:\Windows\system32\netutils.dll
LoadedModule[78]=C:\Windows\system32\srvcli.dll
LoadedModule[79]=C:\Windows\system32\wkscli.dll
LoadedModule[80]=C:\Windows\System32\inetsrv\iprestr.dll
LoadedModule[81]=C:\Windows\System32\inetsrv\modrqflt.dll
LoadedModule[82]=C:\Windows\System32\inetsrv\logcust.dll
LoadedModule[83]=C:\Windows\System32\inetsrv\custerr.dll
LoadedModule[84]=C:\Windows\System32\inetsrv\loghttp.dll
LoadedModule[85]=C:\Windows\System32\inetsrv\isapi.dll
LoadedModule[86]=C:\Windows\System32\inetsrv\filter.dll
LoadedModule[87]=C:\Windows\System32\inetsrv\validcfg.dll
LoadedModule[88]=c:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_filter.dll
LoadedModule[89]=C:\Windows\system32\inetsrv\wbhst_pm.dll
LoadedModule[90]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine.dll
LoadedModule[91]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\4f52500ab48877b85e71430f4f46670f\mscorlib.ni.dll
LoadedModule[92]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
LoadedModule[93]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\a91f32875cb3ba779f1b3ceff1690251\System.ni.dll
LoadedModule[94]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\0a8d99339ffe6b25debb8f8201c27664\System.Core.ni.dll
LoadedModule[95]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web\5b905bd7b71f9fd6bea2d05cc1ae85f8\System.Web.ni.dll
LoadedModule[96]=C:\Windows\system32\sxs.dll
LoadedModule[97]=C:\Windows\system32\RpcRtRemote.dll
LoadedModule[98]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Configuration\fa65f89fd682c459fc5e7bcbd0418317\System.Configuration.ni.dll
LoadedModule[99]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\f4afb233f160b8e55aad4660e45b374c\System.Xml.ni.dll
LoadedModule[100]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[101]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.Build.Uti#\14e16d61fae3cd1d9a1fa79b789f8438\Microsoft.Build.Utilities.v4.0.ni.dll
LoadedModule[102]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Cach#\8fdbe304abab0631b8a4310b35f3d93a\System.Runtime.Caching.ni.dll
LoadedModule[103]=C:\Windows\system32\shfolder.dll
LoadedModule[104]=C:\Windows\system32\SHELL32.dll
LoadedModule[105]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.JScript\85204dde340780329b569b025e249c23\Microsoft.JScript.ni.dll
LoadedModule[106]=C:\Windows\system32\version.dll
LoadedModule[107]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_Code.cgixlnxh.dll
LoadedModule[108]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data.Linq\feaa494ad67542d2060b31b9eeb6458b\System.Data.Linq.ni.dll
LoadedModule[109]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data\b928128fca867546a858a1a39240d85c\System.Data.ni.dll
LoadedModule[110]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
LoadedModule[111]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\595a888a\f26c0653_7f81cd01\HtmlAgilityPack.dll
LoadedModule[112]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Drawing\5ae853f556290da9399b15b3619f7e15\System.Drawing.ni.dll
LoadedModule[113]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\85ba5013\f0c8f388_706bce01\TweetSharp.dll
LoadedModule[114]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web.Extensio#\0180a2d993d2a9699cf07f7163524fff\System.Web.Extensions.ni.dll
LoadedModule[115]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Transactions\7b2099a1386e38ff198a51939304ce6e\System.Transactions.ni.dll
LoadedModule[116]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Transactions\v4.0_4.0.0.0__b77a5c561934e089\System.Transactions.dll
LoadedModule[117]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_global.asax.yxdky-qn.dll
LoadedModule[118]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel#\7a5a5ff4a0b3bb4ba4bcc13166918e36\System.ServiceModel.Activation.ni.dll
LoadedModule[119]=C:\Windows\system32\bcrypt.dll
LoadedModule[120]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Dura#\799274e49455d0fe4ca563f42143bef2\System.Runtime.DurableInstancing.ni.dll
LoadedModule[121]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Numerics\a66416296451fe6d2d8a5506ca41b23d\System.Numerics.ni.dll
LoadedModule[122]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel\15834d73d2846fc01ed54488ccfff5c8\System.ServiceModel.ni.dll
LoadedModule[123]=C:\Windows\assembly\NativeImages_v4.0.30319_64\SMDiagnostics\31f93b6be386908ff2727bcd825de0ca\SMDiagnostics.ni.dll
LoadedModule[124]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xaml.Hosting\cf8401f4952deb5303e0d7fd459ce530\System.Xaml.Hosting.ni.dll
LoadedModule[125]=C:\Windows\system32\inetsrv\gzip.dll
LoadedModule[126]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\3d63b311\fe7c9b8a_706bce01\Hammock.ClientProfile.dll
LoadedModule[127]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\6a128bd2\c184e08a_706bce01\Newtonsoft.Json.dll
LoadedModule[128]=C:\Windows\system32\rasapi32.dll
LoadedModule[129]=C:\Windows\system32\rasman.dll
LoadedModule[130]=C:\Windows\system32\rtutils.dll
LoadedModule[131]=C:\Windows\system32\winhttp.dll
LoadedModule[132]=C:\Windows\system32\webio.dll
LoadedModule[133]=C:\Windows\system32\credssp.dll
LoadedModule[134]=C:\Windows\system32\dhcpcsvc6.DLL
LoadedModule[135]=C:\Windows\system32\dhcpcsvc.DLL
LoadedModule[136]=C:\Windows\system32\security.dll
LoadedModule[137]=C:\Windows\system32\schannel.DLL
LoadedModule[138]=C:\Windows\system32\ncrypt.dll
LoadedModule[139]=C:\Windows\system32\bcryptprimitives.dll
LoadedModule[140]=C:\Windows\system32\GPAPI.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=IIS Worker Process
AppPath=c:\windows\system32\inetsrv\w3wp.exe
That nlssorting.dll seems to crop up a lot but I can't seem to find anything online related. The only thing I can find which matches my error is here, but that doesn't really help me.
I'm completely stumped as to where to go from here to fix this. Here's what I've tried:
Loading up IIS log files and trying every request from about 30 minutes before a crash, and none of the pages cause any errors.
Searching my code for any recursion which might cause a stackoverflow, but there isn't any
trawling online for ANYTHING that might help
Has anyone else ever had any problems with nlssorting.dll ? Can i get some more information from the .wer file that might help me pin point where this is happening?
Thanks in advance for any help!
UPDATE
I was using a 3rd party DLL, which was causing a stack overflow exception (0xc00000fd)
After more investigation, it was only happening after a certain chain of events happened - hence the "random" in the title. Removing the DLL fixed the problem.
We had the same problem with one of our sites. Using SVN we tracked it down to a method that was scaning for images within a folder.
I modified the code as follows:
Checking array length of scan results to be > 0 instead of == 1
Adding CultureInfo.InvariantCulture to all Int32.ToString() calls
After this we no longer experienced the error. The exact reason is still unknown.
I believe that none of the above points should make a difference in our environment. I believe that the problem could have been, people modifying image files and folders while the image scanning method was called.
I hope this helps somebody.
For anyone who's curious, this is a PITA to debug. Here are three reasons rumored for this to happen:
(1) Stack overflows, as in the original post.
(2) Too much CPU / memory usage, which becomes obvious and rapid fail protection closes the process.
(3) Unable to respond to pings / requests due to application hogging resources, but in a way that rapid fail protection deems appropriate to end the process, not explicitly because of either (1) or (2).
Our solution was to add manual log tracing in the production environment until we eventually found recursion which was leading the application to be stopped by reliability services (for inability to respond to pings, or process randomly crashing) rather than throwing an in-application exception.
I had the issue where w3p would throw an unhandled error as soon I spun up the site\api URI from the web browser, then it would crash.
I was able to pinpoint what part of my code was causing it in my case it was in the Owin Startup class and I was reading some configuration records from a database but prior to that it gets the connection string from a configuration file outside of the web app directory.
I checked the ownership of the folder it showed my account but apparently the subfolders were not owned by me so I set ownership to me again, clicked OK to allow permissions to traverse the objects and voila bye bye w3p error and the API loaded.
So in my case it was an access denied error on the folder \ file that contained the connection string.
I am working on this WCF-WPF app. I am frequently getting this error on an asynchronous XamDataGrid. Trying to make a WCF call and populate the Grid. I initially thought this is something to do with the large amounts of data returned from WCF and I made the call return data is much small pieces, by calling only for a category only. This error comes up randomly, not always on the same set of data. Reproducible enough in 15-20 tries.
I am running Windows XP (32bit), Dual Core, with 4GB of ram. When this exception is raised, client machine only uses about 2GB of RAM, and on the server W3WP is using only 800MB (of 6GB, 3 cores. Total memory use on server is ~2GB).
This is only happening on XP machines. Does not get this error on Windows 7 box.
Please guide me how to resolve this issue.
Thanks in advance
Event Viewer Logs this Message:
Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 5000
Date: 10/13/2010
Time: 10:50:07 AM
User: N/A
Computer: COMP-DC7800
Description:
EventType clr20r3, P1 appname.exe, P2 2.0.0.21872, P3 4cb0a1b1, P4 mscorlib, P5 2.0.0.0, P6 492b834a, P7 35df, P8 45, P9 system.outofmemoryexception, P10 NIL.
Below id the Exception details:
System.OutOfMemoryException was unhandled
Message="Exception of type 'System.OutOfMemoryException' was thrown."
Source="mscorlib"
StackTrace:
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Xml.XmlMtomReader.MimePart.GetBuffer(Int32 maxBuffer, Int32& remaining)
at System.Xml.XmlMtomReader.Initialize(Stream stream, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize)
at System.Xml.XmlMtomReader.SetInput(Stream stream, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.Xml.XmlMtomReader.SetInput(Byte[] buffer, Int32 offset, Int32 count, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.ServiceModel.Channels.MtomMessageEncoder.MtomBufferedMessageData.TakeXmlReader()
at System.ServiceModel.Channels.BufferedMessageData.DoTakeXmlReader()
at System.ServiceModel.Channels.BufferedMessageData.GetMessageReader()
at System.ServiceModel.Channels.MessageHeaders.GetBufferedMessageHeaderReaderAtHeaderContents(IBufferedMessageData bufferedMessageData)
at System.ServiceModel.Channels.MessageHeaders.GetBufferedMessageHeaderReader(IBufferedMessageData bufferedMessageData, Int32 bufferedMessageHeaderIndex)
at System.ServiceModel.Channels.MessageHeaders.GetReaderAtHeader(Int32 headerIndex)
at System.ServiceModel.Channels.WsrmMessageInfo.Get(MessageVersion messageVersion, ReliableMessagingVersion reliableMessagingVersion, IChannel channel, ISession session, Message message, Boolean csrOnly)
at System.ServiceModel.Channels.ReliableDuplexSessionChannel.HandleReceiveComplete(IAsyncResult result)
at System.ServiceModel.Channels.ReliableDuplexSessionChannel.OnReceiveCompletedStatic(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputComplete(IAsyncResult result)
at System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputCompleteStatic(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.InputQueue`1.AsyncQueueReader.Set(Item item)
at System.ServiceModel.Channels.InputQueue`1.Dispatch()
at System.ServiceModel.Channels.InputQueue`1.OnDispatchCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
OutOfMemory exceptions can happen for a lot of reasons in general, and in web apps in particular.
First, if you're running on IIS, there are settings specific to IIS to limit memory, see here: http://blogs.msdn.com/b/pfedev/archive/2009/01/22/memory-based-recycling-in-iis-6-0.aspx
Depending on your configuration, pools, etc., these limits may not apply to .NET web applications.
Now, in ASP.NET, there is also a setting memoryLimit in the machine.config file that specifies the maximum allowed memory size, as a percentage of total system memory, that the worker process can consume before ASP.NET launches a new process and reassigns existing requests.
The default is 60%, which is equal to 1.2G on a 2G machine.
However,... whatever you configure, 800M is the practical limit for ASP.NET apps on a 32-bit machine, see a good explanation here: Understanding ASP.Net memory
The solution to avoid this is to change the way your system works and break it in smaller pieces.
That's why by default, WCF has been carefully configured with so many limits (see here for an extensive list: http://weblogs.asp.net/paolopia/archive/2008/03/23/wcf-configuration-default-limits-concurrency-and-scalability.aspx) ... which almost every developer tends to change and max-out as soon as he discovers it :-)
You won't be able to utilise that 4 GB of RAM with a 32 bit OS.
http://chris.pirillo.com/32-bit-windows-and-4gb-of-ram/
If your application is consuming more and more memory as you use it, you may want to look at your memory management. Look at implementing the IDisposable interface for classes with unmanaged resources. Also, avoid keeping references to objects once you've finished using them. Events are particularly bad for this.
How do events cause memory leaks in C# and how do Weak References help mitigate that?
You could also look at streaming to a temporary file instead of streaming to memory.
Feel Relieved to find the answer that Microsoft Screwed it up all... :). that blame almost saved my Job... Here is the story..
WCF has a default message size of 64kb. One cannot find the reason for this on MSDN.. and so when we need to change the max message size to whatever we want. The problem is that the threshold for LOH is 85000b. If the message size is more than that, the object will be placed in LOH, your profiler will show that the objects have collected.. but when you see the LOH size in Sysinternals Process Explorer, Process properties window, you still see that the memory using is increasing. At around 800MB, our mighty Dr Watson comes up and kick of this application. Apparently this is a known issue with WCF & .net 2.0
Issue still exists in .net 4.0 but Microsoft released a quickfix. Sysinternals Process explorer is the tool which helped diagnose this issue. See Screenshot