ASP.NET WebAPI2 application on .NET 4.6.2, hosted on IIS on Windows Server 2016. From time to time, there is a lot (hundreds) of requests stuck for hours (despite the fact I have request timeout 60s set) with no CPU usage. So, I took the memory dump of w3wp process, along with sos.dll, clr.dll and mscordacwks.dll and all my project's dlls and pdbs from bin directory from server and used WinDbg as described in many blogs and tutorials. But, in all of them, they are able to directly see CLR stack by calling ~*e !clrstack. I can see CLR stacktrace for some Redis and ApplicationInsights workers, but for all other managed threads I can see only:
OS Thread Id: 0x1124 (3)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
!dumpstack for any of these gives just this:
0:181> !dumpstack
OS Thread Id: 0x1754 (181)
Current frame: ntdll!NtWaitForSingleObject+0x14
Child-SP RetAddr Caller, Callee
000000b942c7f6a0 00007fff33d63acf KERNELBASE!WaitForSingleObjectEx+0x8f, calling ntdll!NtWaitForSingleObject
000000b942c7f740 00007fff253377a6 clr!CLRSemaphore::Wait+0x8a, calling kernel32!WaitForSingleObjectEx
000000b942c7f7b0 00007fff25335331 clr!GCCoop::GCCoop+0xe, calling clr!GetThread
000000b942c7f800 00007fff25337916 clr!ThreadpoolMgr::UnfairSemaphore::Wait+0xf1, calling clr!CLRSemaphore::Wait
000000b942c7f840 00007fff253378b1 clr!ThreadpoolMgr::WorkerThreadStart+0x2d1, calling clr!ThreadpoolMgr::UnfairSemaphore::Wait
000000b942c7f8e0 00007fff253d952f clr!Thread::intermediateThreadProc+0x86
000000b942c7f9e0 00007fff253d950f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
000000b942c7fa20 00007fff37568364 kernel32!BaseThreadInitThunk+0x14, calling ntdll!LdrpDispatchUserCallTarget
000000b942c7fa50 00007fff3773e821 ntdll!RtlUserThreadStart+0x21, calling ntdll!LdrpDispatchUserCallTarget
So I have no idea, where to look for bug in my code.
(here is the full result:
https://gist.github.com/rouen-sk/eff11844557521de367fa9182cb94a82
and here is the results of !threads:
https://gist.github.com/rouen-sk/b61cba97a4d8300c08d6a8808c4bff6e)
What can I do? Google search for GetFrameContext failed gives nothing helpful.
As mentioned, this is not trivial, however you can find a case study of similar problem here: https://blogs.msdn.microsoft.com/rodneyviana/2015/03/27/the-case-of-the-non-responsive-mvc-web-application/
In a nutshell:
Download NetExt. It is the zip file here:
https://github.com/rodneyviana/netext/tree/master/Binaries
Open your dump and load NetExt
Run !windex to index the heap
Run !whttp -order -running to see a list of running requests
If the requests contains thread number you can go to the thread to see what is happening
If the requests contains --- instead of thread number, they are waiting a thread and this is a sign that some throttling is happening
If it is a WCF service, run !wservice to see the services
Run !wruntime to see runtime information
Run !wapppool to see Application Pool information
Run !wdae to list all errors
... And so it goes. When you do this again and again you will be able to spot issues easily
Related
We have a clone of draw.io. We have made some modifications to the UI for increased functionality. This works on localhost. We then did the Google Drive authentication process. But it was done on version 12.1.7 of draw.io. That integrated with google drive, except when re-opening a previously saved file.[Error with this version discussed here:https://stackoverflow.com/questions/62889536/no-access-control-allow-origin-header-is-present-on-the-requested-resource]
So, we ported our changes to the recent version of draw.io ,version 13.4.5 (https://github.com/jgraph/drawio/releases/tag/v13.4.5). Now, we get a different error. When trying to authenticate <screen shot of pop-up just before error> we get:
Login attempts failed.Please try again later <screenshot of error message>. When we look at developer tools, we see this
HTTP Status 500 - Can't make API call memcache.Set in a thread that is neither the original request thread nor a thread created by ThreadManager
type Exception report
message Can't make API call memcache.Set in a thread that is neither the original request thread nor a thread created by ThreadManager
description The server encountered an internal error that prevented it from fulfilling this request.
exception
com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call memcache.Set in a thread that is neither the original request thread nor a thread created by ThreadManager
com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:800)
com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:175)
com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:172)
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:26)
com.google.appengine.api.memcache.MemcacheServiceImpl.put(MemcacheServiceImpl.java:69)
com.google.appengine.api.memcache.stdimpl.GCache.put(GCache.java:157)
com.mxgraph.online.AbsAuthServlet.doGet(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.32 (Ubuntu) logs.
Notice the line in <com.mxgraph.online.AbsAuthServlet>. It says “doGet” is throwing error. I am new to servlets and don’t know how to proceed. I did not find “doGet” in the file AbsAuthServlet.java .
Any help to solve this issue will be great.
Jscript Unexpected exceptions
After Windows Server patch Vulnerability (CVE-2019-1367) released in 23. September
Windows Server 2019 (KB4522015) https://support.microsoft.com/en-us/help/4522015/windows-10-update-kb4522015
Windows Server 2016 (KB4522010)
Windows Server 2008 R2 for x64-based Systems Service Pack 1 (KB4522007)
Windows Server 2012 (KB4522007)
Windows Server 2012 R2 (KB4522007)
Updated 07.10.2019 Also "Preview of Monthly Rollup" and "Monthly Rollup" packages are affected and doesn't fix the specific Jscript Workflow issues
Windows Server 2019: KB4516077, KB4524148
Windows Server 2016: KB4516061, KB4524152
Windows Server 2012 R2: KB4516041, KB4524156
in the classic ASP application on several workflow cases are occurring jscript Unexpected errors in server side:
Active Server Pages error 'ASP 0115'
A trappable error (C0000005) occurred in an external object. The script cannot continue running
Active Server Pages error 'ASP 0240'
A ScriptEngine threw exception 'C0000005' in 'IActiveScript::Close()' from 'CActiveScriptEngine::FinalRelease()'.
Patch
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer, aka
'Scripting Engine Memory Corruption Vulnerability'. This CVE ID is
unique from CVE-2019-1221.
https://www.cvedetails.com/cve/CVE-2019-1367/
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer. The
vulnerability could corrupt memory in such a way that an attacker
could run arbitrary code in the context of the current user. An
attacker who successfully exploited the vulnerability could gain the
same user rights as the current user. In a web-based attack scenario,
an attacker could host a specially crafted website that is designed to
exploit the vulnerability through Internet Explorer and then convince
a user to view the website, for example, by sending an email. The
security update addresses the vulnerability by modifying how the
scripting engine handles objects in memory.
https://blog.qualys.com/laws-of-vulnerabilities/2019/09/24/microsoft-releases-out-of-band-security-updates
Patch is said to address issue in Memory management. Doesn't specify what exactly changes, what are the new limitations. But seems that it causes some side-effect failure cases.
Error nature
The errors cannot be handled by regular try-catch approach
The errors causes workflow interruptions
The exception seemingly happens only one time entering the specific workflow, and on repeated web-request for the same routine the code succeeds (Until App-pool restarted).
Sometimes exceptions is entering the workflow first, second or third time.
The exception only happens if IIS ASP Debugging Properties - Enable Server-side Debugging is set to False
Background
Verified that the issue is present on all tested Server instances with the patch. Also isolated the patch by checking the State before and after applying the patch (Server 2012 R2, Server 2016, Windows 10 - 1809)
From Classic ASP Server cannot handle the issue, with try-catch,
General error is returned - Script Error Message
Or if turned off (ASP - Send Errors To Browser) ASP error codes with page where they happen
Event Viewer also registers those errors, but without additional information
Global.asa doesn't offer global error handling, ASP Server object Server.GetLastError() doesn't catch the exception
Explored exceptions with
DebugDiag
Sysinternals Process Monitor
IIS - Failed Request Tracing
Environment
App-Pool: Classic pipeline mode, Enable 32-Bit Applications: True
Application: ASP
ClientL IE 11 Enterprise mode, with ActiveX enabled
Application pool identity is Impersonated in Web request calls
Issues, identified
1 In w3wp__V...__First chance exception 0XC0000005.dmp the assembly
instruction at msvcrt!memcpy+198 ###
in C:\Windows\System32\msvcrt.dll from Microsoft Corporation has caused an access
violation exception (0xC0000005) when trying to read from memory location 0x0000000a on thread 33
Instruction Address
Source
[0x7532a2d8] msvcrt!memcpy+198
[0x6ac17deb] jscript!AString::CopyToBuffer+4b
[0x6ac10524] jscript!AString::ConvertToBSTR+1bb74
[0x6abdf6b7] jscript!PrepareInvoke+277
[0x6abf52df] jscript!InvokeDispatch+8f
[0x6abe2f03] jscript!VAR::InvokeByDispID+523
[0x6abdbde0] jscript!NameTbl::InvokeInternal+270
[0x6abe2b17] jscript!VAR::InvokeByDispID+137
[0x6abe6083] jscript!CScriptRuntime::Run+2db3
...
Followed by - Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b7c2d77] jscript!VarStack::ScavengeRoots+27
[0x6b7c2b89] jscript!GcContext::CollectCore+79
[0x6b7c2af4] jscript!GcContext::Collect+1b
[0x6b7bca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b7a604a] jscript!CSession::Close+18a
[0x6b7a32d9] jscript!COleScript::CloseInternal+13b
[0x6b7a2d36] jscript!COleScript::Close+16
[0x6b8a71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
Not identified the exact line that causes the issue, the FailedRequestTrace last record is assigning string variable from Application Scope xml object attribute. (CurrentStatement return attrib.text)
Similar case - access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b907e09] jscript!AString::CopyToBuffer+69
[0x6b900524] jscript!AString::ConvertToBSTR+1bb74
[0x6b8e49a7] jscript!VAR::ConvertASTRtoBSTR+13
[0x6b8c49e8] jscript!VAR::GetValue+58
[0x6b8e0f34] jscript!ConvertToString+58
[0x6b922fbf] jscript!JsString+4f
[0x6b8d92e6] jscript!NatFncObj::Call+e6
...
Followed by - access violation exception (0xC0000005) when trying to read from memory location 0x004e0049
[0x6b8e2d77] jscript!VarStack::ScavengeRoots+27
[0x6b8e2b89] jscript!GcContext::CollectCore+79
[0x6b8e2af4] jscript!GcContext::Collect+1b
[0x6b8dca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b8c604a] jscript!CSession::Close+18a
[0x6b8c32d9] jscript!COleScript::CloseInternal+13b
[0x6b8c2d36] jscript!COleScript::Close+16
[0x6bfb71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
2 In w3wp__...__Second_Chance_Exception_C0000005.dmp the assembly instruction at asp!CResponseBuffer::Write+3a
in \?\C:\Windows\System32\inetsrv\asp.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000014 on thread 32
[0x6f042e88] asp!CResponseBuffer::Write+3a
[0x6f0452ea] asp!CResponse::WriteSz+4c
[0x6f02dd3b] asp!CErrInfo::LogErrortoBrowser+ff
[0x6f02d4c9] asp!CErrInfo::LogErrortoBrowserWrapper+d7
[0x6f02d047] asp!CErrInfo::LogError+e8
[0x6f02e241] asp!HandleError+116
[0x6f02f009] asp!HandleErrorMissingFilename+df
[0x6f04941b] asp!CActiveScriptEngine::Call+bb
[0x6f030eff] asp!CallScriptFunctionOfEngine+4d
[0x6f02f99f] asp!ExecuteRequest+173
[0x6f02f828] asp!Execute+23d
[0x6f035c6f] asp!CHitObj::ViperAsyncCallback+467
[0x6f05df53] asp!CViperAsyncRequest::OnCall+73
[0x6eefd325] comsvcs!CSTAActivityWork::STAActivityWorkHelper+45
[0x77098346] combase!EnterForCallback+16e [onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2] onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2
[0x7709816d] combase!SwitchForCallback+206 [onecore\com\combase\dcomrem\crossctx.cxx # 1694] onecore\com\combase\dcomrem\crossctx.cxx # 1694
[0x7709bae4] combase!PerformCallback+bc [onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16] onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16
[0x7709b7f9] combase!CObjectContext::InternalContextCallback+119 [onecore\com\combase\dcomrem\context.cxx # 4421 + 1a] onecore\com\combase\dcomrem\context.cxx # 4421 + 1a
[0x77198e66] combase!CObjectContext::DoCallback+26 [onecore\com\combase\dcomrem\context.cxx # 4254] onecore\com\combase\dcomrem\context.cxx # 4254
[0x6eefd015] comsvcs!CSTAActivityWork::DoWork+175
[0x6eeff0e0] comsvcs!CSTAThread::DoWork+26
[0x6eeff599] comsvcs!CSTAThread::ProcessQueueWork+48
[0x6eeff8dd] comsvcs!CSTAThread::WorkerLoop+13d
[0x76577e71] msvcrt!_callthreadstartex+25
[0x76577f31] msvcrt!_threadstartex+61
[0x765f0419] kernel32!BaseThreadInitThunk+19
[0x77d5662d] ntdll!__RtlUserThreadStart+2f
[0x77d565fd] ntdll!_RtlUserThreadStart+1b
...
Most likely comes from writing to log file
ioo_fso = Server.CreateObject("Scripting.FileSystemObject");
...
loo_file = loo_fso.OpenTextFile(ls_filename,8,true);
...
try {
loo_file.WriteLine("[" + str + "]")} catch (ee) {}
Proces-Monitor shows "SHARING VIOLATION" log records for w3wp.exe on accessing the log file
3 Also experienced ASP 0115 right after custom Server custom component creation
var pbkdf2;
try {
pbkdf2 = Server.CreateObject("Pbkdf2");
pbkdf2.hashPassword(ls_newpassword, 100000);
} catch (e) {
addToLogg("Login:CreateObject failed for Pbkdf2, " + e.description);
}
from FailedReqLogFiles logs, but not identified yet in DebugDiag
Questions
I know ASP Jscript is an old, out-phasing technology, but there should be plenty of Enterprise solution still out there, so could be someone else also runs into these issues.
I would expect the Jscript to fall in regular way, so that the error situation could be handled
Has someone else come upon similar situation?
What are the new restrictions on jscript code?
Are there ways to handle these fails in server-side, prior to returning response to client?
Maybe there is some ASP/jscript environment settings, memory management settings, Windows privileges, permissions that can potentially solve the issues?
Solution
As hinted by #Max (below) the last Microsoft KBs fixes the Jscript Workflow issues.
The only comment in KB that reflects jscript is
Addresses an issue with applications and printer drivers that utilize
the Windows JavaScript engine (jscript.dll) for processing print jobs.
So apparently the fix in common jscript handling
The summary of KBs that fix the issue
Server 2019 1809: KB4519338
Server 2019 1903: KB4517389
Server 2016: KB4519998
Server 2012 R2 : KB4520005
Server 2012: KB4520007
Server 2008 R2: KB4519976
Server 2008 (with SP2): KB4520002
Not required to uninstall previous KB updates.
See that previous monthly Rollup (October 3) is no longer present in Window Updates after installing the new one.
While I didn't manage to isolate the main "First chance exception 0xC0000005" from the Workflows:
After applying the October 8 KB the issue 0xC0000005 is no longer there in the previously failing Workflows
The isolated second chance exceptions are also not reproduced, see ntdl Access Violation case, Server Object Creation case
My isolated Regex samples with test data from the failing workflow didn't reproduce the Exception, so I cannot confirm that the regex use directly caused the issues
However VBScript proved not to be affected, as pointed out by #Lee
This appears to be an unfortunate issue in the basket, while fixing memory corruption creating a several ones.
We also experienced these same errors related to CVE-2019-1367 and classic ASP. We narrowed the scope of the errors to a few places we were using JScript instead of VBScript to do JSON conversions, then we narrowed it further to were we were using regex. We worked around the errors by rewriting the functionality that was in JScript code in VBScript.
I found this article that refers to CVE-2019-13670 with a very similar number and very similar wording: Google Chrome could allow a remote attacker to execute arbitrary code on the system, caused by a V8 memory corruption in regex..
CVE-2019-1367 is specific to Internet Explorer and updated C\Windows\system32\JScript.dll. From that, I'm guessing IE's javascript engine and classic ASP JScript engine are both handled by JScript.dll? Wild guess. CVE-2019-13670 is specific to Chrome (which I assume doesn't use JScript.dll), but it mentioned regex and we found our problems to be specific to regex usage in JScript.
My group is also experiencing these issues. Our entire legacy system is written in ASP
with JScript. The KB4522007 update was installed on 2019-09-25, at which point we noticed the errors. In addition to the error mentioned in the original post, we are experiencing other errors as well:
A trappable error (C0000005) occurred in an external object. The script cannot continue running.
The function returned |.
Function expected
Variable uses an Automation type not supported in JScript
Object doesn't support this property or method
These are all errors happening in unmodified files, and were never observed prior to the update. The errors are periodic, and not systematically reproducible...
Removing the KB4522007 updated caused the errors to go away.
it seems the latest updates from Microsoft fixes the problem.
We've one IIS application server on which we have deployed ASP.NET application.
Now the issue is Server is hanging very frequently and every time we are recycling or restarting application pool. Could you please help me to fix it.
What I've tried:
1) Looked into IIS log and found that there are few requests throwing 500 error some time. All the time except few instances same requests are working fine and getting 200 status.
2) Few resources are taking longer time like
a) "Report.aspx" - State- SendResponse - Time Elapsed- 2-3 minutes
I checked the query and found that sp is returning data in 00:00:02.
b) "NewReport.aspx" - State- ExecuteRequestHandler- Time Elapsed- 1 minutes
I checked the query and found that sp is returning data too quickly.
3) Default recycle time is 29 hrs, and each application is using individual pool.
4) In code application connection has been opened and not closed at very few places.
5) We are using connection pooling, having 2 different connection string with Max Pool=500 and min Pool=50 each.
6) Most of the time there is no error in Event Viewer, some time have received TCP/IP error.
What I am looking for:
1) Do I need to make any changes in IIS to track down or please help if any changes required in IIS?
2) Is there any setting in SQL Server which can help me to find out or fix it?
3) Please help if anything else I've to do to fix this issue from your previous experiences.
I think you should capture a hang dump for W3WP.exe with Procdump or debug diagnostic tool first. We need to check all managed stack trace via WINDBG, sos.dll and mex extension. So that we can figure out worker thread is pending on which method and which module. Then might tell us about how to fix this hang issue. If you could find a way to always reproduce this problem. The problem would become much easier.
This link is about how to capture hang dump.
https://blogs.msdn.microsoft.com/asiatech/2014/01/09/debug-diagnostic-2-0-generate-a-manual-hang-dump-for-all-processes-owned-by-iis/
If you don't know how to analyze dump file manually, debug diagnostic analysis tool could help you achieve this.
I have a website which has been up and running absolutely fine for about 8 months now. It's running .net 4 intergrated mode.
Recently i've started to get some "random" w3wp.exe crashes, and after 5 of them, IIS rapid fail protection kicks in and I have to manually login to the server and start the application pool again.
Here's what the event viewer says for the Error:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: nlssorting.dll, version: 4.0.30319.296, time stamp: 0x504835c7
Exception code: 0xc00000fd
Fault offset: 0x000000000000191f
Faulting process id: 0x1998
Faulting application start time: 0x01ce6e6b9b80c949
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
Report Id: d9cf3164-da5e-11e2-8cc5-f46d0440f6d5
Straight after the crashes, i get an "Information" log in the event viewer which at the bottom gives me the location of a .wer file.
This is what the .wer files contains:
Version=1
EventType=APPCRASH
EventTime=130162847687759734
ReportType=2
Consent=1
ReportIdentifier=d7c5e520-da5e-11e2-8cc5-f46d0440f6d5
IntegratorReportIdentifier=d7c5e51f-da5e-11e2-8cc5-f46d0440f6d5
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=w3wp.exe
Sig[1].Name=Application Version
Sig[1].Value=7.5.7601.17514
Sig[2].Name=Application Timestamp
Sig[2].Value=4ce7afa2
Sig[3].Name=Fault Module Name
Sig[3].Value=nlssorting.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=4.0.30319.296
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=504835c7
Sig[6].Name=Exception Code
Sig[6].Value=c00000fd
Sig[7].Name=Exception Offset
Sig[7].Value=000000000000197d
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.1296.17
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=6141
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=61419d6dee6cf74b8ac2b00b4c3b3373
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=c19b
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=c19b8acf029a3088171b1f5f3dd9dc77
UI[2]=c:\windows\system32\inetsrv\w3wp.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=IIS Worker Process stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[5]=C:\Windows\system32\msvcrt.dll
LoadedModule[6]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[7]=C:\Windows\system32\RPCRT4.dll
LoadedModule[8]=C:\Windows\system32\pcwum.DLL
LoadedModule[9]=C:\Windows\system32\USER32.dll
LoadedModule[10]=C:\Windows\system32\GDI32.dll
LoadedModule[11]=C:\Windows\system32\LPK.dll
LoadedModule[12]=C:\Windows\system32\USP10.dll
LoadedModule[13]=C:\Windows\system32\ole32.dll
LoadedModule[14]=c:\windows\system32\inetsrv\IISUTIL.dll
LoadedModule[15]=C:\Windows\system32\IMM32.DLL
LoadedModule[16]=C:\Windows\system32\MSCTF.dll
LoadedModule[17]=C:\Windows\system32\CRYPTBASE.dll
LoadedModule[18]=C:\Windows\system32\ntmarta.dll
LoadedModule[19]=C:\Windows\system32\WLDAP32.dll
LoadedModule[20]=c:\windows\system32\inetsrv\w3wphost.dll
LoadedModule[21]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll
LoadedModule[23]=C:\Windows\system32\CRYPT32.dll
LoadedModule[24]=C:\Windows\system32\MSASN1.dll
LoadedModule[25]=C:\Windows\system32\XmlLite.dll
LoadedModule[26]=C:\Windows\system32\ktmw32.dll
LoadedModule[27]=c:\windows\system32\inetsrv\IISRES.DLL
LoadedModule[28]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[29]=C:\Windows\system32\rsaenh.dll
LoadedModule[30]=C:\Windows\system32\mscoree.dll
LoadedModule[31]=C:\Windows\system32\CLBCatQ.DLL
LoadedModule[32]=C:\Windows\system32\mlang.dll
LoadedModule[33]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine4.dll
LoadedModule[34]=C:\Windows\system32\MSVCR100_CLR0400.dll
LoadedModule[35]=C:\Windows\system32\USERENV.dll
LoadedModule[36]=C:\Windows\system32\profapi.dll
LoadedModule[37]=C:\Windows\system32\PSAPI.DLL
LoadedModule[38]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[39]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[40]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[41]=C:\Windows\system32\inetsrv\iiscore.dll
LoadedModule[42]=c:\windows\system32\inetsrv\W3TP.dll
LoadedModule[43]=c:\windows\system32\inetsrv\w3dt.dll
LoadedModule[44]=C:\Windows\system32\HTTPAPI.dll
LoadedModule[45]=C:\Windows\system32\slc.dll
LoadedModule[46]=C:\Windows\system32\WS2_32.dll
LoadedModule[47]=C:\Windows\system32\NSI.dll
LoadedModule[48]=C:\Windows\system32\Normaliz.dll
LoadedModule[49]=C:\Windows\system32\faultrep.dll
LoadedModule[50]=C:\Windows\system32\Secur32.dll
LoadedModule[51]=C:\Windows\system32\SSPICLI.DLL
LoadedModule[52]=C:\Windows\system32\NLAapi.dll
LoadedModule[53]=C:\Windows\system32\napinsp.dll
LoadedModule[54]=C:\Windows\System32\mswsock.dll
LoadedModule[55]=C:\Windows\system32\DNSAPI.dll
LoadedModule[56]=C:\Windows\System32\winrnr.dll
LoadedModule[57]=C:\Windows\System32\wshtcpip.dll
LoadedModule[58]=C:\Windows\System32\wship6.dll
LoadedModule[59]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[60]=C:\Windows\system32\WINNSI.DLL
LoadedModule[61]=C:\Windows\system32\rasadhlp.dll
LoadedModule[62]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[63]=C:\Windows\System32\inetsrv\cachuri.dll
LoadedModule[64]=C:\Windows\System32\inetsrv\cachfile.dll
LoadedModule[65]=C:\Windows\System32\inetsrv\cachtokn.dll
LoadedModule[66]=C:\Windows\System32\inetsrv\cachhttp.dll
LoadedModule[67]=C:\Windows\System32\inetsrv\compdyn.dll
LoadedModule[68]=C:\Windows\System32\inetsrv\compstat.dll
LoadedModule[69]=C:\Windows\System32\inetsrv\defdoc.dll
LoadedModule[70]=C:\Windows\System32\inetsrv\protsup.dll
LoadedModule[71]=C:\Windows\System32\inetsrv\redirect.dll
LoadedModule[72]=C:\Windows\System32\inetsrv\static.dll
LoadedModule[73]=C:\Windows\System32\inetsrv\authanon.dll
LoadedModule[74]=C:\Windows\System32\inetsrv\authbas.dll
LoadedModule[75]=C:\Windows\System32\inetsrv\authsspi.dll
LoadedModule[76]=C:\Windows\system32\NETAPI32.dll
LoadedModule[77]=C:\Windows\system32\netutils.dll
LoadedModule[78]=C:\Windows\system32\srvcli.dll
LoadedModule[79]=C:\Windows\system32\wkscli.dll
LoadedModule[80]=C:\Windows\System32\inetsrv\iprestr.dll
LoadedModule[81]=C:\Windows\System32\inetsrv\modrqflt.dll
LoadedModule[82]=C:\Windows\System32\inetsrv\logcust.dll
LoadedModule[83]=C:\Windows\System32\inetsrv\custerr.dll
LoadedModule[84]=C:\Windows\System32\inetsrv\loghttp.dll
LoadedModule[85]=C:\Windows\System32\inetsrv\isapi.dll
LoadedModule[86]=C:\Windows\System32\inetsrv\filter.dll
LoadedModule[87]=C:\Windows\System32\inetsrv\validcfg.dll
LoadedModule[88]=c:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_filter.dll
LoadedModule[89]=C:\Windows\system32\inetsrv\wbhst_pm.dll
LoadedModule[90]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine.dll
LoadedModule[91]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\4f52500ab48877b85e71430f4f46670f\mscorlib.ni.dll
LoadedModule[92]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
LoadedModule[93]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\a91f32875cb3ba779f1b3ceff1690251\System.ni.dll
LoadedModule[94]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\0a8d99339ffe6b25debb8f8201c27664\System.Core.ni.dll
LoadedModule[95]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web\5b905bd7b71f9fd6bea2d05cc1ae85f8\System.Web.ni.dll
LoadedModule[96]=C:\Windows\system32\sxs.dll
LoadedModule[97]=C:\Windows\system32\RpcRtRemote.dll
LoadedModule[98]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Configuration\fa65f89fd682c459fc5e7bcbd0418317\System.Configuration.ni.dll
LoadedModule[99]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\f4afb233f160b8e55aad4660e45b374c\System.Xml.ni.dll
LoadedModule[100]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[101]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.Build.Uti#\14e16d61fae3cd1d9a1fa79b789f8438\Microsoft.Build.Utilities.v4.0.ni.dll
LoadedModule[102]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Cach#\8fdbe304abab0631b8a4310b35f3d93a\System.Runtime.Caching.ni.dll
LoadedModule[103]=C:\Windows\system32\shfolder.dll
LoadedModule[104]=C:\Windows\system32\SHELL32.dll
LoadedModule[105]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.JScript\85204dde340780329b569b025e249c23\Microsoft.JScript.ni.dll
LoadedModule[106]=C:\Windows\system32\version.dll
LoadedModule[107]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_Code.cgixlnxh.dll
LoadedModule[108]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data.Linq\feaa494ad67542d2060b31b9eeb6458b\System.Data.Linq.ni.dll
LoadedModule[109]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data\b928128fca867546a858a1a39240d85c\System.Data.ni.dll
LoadedModule[110]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
LoadedModule[111]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\595a888a\f26c0653_7f81cd01\HtmlAgilityPack.dll
LoadedModule[112]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Drawing\5ae853f556290da9399b15b3619f7e15\System.Drawing.ni.dll
LoadedModule[113]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\85ba5013\f0c8f388_706bce01\TweetSharp.dll
LoadedModule[114]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web.Extensio#\0180a2d993d2a9699cf07f7163524fff\System.Web.Extensions.ni.dll
LoadedModule[115]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Transactions\7b2099a1386e38ff198a51939304ce6e\System.Transactions.ni.dll
LoadedModule[116]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Transactions\v4.0_4.0.0.0__b77a5c561934e089\System.Transactions.dll
LoadedModule[117]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_global.asax.yxdky-qn.dll
LoadedModule[118]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel#\7a5a5ff4a0b3bb4ba4bcc13166918e36\System.ServiceModel.Activation.ni.dll
LoadedModule[119]=C:\Windows\system32\bcrypt.dll
LoadedModule[120]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Dura#\799274e49455d0fe4ca563f42143bef2\System.Runtime.DurableInstancing.ni.dll
LoadedModule[121]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Numerics\a66416296451fe6d2d8a5506ca41b23d\System.Numerics.ni.dll
LoadedModule[122]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel\15834d73d2846fc01ed54488ccfff5c8\System.ServiceModel.ni.dll
LoadedModule[123]=C:\Windows\assembly\NativeImages_v4.0.30319_64\SMDiagnostics\31f93b6be386908ff2727bcd825de0ca\SMDiagnostics.ni.dll
LoadedModule[124]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xaml.Hosting\cf8401f4952deb5303e0d7fd459ce530\System.Xaml.Hosting.ni.dll
LoadedModule[125]=C:\Windows\system32\inetsrv\gzip.dll
LoadedModule[126]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\3d63b311\fe7c9b8a_706bce01\Hammock.ClientProfile.dll
LoadedModule[127]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\6a128bd2\c184e08a_706bce01\Newtonsoft.Json.dll
LoadedModule[128]=C:\Windows\system32\rasapi32.dll
LoadedModule[129]=C:\Windows\system32\rasman.dll
LoadedModule[130]=C:\Windows\system32\rtutils.dll
LoadedModule[131]=C:\Windows\system32\winhttp.dll
LoadedModule[132]=C:\Windows\system32\webio.dll
LoadedModule[133]=C:\Windows\system32\credssp.dll
LoadedModule[134]=C:\Windows\system32\dhcpcsvc6.DLL
LoadedModule[135]=C:\Windows\system32\dhcpcsvc.DLL
LoadedModule[136]=C:\Windows\system32\security.dll
LoadedModule[137]=C:\Windows\system32\schannel.DLL
LoadedModule[138]=C:\Windows\system32\ncrypt.dll
LoadedModule[139]=C:\Windows\system32\bcryptprimitives.dll
LoadedModule[140]=C:\Windows\system32\GPAPI.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=IIS Worker Process
AppPath=c:\windows\system32\inetsrv\w3wp.exe
That nlssorting.dll seems to crop up a lot but I can't seem to find anything online related. The only thing I can find which matches my error is here, but that doesn't really help me.
I'm completely stumped as to where to go from here to fix this. Here's what I've tried:
Loading up IIS log files and trying every request from about 30 minutes before a crash, and none of the pages cause any errors.
Searching my code for any recursion which might cause a stackoverflow, but there isn't any
trawling online for ANYTHING that might help
Has anyone else ever had any problems with nlssorting.dll ? Can i get some more information from the .wer file that might help me pin point where this is happening?
Thanks in advance for any help!
UPDATE
I was using a 3rd party DLL, which was causing a stack overflow exception (0xc00000fd)
After more investigation, it was only happening after a certain chain of events happened - hence the "random" in the title. Removing the DLL fixed the problem.
We had the same problem with one of our sites. Using SVN we tracked it down to a method that was scaning for images within a folder.
I modified the code as follows:
Checking array length of scan results to be > 0 instead of == 1
Adding CultureInfo.InvariantCulture to all Int32.ToString() calls
After this we no longer experienced the error. The exact reason is still unknown.
I believe that none of the above points should make a difference in our environment. I believe that the problem could have been, people modifying image files and folders while the image scanning method was called.
I hope this helps somebody.
For anyone who's curious, this is a PITA to debug. Here are three reasons rumored for this to happen:
(1) Stack overflows, as in the original post.
(2) Too much CPU / memory usage, which becomes obvious and rapid fail protection closes the process.
(3) Unable to respond to pings / requests due to application hogging resources, but in a way that rapid fail protection deems appropriate to end the process, not explicitly because of either (1) or (2).
Our solution was to add manual log tracing in the production environment until we eventually found recursion which was leading the application to be stopped by reliability services (for inability to respond to pings, or process randomly crashing) rather than throwing an in-application exception.
I had the issue where w3p would throw an unhandled error as soon I spun up the site\api URI from the web browser, then it would crash.
I was able to pinpoint what part of my code was causing it in my case it was in the Owin Startup class and I was reading some configuration records from a database but prior to that it gets the connection string from a configuration file outside of the web app directory.
I checked the ownership of the folder it showed my account but apparently the subfolders were not owned by me so I set ownership to me again, clicked OK to allow permissions to traverse the objects and voila bye bye w3p error and the API loaded.
So in my case it was an access denied error on the folder \ file that contained the connection string.
i have created one web application in asp.net[.net 4 framework and visual studio 2010].Now i am running with IIS 5.1 version.When i running my application, i am getting the following error
Server Application Unavailable
The web application you are attempting to access on this web server is currently unavailable. Please hit the "Refresh" button in your web browser to retry your request.
Administrator Note: An error message detailing the cause of this specific request failure can be found in the application event log of the web server. Please review this log entry to discover what caused this error to occur.
I am also checked with EventViewer... It says
Event Type: Error
Event Source: ASP.NET 4.0.30319.0
Event Category: None
Event ID: 1001
Date: 8/10/12
Time: 9:55:47 AM
User: N/A
Computer: SARAVANAN-
Description:
aspnet_wp.exe (PID: 1724) was recycled because memory consumption exceeded the 1221 MB (60 percent of available RAM).
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
I dont know how to resolve this issue? Is there any way to increase the memory consumption limit of RAM for my IIS.
Please guide me to get out of this issue...
Bottom line - you should debug the code to seee why it takes this much memory.
But to answer your actual question, yes, there is a way to increase this limit.
In the machine.config file there is a section named 'processModel' and by default it looks like this:
<processModel autoConfig="true"/>
If you change it to add a 'memoryLimit' value this will modify the percentage value used to determine when to recycle the process. For example:
<processModel autoConfig="true" memoryLimit="90"/>
This would allow the process to consume 90% of available RAM before the process was recycled. The default is '60'. You can of course also set a lower value.