Error "ASP 0115 a Trappable Error Has Occurred" after Microsoft patch CVE-2019-1367 - asp-classic

Jscript Unexpected exceptions
After Windows Server patch Vulnerability (CVE-2019-1367) released in 23. September
Windows Server 2019 (KB4522015) https://support.microsoft.com/en-us/help/4522015/windows-10-update-kb4522015
Windows Server 2016 (KB4522010)
Windows Server 2008 R2 for x64-based Systems Service Pack 1 (KB4522007)
Windows Server 2012 (KB4522007)
Windows Server 2012 R2 (KB4522007)
Updated 07.10.2019 Also "Preview of Monthly Rollup" and "Monthly Rollup" packages are affected and doesn't fix the specific Jscript Workflow issues
Windows Server 2019: KB4516077, KB4524148
Windows Server 2016: KB4516061, KB4524152
Windows Server 2012 R2: KB4516041, KB4524156
in the classic ASP application on several workflow cases are occurring jscript Unexpected errors in server side:
Active Server Pages error 'ASP 0115'
A trappable error (C0000005) occurred in an external object. The script cannot continue running
Active Server Pages error 'ASP 0240'
A ScriptEngine threw exception 'C0000005' in 'IActiveScript::Close()' from 'CActiveScriptEngine::FinalRelease()'.
Patch
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer, aka
'Scripting Engine Memory Corruption Vulnerability'. This CVE ID is
unique from CVE-2019-1221.
https://www.cvedetails.com/cve/CVE-2019-1367/
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer. The
vulnerability could corrupt memory in such a way that an attacker
could run arbitrary code in the context of the current user. An
attacker who successfully exploited the vulnerability could gain the
same user rights as the current user. In a web-based attack scenario,
an attacker could host a specially crafted website that is designed to
exploit the vulnerability through Internet Explorer and then convince
a user to view the website, for example, by sending an email. The
security update addresses the vulnerability by modifying how the
scripting engine handles objects in memory.
https://blog.qualys.com/laws-of-vulnerabilities/2019/09/24/microsoft-releases-out-of-band-security-updates
Patch is said to address issue in Memory management. Doesn't specify what exactly changes, what are the new limitations. But seems that it causes some side-effect failure cases.
Error nature
The errors cannot be handled by regular try-catch approach
The errors causes workflow interruptions
The exception seemingly happens only one time entering the specific workflow, and on repeated web-request for the same routine the code succeeds (Until App-pool restarted).
Sometimes exceptions is entering the workflow first, second or third time.
The exception only happens if IIS ASP Debugging Properties - Enable Server-side Debugging is set to False
Background
Verified that the issue is present on all tested Server instances with the patch. Also isolated the patch by checking the State before and after applying the patch (Server 2012 R2, Server 2016, Windows 10 - 1809)
From Classic ASP Server cannot handle the issue, with try-catch,
General error is returned - Script Error Message
Or if turned off (ASP - Send Errors To Browser) ASP error codes with page where they happen
Event Viewer also registers those errors, but without additional information
Global.asa doesn't offer global error handling, ASP Server object Server.GetLastError() doesn't catch the exception
Explored exceptions with
DebugDiag
Sysinternals Process Monitor
IIS - Failed Request Tracing
Environment
App-Pool: Classic pipeline mode, Enable 32-Bit Applications: True
Application: ASP
ClientL IE 11 Enterprise mode, with ActiveX enabled
Application pool identity is Impersonated in Web request calls
Issues, identified
1 In w3wp__V...__First chance exception 0XC0000005.dmp the assembly
instruction at msvcrt!memcpy+198 ###
in C:\Windows\System32\msvcrt.dll from Microsoft Corporation has caused an access
violation exception (0xC0000005) when trying to read from memory location 0x0000000a on thread 33
Instruction Address
Source
[0x7532a2d8] msvcrt!memcpy+198
[0x6ac17deb] jscript!AString::CopyToBuffer+4b
[0x6ac10524] jscript!AString::ConvertToBSTR+1bb74
[0x6abdf6b7] jscript!PrepareInvoke+277
[0x6abf52df] jscript!InvokeDispatch+8f
[0x6abe2f03] jscript!VAR::InvokeByDispID+523
[0x6abdbde0] jscript!NameTbl::InvokeInternal+270
[0x6abe2b17] jscript!VAR::InvokeByDispID+137
[0x6abe6083] jscript!CScriptRuntime::Run+2db3
...
Followed by - Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b7c2d77] jscript!VarStack::ScavengeRoots+27
[0x6b7c2b89] jscript!GcContext::CollectCore+79
[0x6b7c2af4] jscript!GcContext::Collect+1b
[0x6b7bca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b7a604a] jscript!CSession::Close+18a
[0x6b7a32d9] jscript!COleScript::CloseInternal+13b
[0x6b7a2d36] jscript!COleScript::Close+16
[0x6b8a71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
Not identified the exact line that causes the issue, the FailedRequestTrace last record is assigning string variable from Application Scope xml object attribute. (CurrentStatement return attrib.text)
Similar case - access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b907e09] jscript!AString::CopyToBuffer+69
[0x6b900524] jscript!AString::ConvertToBSTR+1bb74
[0x6b8e49a7] jscript!VAR::ConvertASTRtoBSTR+13
[0x6b8c49e8] jscript!VAR::GetValue+58
[0x6b8e0f34] jscript!ConvertToString+58
[0x6b922fbf] jscript!JsString+4f
[0x6b8d92e6] jscript!NatFncObj::Call+e6
...
Followed by - access violation exception (0xC0000005) when trying to read from memory location 0x004e0049
[0x6b8e2d77] jscript!VarStack::ScavengeRoots+27
[0x6b8e2b89] jscript!GcContext::CollectCore+79
[0x6b8e2af4] jscript!GcContext::Collect+1b
[0x6b8dca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b8c604a] jscript!CSession::Close+18a
[0x6b8c32d9] jscript!COleScript::CloseInternal+13b
[0x6b8c2d36] jscript!COleScript::Close+16
[0x6bfb71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
2 In w3wp__...__Second_Chance_Exception_C0000005.dmp the assembly instruction at asp!CResponseBuffer::Write+3a
in \?\C:\Windows\System32\inetsrv\asp.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000014 on thread 32
[0x6f042e88] asp!CResponseBuffer::Write+3a
[0x6f0452ea] asp!CResponse::WriteSz+4c
[0x6f02dd3b] asp!CErrInfo::LogErrortoBrowser+ff
[0x6f02d4c9] asp!CErrInfo::LogErrortoBrowserWrapper+d7
[0x6f02d047] asp!CErrInfo::LogError+e8
[0x6f02e241] asp!HandleError+116
[0x6f02f009] asp!HandleErrorMissingFilename+df
[0x6f04941b] asp!CActiveScriptEngine::Call+bb
[0x6f030eff] asp!CallScriptFunctionOfEngine+4d
[0x6f02f99f] asp!ExecuteRequest+173
[0x6f02f828] asp!Execute+23d
[0x6f035c6f] asp!CHitObj::ViperAsyncCallback+467
[0x6f05df53] asp!CViperAsyncRequest::OnCall+73
[0x6eefd325] comsvcs!CSTAActivityWork::STAActivityWorkHelper+45
[0x77098346] combase!EnterForCallback+16e [onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2] onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2
[0x7709816d] combase!SwitchForCallback+206 [onecore\com\combase\dcomrem\crossctx.cxx # 1694] onecore\com\combase\dcomrem\crossctx.cxx # 1694
[0x7709bae4] combase!PerformCallback+bc [onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16] onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16
[0x7709b7f9] combase!CObjectContext::InternalContextCallback+119 [onecore\com\combase\dcomrem\context.cxx # 4421 + 1a] onecore\com\combase\dcomrem\context.cxx # 4421 + 1a
[0x77198e66] combase!CObjectContext::DoCallback+26 [onecore\com\combase\dcomrem\context.cxx # 4254] onecore\com\combase\dcomrem\context.cxx # 4254
[0x6eefd015] comsvcs!CSTAActivityWork::DoWork+175
[0x6eeff0e0] comsvcs!CSTAThread::DoWork+26
[0x6eeff599] comsvcs!CSTAThread::ProcessQueueWork+48
[0x6eeff8dd] comsvcs!CSTAThread::WorkerLoop+13d
[0x76577e71] msvcrt!_callthreadstartex+25
[0x76577f31] msvcrt!_threadstartex+61
[0x765f0419] kernel32!BaseThreadInitThunk+19
[0x77d5662d] ntdll!__RtlUserThreadStart+2f
[0x77d565fd] ntdll!_RtlUserThreadStart+1b
...
Most likely comes from writing to log file
ioo_fso = Server.CreateObject("Scripting.FileSystemObject");
...
loo_file = loo_fso.OpenTextFile(ls_filename,8,true);
...
try {
loo_file.WriteLine("[" + str + "]")} catch (ee) {}
Proces-Monitor shows "SHARING VIOLATION" log records for w3wp.exe on accessing the log file
3 Also experienced ASP 0115 right after custom Server custom component creation
var pbkdf2;
try {
pbkdf2 = Server.CreateObject("Pbkdf2");
pbkdf2.hashPassword(ls_newpassword, 100000);
} catch (e) {
addToLogg("Login:CreateObject failed for Pbkdf2, " + e.description);
}
from FailedReqLogFiles logs, but not identified yet in DebugDiag
Questions
I know ASP Jscript is an old, out-phasing technology, but there should be plenty of Enterprise solution still out there, so could be someone else also runs into these issues.
I would expect the Jscript to fall in regular way, so that the error situation could be handled
Has someone else come upon similar situation?
What are the new restrictions on jscript code?
Are there ways to handle these fails in server-side, prior to returning response to client?
Maybe there is some ASP/jscript environment settings, memory management settings, Windows privileges, permissions that can potentially solve the issues?
Solution
As hinted by #Max (below) the last Microsoft KBs fixes the Jscript Workflow issues.
The only comment in KB that reflects jscript is
Addresses an issue with applications and printer drivers that utilize
the Windows JavaScript engine (jscript.dll) for processing print jobs.
So apparently the fix in common jscript handling
The summary of KBs that fix the issue
Server 2019 1809: KB4519338
Server 2019 1903: KB4517389
Server 2016: KB4519998
Server 2012 R2 : KB4520005
Server 2012: KB4520007
Server 2008 R2: KB4519976
Server 2008 (with SP2): KB4520002
Not required to uninstall previous KB updates.
See that previous monthly Rollup (October 3) is no longer present in Window Updates after installing the new one.
While I didn't manage to isolate the main "First chance exception 0xC0000005" from the Workflows:
After applying the October 8 KB the issue 0xC0000005 is no longer there in the previously failing Workflows
The isolated second chance exceptions are also not reproduced, see ntdl Access Violation case, Server Object Creation case
My isolated Regex samples with test data from the failing workflow didn't reproduce the Exception, so I cannot confirm that the regex use directly caused the issues
However VBScript proved not to be affected, as pointed out by #Lee
This appears to be an unfortunate issue in the basket, while fixing memory corruption creating a several ones.

We also experienced these same errors related to CVE-2019-1367 and classic ASP. We narrowed the scope of the errors to a few places we were using JScript instead of VBScript to do JSON conversions, then we narrowed it further to were we were using regex. We worked around the errors by rewriting the functionality that was in JScript code in VBScript.
I found this article that refers to CVE-2019-13670 with a very similar number and very similar wording: Google Chrome could allow a remote attacker to execute arbitrary code on the system, caused by a V8 memory corruption in regex..
CVE-2019-1367 is specific to Internet Explorer and updated C\Windows\system32\JScript.dll. From that, I'm guessing IE's javascript engine and classic ASP JScript engine are both handled by JScript.dll? Wild guess. CVE-2019-13670 is specific to Chrome (which I assume doesn't use JScript.dll), but it mentioned regex and we found our problems to be specific to regex usage in JScript.

My group is also experiencing these issues. Our entire legacy system is written in ASP
with JScript. The KB4522007 update was installed on 2019-09-25, at which point we noticed the errors. In addition to the error mentioned in the original post, we are experiencing other errors as well:
A trappable error (C0000005) occurred in an external object. The script cannot continue running.
The function returned |.
Function expected
Variable uses an Automation type not supported in JScript
Object doesn't support this property or method
These are all errors happening in unmodified files, and were never observed prior to the update. The errors are periodic, and not systematically reproducible...
Removing the KB4522007 updated caused the errors to go away.

it seems the latest updates from Microsoft fixes the problem.

Related

IIS hung requests - can't see CLR stacktraces in memory dump

ASP.NET WebAPI2 application on .NET 4.6.2, hosted on IIS on Windows Server 2016. From time to time, there is a lot (hundreds) of requests stuck for hours (despite the fact I have request timeout 60s set) with no CPU usage. So, I took the memory dump of w3wp process, along with sos.dll, clr.dll and mscordacwks.dll and all my project's dlls and pdbs from bin directory from server and used WinDbg as described in many blogs and tutorials. But, in all of them, they are able to directly see CLR stack by calling ~*e !clrstack. I can see CLR stacktrace for some Redis and ApplicationInsights workers, but for all other managed threads I can see only:
OS Thread Id: 0x1124 (3)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
!dumpstack for any of these gives just this:
0:181> !dumpstack
OS Thread Id: 0x1754 (181)
Current frame: ntdll!NtWaitForSingleObject+0x14
Child-SP RetAddr Caller, Callee
000000b942c7f6a0 00007fff33d63acf KERNELBASE!WaitForSingleObjectEx+0x8f, calling ntdll!NtWaitForSingleObject
000000b942c7f740 00007fff253377a6 clr!CLRSemaphore::Wait+0x8a, calling kernel32!WaitForSingleObjectEx
000000b942c7f7b0 00007fff25335331 clr!GCCoop::GCCoop+0xe, calling clr!GetThread
000000b942c7f800 00007fff25337916 clr!ThreadpoolMgr::UnfairSemaphore::Wait+0xf1, calling clr!CLRSemaphore::Wait
000000b942c7f840 00007fff253378b1 clr!ThreadpoolMgr::WorkerThreadStart+0x2d1, calling clr!ThreadpoolMgr::UnfairSemaphore::Wait
000000b942c7f8e0 00007fff253d952f clr!Thread::intermediateThreadProc+0x86
000000b942c7f9e0 00007fff253d950f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
000000b942c7fa20 00007fff37568364 kernel32!BaseThreadInitThunk+0x14, calling ntdll!LdrpDispatchUserCallTarget
000000b942c7fa50 00007fff3773e821 ntdll!RtlUserThreadStart+0x21, calling ntdll!LdrpDispatchUserCallTarget
So I have no idea, where to look for bug in my code.
(here is the full result:
https://gist.github.com/rouen-sk/eff11844557521de367fa9182cb94a82
and here is the results of !threads:
https://gist.github.com/rouen-sk/b61cba97a4d8300c08d6a8808c4bff6e)
What can I do? Google search for GetFrameContext failed gives nothing helpful.
As mentioned, this is not trivial, however you can find a case study of similar problem here: https://blogs.msdn.microsoft.com/rodneyviana/2015/03/27/the-case-of-the-non-responsive-mvc-web-application/
In a nutshell:
Download NetExt. It is the zip file here:
https://github.com/rodneyviana/netext/tree/master/Binaries
Open your dump and load NetExt
Run !windex to index the heap
Run !whttp -order -running to see a list of running requests
If the requests contains thread number you can go to the thread to see what is happening
If the requests contains --- instead of thread number, they are waiting a thread and this is a sign that some throttling is happening
If it is a WCF service, run !wservice to see the services
Run !wruntime to see runtime information
Run !wapppool to see Application Pool information
Run !wdae to list all errors
... And so it goes. When you do this again and again you will be able to spot issues easily

Random w3wp.exe crashes in .net 4

I have a website which has been up and running absolutely fine for about 8 months now. It's running .net 4 intergrated mode.
Recently i've started to get some "random" w3wp.exe crashes, and after 5 of them, IIS rapid fail protection kicks in and I have to manually login to the server and start the application pool again.
Here's what the event viewer says for the Error:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: nlssorting.dll, version: 4.0.30319.296, time stamp: 0x504835c7
Exception code: 0xc00000fd
Fault offset: 0x000000000000191f
Faulting process id: 0x1998
Faulting application start time: 0x01ce6e6b9b80c949
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
Report Id: d9cf3164-da5e-11e2-8cc5-f46d0440f6d5
Straight after the crashes, i get an "Information" log in the event viewer which at the bottom gives me the location of a .wer file.
This is what the .wer files contains:
Version=1
EventType=APPCRASH
EventTime=130162847687759734
ReportType=2
Consent=1
ReportIdentifier=d7c5e520-da5e-11e2-8cc5-f46d0440f6d5
IntegratorReportIdentifier=d7c5e51f-da5e-11e2-8cc5-f46d0440f6d5
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=w3wp.exe
Sig[1].Name=Application Version
Sig[1].Value=7.5.7601.17514
Sig[2].Name=Application Timestamp
Sig[2].Value=4ce7afa2
Sig[3].Name=Fault Module Name
Sig[3].Value=nlssorting.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=4.0.30319.296
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=504835c7
Sig[6].Name=Exception Code
Sig[6].Value=c00000fd
Sig[7].Name=Exception Offset
Sig[7].Value=000000000000197d
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.1296.17
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=6141
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=61419d6dee6cf74b8ac2b00b4c3b3373
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=c19b
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=c19b8acf029a3088171b1f5f3dd9dc77
UI[2]=c:\windows\system32\inetsrv\w3wp.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=IIS Worker Process stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[5]=C:\Windows\system32\msvcrt.dll
LoadedModule[6]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[7]=C:\Windows\system32\RPCRT4.dll
LoadedModule[8]=C:\Windows\system32\pcwum.DLL
LoadedModule[9]=C:\Windows\system32\USER32.dll
LoadedModule[10]=C:\Windows\system32\GDI32.dll
LoadedModule[11]=C:\Windows\system32\LPK.dll
LoadedModule[12]=C:\Windows\system32\USP10.dll
LoadedModule[13]=C:\Windows\system32\ole32.dll
LoadedModule[14]=c:\windows\system32\inetsrv\IISUTIL.dll
LoadedModule[15]=C:\Windows\system32\IMM32.DLL
LoadedModule[16]=C:\Windows\system32\MSCTF.dll
LoadedModule[17]=C:\Windows\system32\CRYPTBASE.dll
LoadedModule[18]=C:\Windows\system32\ntmarta.dll
LoadedModule[19]=C:\Windows\system32\WLDAP32.dll
LoadedModule[20]=c:\windows\system32\inetsrv\w3wphost.dll
LoadedModule[21]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll
LoadedModule[23]=C:\Windows\system32\CRYPT32.dll
LoadedModule[24]=C:\Windows\system32\MSASN1.dll
LoadedModule[25]=C:\Windows\system32\XmlLite.dll
LoadedModule[26]=C:\Windows\system32\ktmw32.dll
LoadedModule[27]=c:\windows\system32\inetsrv\IISRES.DLL
LoadedModule[28]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[29]=C:\Windows\system32\rsaenh.dll
LoadedModule[30]=C:\Windows\system32\mscoree.dll
LoadedModule[31]=C:\Windows\system32\CLBCatQ.DLL
LoadedModule[32]=C:\Windows\system32\mlang.dll
LoadedModule[33]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine4.dll
LoadedModule[34]=C:\Windows\system32\MSVCR100_CLR0400.dll
LoadedModule[35]=C:\Windows\system32\USERENV.dll
LoadedModule[36]=C:\Windows\system32\profapi.dll
LoadedModule[37]=C:\Windows\system32\PSAPI.DLL
LoadedModule[38]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[39]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[40]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[41]=C:\Windows\system32\inetsrv\iiscore.dll
LoadedModule[42]=c:\windows\system32\inetsrv\W3TP.dll
LoadedModule[43]=c:\windows\system32\inetsrv\w3dt.dll
LoadedModule[44]=C:\Windows\system32\HTTPAPI.dll
LoadedModule[45]=C:\Windows\system32\slc.dll
LoadedModule[46]=C:\Windows\system32\WS2_32.dll
LoadedModule[47]=C:\Windows\system32\NSI.dll
LoadedModule[48]=C:\Windows\system32\Normaliz.dll
LoadedModule[49]=C:\Windows\system32\faultrep.dll
LoadedModule[50]=C:\Windows\system32\Secur32.dll
LoadedModule[51]=C:\Windows\system32\SSPICLI.DLL
LoadedModule[52]=C:\Windows\system32\NLAapi.dll
LoadedModule[53]=C:\Windows\system32\napinsp.dll
LoadedModule[54]=C:\Windows\System32\mswsock.dll
LoadedModule[55]=C:\Windows\system32\DNSAPI.dll
LoadedModule[56]=C:\Windows\System32\winrnr.dll
LoadedModule[57]=C:\Windows\System32\wshtcpip.dll
LoadedModule[58]=C:\Windows\System32\wship6.dll
LoadedModule[59]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[60]=C:\Windows\system32\WINNSI.DLL
LoadedModule[61]=C:\Windows\system32\rasadhlp.dll
LoadedModule[62]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[63]=C:\Windows\System32\inetsrv\cachuri.dll
LoadedModule[64]=C:\Windows\System32\inetsrv\cachfile.dll
LoadedModule[65]=C:\Windows\System32\inetsrv\cachtokn.dll
LoadedModule[66]=C:\Windows\System32\inetsrv\cachhttp.dll
LoadedModule[67]=C:\Windows\System32\inetsrv\compdyn.dll
LoadedModule[68]=C:\Windows\System32\inetsrv\compstat.dll
LoadedModule[69]=C:\Windows\System32\inetsrv\defdoc.dll
LoadedModule[70]=C:\Windows\System32\inetsrv\protsup.dll
LoadedModule[71]=C:\Windows\System32\inetsrv\redirect.dll
LoadedModule[72]=C:\Windows\System32\inetsrv\static.dll
LoadedModule[73]=C:\Windows\System32\inetsrv\authanon.dll
LoadedModule[74]=C:\Windows\System32\inetsrv\authbas.dll
LoadedModule[75]=C:\Windows\System32\inetsrv\authsspi.dll
LoadedModule[76]=C:\Windows\system32\NETAPI32.dll
LoadedModule[77]=C:\Windows\system32\netutils.dll
LoadedModule[78]=C:\Windows\system32\srvcli.dll
LoadedModule[79]=C:\Windows\system32\wkscli.dll
LoadedModule[80]=C:\Windows\System32\inetsrv\iprestr.dll
LoadedModule[81]=C:\Windows\System32\inetsrv\modrqflt.dll
LoadedModule[82]=C:\Windows\System32\inetsrv\logcust.dll
LoadedModule[83]=C:\Windows\System32\inetsrv\custerr.dll
LoadedModule[84]=C:\Windows\System32\inetsrv\loghttp.dll
LoadedModule[85]=C:\Windows\System32\inetsrv\isapi.dll
LoadedModule[86]=C:\Windows\System32\inetsrv\filter.dll
LoadedModule[87]=C:\Windows\System32\inetsrv\validcfg.dll
LoadedModule[88]=c:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_filter.dll
LoadedModule[89]=C:\Windows\system32\inetsrv\wbhst_pm.dll
LoadedModule[90]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine.dll
LoadedModule[91]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\4f52500ab48877b85e71430f4f46670f\mscorlib.ni.dll
LoadedModule[92]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
LoadedModule[93]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\a91f32875cb3ba779f1b3ceff1690251\System.ni.dll
LoadedModule[94]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\0a8d99339ffe6b25debb8f8201c27664\System.Core.ni.dll
LoadedModule[95]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web\5b905bd7b71f9fd6bea2d05cc1ae85f8\System.Web.ni.dll
LoadedModule[96]=C:\Windows\system32\sxs.dll
LoadedModule[97]=C:\Windows\system32\RpcRtRemote.dll
LoadedModule[98]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Configuration\fa65f89fd682c459fc5e7bcbd0418317\System.Configuration.ni.dll
LoadedModule[99]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\f4afb233f160b8e55aad4660e45b374c\System.Xml.ni.dll
LoadedModule[100]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[101]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.Build.Uti#\14e16d61fae3cd1d9a1fa79b789f8438\Microsoft.Build.Utilities.v4.0.ni.dll
LoadedModule[102]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Cach#\8fdbe304abab0631b8a4310b35f3d93a\System.Runtime.Caching.ni.dll
LoadedModule[103]=C:\Windows\system32\shfolder.dll
LoadedModule[104]=C:\Windows\system32\SHELL32.dll
LoadedModule[105]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.JScript\85204dde340780329b569b025e249c23\Microsoft.JScript.ni.dll
LoadedModule[106]=C:\Windows\system32\version.dll
LoadedModule[107]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_Code.cgixlnxh.dll
LoadedModule[108]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data.Linq\feaa494ad67542d2060b31b9eeb6458b\System.Data.Linq.ni.dll
LoadedModule[109]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data\b928128fca867546a858a1a39240d85c\System.Data.ni.dll
LoadedModule[110]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
LoadedModule[111]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\595a888a\f26c0653_7f81cd01\HtmlAgilityPack.dll
LoadedModule[112]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Drawing\5ae853f556290da9399b15b3619f7e15\System.Drawing.ni.dll
LoadedModule[113]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\85ba5013\f0c8f388_706bce01\TweetSharp.dll
LoadedModule[114]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web.Extensio#\0180a2d993d2a9699cf07f7163524fff\System.Web.Extensions.ni.dll
LoadedModule[115]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Transactions\7b2099a1386e38ff198a51939304ce6e\System.Transactions.ni.dll
LoadedModule[116]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Transactions\v4.0_4.0.0.0__b77a5c561934e089\System.Transactions.dll
LoadedModule[117]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_global.asax.yxdky-qn.dll
LoadedModule[118]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel#\7a5a5ff4a0b3bb4ba4bcc13166918e36\System.ServiceModel.Activation.ni.dll
LoadedModule[119]=C:\Windows\system32\bcrypt.dll
LoadedModule[120]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Dura#\799274e49455d0fe4ca563f42143bef2\System.Runtime.DurableInstancing.ni.dll
LoadedModule[121]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Numerics\a66416296451fe6d2d8a5506ca41b23d\System.Numerics.ni.dll
LoadedModule[122]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel\15834d73d2846fc01ed54488ccfff5c8\System.ServiceModel.ni.dll
LoadedModule[123]=C:\Windows\assembly\NativeImages_v4.0.30319_64\SMDiagnostics\31f93b6be386908ff2727bcd825de0ca\SMDiagnostics.ni.dll
LoadedModule[124]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xaml.Hosting\cf8401f4952deb5303e0d7fd459ce530\System.Xaml.Hosting.ni.dll
LoadedModule[125]=C:\Windows\system32\inetsrv\gzip.dll
LoadedModule[126]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\3d63b311\fe7c9b8a_706bce01\Hammock.ClientProfile.dll
LoadedModule[127]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\6a128bd2\c184e08a_706bce01\Newtonsoft.Json.dll
LoadedModule[128]=C:\Windows\system32\rasapi32.dll
LoadedModule[129]=C:\Windows\system32\rasman.dll
LoadedModule[130]=C:\Windows\system32\rtutils.dll
LoadedModule[131]=C:\Windows\system32\winhttp.dll
LoadedModule[132]=C:\Windows\system32\webio.dll
LoadedModule[133]=C:\Windows\system32\credssp.dll
LoadedModule[134]=C:\Windows\system32\dhcpcsvc6.DLL
LoadedModule[135]=C:\Windows\system32\dhcpcsvc.DLL
LoadedModule[136]=C:\Windows\system32\security.dll
LoadedModule[137]=C:\Windows\system32\schannel.DLL
LoadedModule[138]=C:\Windows\system32\ncrypt.dll
LoadedModule[139]=C:\Windows\system32\bcryptprimitives.dll
LoadedModule[140]=C:\Windows\system32\GPAPI.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=IIS Worker Process
AppPath=c:\windows\system32\inetsrv\w3wp.exe
That nlssorting.dll seems to crop up a lot but I can't seem to find anything online related. The only thing I can find which matches my error is here, but that doesn't really help me.
I'm completely stumped as to where to go from here to fix this. Here's what I've tried:
Loading up IIS log files and trying every request from about 30 minutes before a crash, and none of the pages cause any errors.
Searching my code for any recursion which might cause a stackoverflow, but there isn't any
trawling online for ANYTHING that might help
Has anyone else ever had any problems with nlssorting.dll ? Can i get some more information from the .wer file that might help me pin point where this is happening?
Thanks in advance for any help!
UPDATE
I was using a 3rd party DLL, which was causing a stack overflow exception (0xc00000fd)
After more investigation, it was only happening after a certain chain of events happened - hence the "random" in the title. Removing the DLL fixed the problem.
We had the same problem with one of our sites. Using SVN we tracked it down to a method that was scaning for images within a folder.
I modified the code as follows:
Checking array length of scan results to be > 0 instead of == 1
Adding CultureInfo.InvariantCulture to all Int32.ToString() calls
After this we no longer experienced the error. The exact reason is still unknown.
I believe that none of the above points should make a difference in our environment. I believe that the problem could have been, people modifying image files and folders while the image scanning method was called.
I hope this helps somebody.
For anyone who's curious, this is a PITA to debug. Here are three reasons rumored for this to happen:
(1) Stack overflows, as in the original post.
(2) Too much CPU / memory usage, which becomes obvious and rapid fail protection closes the process.
(3) Unable to respond to pings / requests due to application hogging resources, but in a way that rapid fail protection deems appropriate to end the process, not explicitly because of either (1) or (2).
Our solution was to add manual log tracing in the production environment until we eventually found recursion which was leading the application to be stopped by reliability services (for inability to respond to pings, or process randomly crashing) rather than throwing an in-application exception.
I had the issue where w3p would throw an unhandled error as soon I spun up the site\api URI from the web browser, then it would crash.
I was able to pinpoint what part of my code was causing it in my case it was in the Owin Startup class and I was reading some configuration records from a database but prior to that it gets the connection string from a configuration file outside of the web app directory.
I checked the ownership of the folder it showed my account but apparently the subfolders were not owned by me so I set ownership to me again, clicked OK to allow permissions to traverse the objects and voila bye bye w3p error and the API loaded.
So in my case it was an access denied error on the folder \ file that contained the connection string.

SmartTarget Errors in log file

I don't have any errors with my smart target application, but I do see in the event log, the following error messages:
ERROR 2012-09-19 14:30:09
com.tridion.smarttarget.utils.AmbientDataHelper - can't find defined
trigger-types in claim store (check if your smarttarget cartridge is
up and running)
and:
ERROR 2012-09-19 14:30:11
com.tridion.smarttarget.tags.TimeoutQueryRunner - The fredhopper query
timed out java.util.concurrent.TimeoutException at
java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at
java.util.concurrent.FutureTask.get(Unknown Source) at
com.tridion.smarttarget.tags.TimeoutQueryRunner.executeQuery(TimeoutQueryRunner.java:64)
ERROR 2012-09-19 14:30:11
com.tridion.smarttarget.tags.TimeoutQueryRunner - The fredhopper query
timed out
I would really like to understand what is causing these and how I can remove them. Or some suggested steps to help me debug this would be great :)
As I say, everything is working perfectly, later on in the logs I see the query to ST is correct and the results being generated.
In the event that is helps, I'm running on a 2009 implementation with Smart Target 2010, java 1.5.
thanks
John
Sounds like you might have a trigger configured in ST that does not actually exist in the ADF (or is mismatched). Have you looked through your trigger-types.xml file for anything obvious? Have you disabled an ADF cartridge but not removed the corresponding trigger in the XML perhaps? See the documentation for Defining trigger types.
I think your timeout is coming from the SmartTarget region rather than FredHopper. Sometimes a query that isn't already cached in FredHopper can take a while to return, even though it's ultimately successful. The ST query tag has a timeout (defined in the smarttarget_conf.xml file, or over-ridden with a tag attribute) that it will wait for a response from Fredhopper for before resorting to using the fallback content. This might explain why you see later in the logs that the query is correct and that results are returned. See the documentation for <tcdl:query>.
No conclusive answer for you I'm afraid, but I hope that helps.
The first error is logged if your SmartTarget cartridge is not running -- or if the data that it puts into ADF is lost somehow (e.g. you have disabled sessions in your web server).
In that case, SmartTarget will still do a query but it won't include anything from the Ambient Data Framework in it. If you don't have any triggers based on ambient data, the end result is the same for you.
To get rid of the error, make sure that smarttarget_cartridge is configured correctly.
As for the timeout error, it simply means that the query sent to Fredhopper took longer than the configured time. In that case it will show the fallback content instead. If this is happening a lot, you might want to increase the timeout within smarttarget_conf.xml.
I hope you found the issue, but for future reference, the first error message is raised when the claim "taf:claim:ambientdata:definedtriggertypes" is not set by the SmartTarget cartridge. This can be caused by:
SmartTarget cartridge could not load the the trigger types from the SmartTarget server. The log will show an error "can't retrieve list of defined trigger types from FH".
The HTTP session on your web server is expired during an active visit (the HTTP session expired but the browser is still open) and the claim is "lost".
The server does not support sessions like Peter mentioned.

Tridion broker database connectivity issue

I'm not quite sure what's going on, but several issues are occurring on our website supported by Tridion 2011 SP1 that I can only think is being caused by the broker db
1, Loading the website results in the error with the Ambient Data Module:
[NullReferenceException: Object reference not set to an instance of an object.]
Tridion.ContentDelivery.AmbientData.HttpModule.OnRequestStart(Object sender, EventArgs e) +292
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +79
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +269
2, Audience manager profile synchronisation between the broker and subscription management databases is failing, receiving the following errors from the logs:
Error occurred while fetching a synchronization batch from the presentation system with url: [URL] Profilesync.aspx. Message class java.sql.SQLException No Data Access Object for AudienceManagerProfile java.sql.SQLException: No Data Access Object for AudienceManagerProfile
3, Publishing the website publication repeatedly fails at the transport stage, returning:
Transport failed: Could not transport tcm_0-10689-66560.Content.zip using HTTPS
I know these sound like several issues happening at once, but my site was running fine up until another web publication was added into blueprinting. Since then we've been getting these errors, and like to think it's all related to an issue with the broker db.
Anyone come across something like this?
UPDATE: Should also add the fact that dynamic linking has stopped working as well, which strengthens my belief there is an issue either connecting to or with the broker database
That's full of loaded questions, oh.. boy.. let's take one by one.
Did you check your license file is valid and did not expire recently.
I would rule the license file out first. If that's not the case then see below.
First, Adding publication blueprinting publication does not have any impact on CDA site. So, I would totally rule this out.
Second, Your site up and running until publication added.. This can't be happening. There should be some configurations changed/added on your web site. Sometimes, the configurations updated but the APPPool did not recycled so you don't see the impact. At later time, when the AppPool restarted/recycled you will notice the errors and might have the impression that the site is breaking all of sudden.
Did you double check your configs and dlls?
Are you able to connect to your database directly with broker user
id/password?
Any firewall changes happened recently, try connecting to DB from
CDA Server
For #3, I had this same problem... is it possible that adding the new publication caused the package size to increase from less than 30mb to more than 30mb? If so, check IIS Request Filtering >> Edit Feature Settings and look at Maximum allowed content length. This defaults to 30MB and for us increasing this size to something larger than our package size solved the Transport Failed error (since IIS would just reject our transports due to the size being greater than 30mb).
About issue #1: Could it be you have the Tridion.ContentDelivery.AmbientData.HttpModule configured in the Web.config, but are lacking the java installation? Do you have a cd_ambient_conf.xml in your config folder? And a cd_ambient.jar in the lib?
About issue # 2: you seem to be missing the audience manager DAOs (Data Access Objects). Do you have the following in your cd_storage_conf.xml?
<StorageBindings>
<Bundle src="AudienceManagerDAOBundle.xml"/>
</StorageBindings>
About issue # 3: do you have more information in the cd_transport.log or windows event viewer?
I agree with Ram that the added publication couldn't have caused this. It looks like your CD installation was changed somehow. Did you do an upgrade or something?

MSDeploy issues (WMSVC 500 error)

Having some issues with MSDeploy on a windows server 2008 box, the internal service is throwing a 500 error without putting anything in the server's event logs.
I'm attempting to setup automated deployments using MSBuild/TeamCity/MSDeploy, and this is basically the current halting point, has anyone come across this issue before?
Thanks, Ed
To find out why you are getting this error you should enable logging.
First, enable Failed Request Tracing for the web management service. You can see how to do this by referring to the "Optional: Set Up Tracing" section of this article:
http://learn.iis.net/page.aspx/984/configure-web-deploy/
The "frebs" can be found in:
C:\inetpub\logs\wmsvc\TracingLogFiles\W3SVC1
Open each of the frXXXXXX.xml files with IE and it'll use the freb.xsl transform to generate a nice report.
Don't delete freb.xsl when you're done, it doesn't always get recreated.
Then turn on logging for the web management service:
http://technet.microsoft.com/en-us/library/ff729437(WS.10).aspx
You want to have the following registry entry configured:
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\IIS Extensions\MSDeploy\1]
"EnabledTraceLevel"=dword:00000002
"EnabledTraceSources"=dword:000001ff
You can fiddle with the tracing levels/sources to increase and decrease the verbosity of the logs.
As per the article the management service logs are written to:
%WINDIR%\ServiceProfiles\LocalService\AppData\Local\Temp\WMSvc.log

Resources