The dev environment in our organization recently had the BizTalk CU patch installed. The existing version was BizTalk 2013 R2 and CU1 was installed over it.
After the upgrade the BizTalk admin console is not getting launched. The applications are running fine as some of them have been tested, however the following error pops up while trying to expand BizTalk group in the Administrator:
FX:{9A77C670-2FFD-4ab4-8F20-169C08FB8320}
Object reference not set to an instance of an object.
Exception Type : System.Null reference Exception
Exception Stack Trace:
at Microsoft.BizTalk.ExceptionMessageBox.BtsExceptionMessageBox.ReprocessManagementException(Exception exception, Exception newInnerException)
at Microsoft.BizTalk.ExceptionMessageBox.BtsExceptionMessageBox.ReprocessSpecificExceptions(Exception exception, Exception newInnerException)
at Microsoft.BizTalk.ExceptionMessageBox.BtsExceptionMessageBox.ReprocessExceptionRecursive(Exception exception)
at Microsoft.BizTalk.ExceptionMessageBox.BtsExceptionMessageBox..ctor(Exception exception, ExceptionMessageBoxButtons buttons, ExceptionMessageBoxSymbol symbol)
at Microsoft.BizTalk.SnapIn.Framework.FrameworkNotification.Show(Exception exception, String caption, MessageBoxButtons buttons, MessageBoxIcon icon, Control staMarshaller, IWin32Window parent)
at Microsoft.BizTalk.Administration.SnapIn.GroupNode.FullRefresh(Object o, ResultsChangedEventArgs e)
at Microsoft.BizTalk.Administration.SnapIn.GroupNode.OnExpand(AsyncStatus status)
at Microsoft.ManagementConsole.NodeSyncManager.ProcessRequest(NodeRequestInfo info, IRequestStatus requestStatus)
at Microsoft.ManagementConsole.NamespaceSnapInBase.ProcessRequest(Request request)
at Microsoft.ManagementConsole.SnapIn.ProcessRequest(Request request)
at Microsoft.ManagementConsole.Internal.SnapInClient.Microsoft.ManagementConsole.Internal.IMessageClient.ProcessRequest(Request request)
at Microsoft.ManagementConsole.Internal.IMessageClient.ProcessRequest(Request request)
at Microsoft.ManagementConsole.Executive.RequestStatus.BeginRequest(IMessageClient messageClient, RequestInfo requestInfo)
at Microsoft.ManagementConsole.Executive.SnapInRequestOperation.ProcessRequest()
at Microsoft.ManagementConsole.Executive.Operation.OnThreadTransfer(SimpleOperationCallback callback)
Considering that it may be related to DTC, I restarted the DTC and WMI service and got the wirewall settings checked which seems to be fine. I also restarted the BizTalk box but that dint help.
Note -BizTalk and SQL are in different boxes and when I try connecting to the BizTalk group from another machine it does connect works fine from there.
Has anyone faced this issue, can you please highlight what might be a resolution to this?
This specific issue post CU1 update in BizTalk 2013 R2 can be fixed by executing the following command which fixes the potential WMI namespace corruption during the upgrade :
mofcomp "D:\Program Files (x86)\Microsoft BizTalk Server 2013\Bins32\BTSWMISchema.mof".
Please refer this for more details : Null reference exception when trying to open the BTS admin console…
Related
Jscript Unexpected exceptions
After Windows Server patch Vulnerability (CVE-2019-1367) released in 23. September
Windows Server 2019 (KB4522015) https://support.microsoft.com/en-us/help/4522015/windows-10-update-kb4522015
Windows Server 2016 (KB4522010)
Windows Server 2008 R2 for x64-based Systems Service Pack 1 (KB4522007)
Windows Server 2012 (KB4522007)
Windows Server 2012 R2 (KB4522007)
Updated 07.10.2019 Also "Preview of Monthly Rollup" and "Monthly Rollup" packages are affected and doesn't fix the specific Jscript Workflow issues
Windows Server 2019: KB4516077, KB4524148
Windows Server 2016: KB4516061, KB4524152
Windows Server 2012 R2: KB4516041, KB4524156
in the classic ASP application on several workflow cases are occurring jscript Unexpected errors in server side:
Active Server Pages error 'ASP 0115'
A trappable error (C0000005) occurred in an external object. The script cannot continue running
Active Server Pages error 'ASP 0240'
A ScriptEngine threw exception 'C0000005' in 'IActiveScript::Close()' from 'CActiveScriptEngine::FinalRelease()'.
Patch
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer, aka
'Scripting Engine Memory Corruption Vulnerability'. This CVE ID is
unique from CVE-2019-1221.
https://www.cvedetails.com/cve/CVE-2019-1367/
A remote code execution vulnerability exists in the way that the
scripting engine handles objects in memory in Internet Explorer. The
vulnerability could corrupt memory in such a way that an attacker
could run arbitrary code in the context of the current user. An
attacker who successfully exploited the vulnerability could gain the
same user rights as the current user. In a web-based attack scenario,
an attacker could host a specially crafted website that is designed to
exploit the vulnerability through Internet Explorer and then convince
a user to view the website, for example, by sending an email. The
security update addresses the vulnerability by modifying how the
scripting engine handles objects in memory.
https://blog.qualys.com/laws-of-vulnerabilities/2019/09/24/microsoft-releases-out-of-band-security-updates
Patch is said to address issue in Memory management. Doesn't specify what exactly changes, what are the new limitations. But seems that it causes some side-effect failure cases.
Error nature
The errors cannot be handled by regular try-catch approach
The errors causes workflow interruptions
The exception seemingly happens only one time entering the specific workflow, and on repeated web-request for the same routine the code succeeds (Until App-pool restarted).
Sometimes exceptions is entering the workflow first, second or third time.
The exception only happens if IIS ASP Debugging Properties - Enable Server-side Debugging is set to False
Background
Verified that the issue is present on all tested Server instances with the patch. Also isolated the patch by checking the State before and after applying the patch (Server 2012 R2, Server 2016, Windows 10 - 1809)
From Classic ASP Server cannot handle the issue, with try-catch,
General error is returned - Script Error Message
Or if turned off (ASP - Send Errors To Browser) ASP error codes with page where they happen
Event Viewer also registers those errors, but without additional information
Global.asa doesn't offer global error handling, ASP Server object Server.GetLastError() doesn't catch the exception
Explored exceptions with
DebugDiag
Sysinternals Process Monitor
IIS - Failed Request Tracing
Environment
App-Pool: Classic pipeline mode, Enable 32-Bit Applications: True
Application: ASP
ClientL IE 11 Enterprise mode, with ActiveX enabled
Application pool identity is Impersonated in Web request calls
Issues, identified
1 In w3wp__V...__First chance exception 0XC0000005.dmp the assembly
instruction at msvcrt!memcpy+198 ###
in C:\Windows\System32\msvcrt.dll from Microsoft Corporation has caused an access
violation exception (0xC0000005) when trying to read from memory location 0x0000000a on thread 33
Instruction Address
Source
[0x7532a2d8] msvcrt!memcpy+198
[0x6ac17deb] jscript!AString::CopyToBuffer+4b
[0x6ac10524] jscript!AString::ConvertToBSTR+1bb74
[0x6abdf6b7] jscript!PrepareInvoke+277
[0x6abf52df] jscript!InvokeDispatch+8f
[0x6abe2f03] jscript!VAR::InvokeByDispID+523
[0x6abdbde0] jscript!NameTbl::InvokeInternal+270
[0x6abe2b17] jscript!VAR::InvokeByDispID+137
[0x6abe6083] jscript!CScriptRuntime::Run+2db3
...
Followed by - Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b7c2d77] jscript!VarStack::ScavengeRoots+27
[0x6b7c2b89] jscript!GcContext::CollectCore+79
[0x6b7c2af4] jscript!GcContext::Collect+1b
[0x6b7bca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b7a604a] jscript!CSession::Close+18a
[0x6b7a32d9] jscript!COleScript::CloseInternal+13b
[0x6b7a2d36] jscript!COleScript::Close+16
[0x6b8a71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
Not identified the exact line that causes the issue, the FailedRequestTrace last record is assigning string variable from Application Scope xml object attribute. (CurrentStatement return attrib.text)
Similar case - access violation exception (0xC0000005) when trying to read from memory location 0x00000000
[0x6b907e09] jscript!AString::CopyToBuffer+69
[0x6b900524] jscript!AString::ConvertToBSTR+1bb74
[0x6b8e49a7] jscript!VAR::ConvertASTRtoBSTR+13
[0x6b8c49e8] jscript!VAR::GetValue+58
[0x6b8e0f34] jscript!ConvertToString+58
[0x6b922fbf] jscript!JsString+4f
[0x6b8d92e6] jscript!NatFncObj::Call+e6
...
Followed by - access violation exception (0xC0000005) when trying to read from memory location 0x004e0049
[0x6b8e2d77] jscript!VarStack::ScavengeRoots+27
[0x6b8e2b89] jscript!GcContext::CollectCore+79
[0x6b8e2af4] jscript!GcContext::Collect+1b
[0x6b8dca21] jscript!GcContext::ExhaustiveCollect+21
[0x6b8c604a] jscript!CSession::Close+18a
[0x6b8c32d9] jscript!COleScript::CloseInternal+13b
[0x6b8c2d36] jscript!COleScript::Close+16
[0x6bfb71ce] asp!CActiveScriptEngine::FinalRelease+1be
...
2 In w3wp__...__Second_Chance_Exception_C0000005.dmp the assembly instruction at asp!CResponseBuffer::Write+3a
in \?\C:\Windows\System32\inetsrv\asp.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000014 on thread 32
[0x6f042e88] asp!CResponseBuffer::Write+3a
[0x6f0452ea] asp!CResponse::WriteSz+4c
[0x6f02dd3b] asp!CErrInfo::LogErrortoBrowser+ff
[0x6f02d4c9] asp!CErrInfo::LogErrortoBrowserWrapper+d7
[0x6f02d047] asp!CErrInfo::LogError+e8
[0x6f02e241] asp!HandleError+116
[0x6f02f009] asp!HandleErrorMissingFilename+df
[0x6f04941b] asp!CActiveScriptEngine::Call+bb
[0x6f030eff] asp!CallScriptFunctionOfEngine+4d
[0x6f02f99f] asp!ExecuteRequest+173
[0x6f02f828] asp!Execute+23d
[0x6f035c6f] asp!CHitObj::ViperAsyncCallback+467
[0x6f05df53] asp!CViperAsyncRequest::OnCall+73
[0x6eefd325] comsvcs!CSTAActivityWork::STAActivityWorkHelper+45
[0x77098346] combase!EnterForCallback+16e [onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2] onecore\com\combase\dcomrem\crossctx.cxx # 2072 + 2
[0x7709816d] combase!SwitchForCallback+206 [onecore\com\combase\dcomrem\crossctx.cxx # 1694] onecore\com\combase\dcomrem\crossctx.cxx # 1694
[0x7709bae4] combase!PerformCallback+bc [onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16] onecore\com\combase\dcomrem\crossctx.cxx # 1573 + 16
[0x7709b7f9] combase!CObjectContext::InternalContextCallback+119 [onecore\com\combase\dcomrem\context.cxx # 4421 + 1a] onecore\com\combase\dcomrem\context.cxx # 4421 + 1a
[0x77198e66] combase!CObjectContext::DoCallback+26 [onecore\com\combase\dcomrem\context.cxx # 4254] onecore\com\combase\dcomrem\context.cxx # 4254
[0x6eefd015] comsvcs!CSTAActivityWork::DoWork+175
[0x6eeff0e0] comsvcs!CSTAThread::DoWork+26
[0x6eeff599] comsvcs!CSTAThread::ProcessQueueWork+48
[0x6eeff8dd] comsvcs!CSTAThread::WorkerLoop+13d
[0x76577e71] msvcrt!_callthreadstartex+25
[0x76577f31] msvcrt!_threadstartex+61
[0x765f0419] kernel32!BaseThreadInitThunk+19
[0x77d5662d] ntdll!__RtlUserThreadStart+2f
[0x77d565fd] ntdll!_RtlUserThreadStart+1b
...
Most likely comes from writing to log file
ioo_fso = Server.CreateObject("Scripting.FileSystemObject");
...
loo_file = loo_fso.OpenTextFile(ls_filename,8,true);
...
try {
loo_file.WriteLine("[" + str + "]")} catch (ee) {}
Proces-Monitor shows "SHARING VIOLATION" log records for w3wp.exe on accessing the log file
3 Also experienced ASP 0115 right after custom Server custom component creation
var pbkdf2;
try {
pbkdf2 = Server.CreateObject("Pbkdf2");
pbkdf2.hashPassword(ls_newpassword, 100000);
} catch (e) {
addToLogg("Login:CreateObject failed for Pbkdf2, " + e.description);
}
from FailedReqLogFiles logs, but not identified yet in DebugDiag
Questions
I know ASP Jscript is an old, out-phasing technology, but there should be plenty of Enterprise solution still out there, so could be someone else also runs into these issues.
I would expect the Jscript to fall in regular way, so that the error situation could be handled
Has someone else come upon similar situation?
What are the new restrictions on jscript code?
Are there ways to handle these fails in server-side, prior to returning response to client?
Maybe there is some ASP/jscript environment settings, memory management settings, Windows privileges, permissions that can potentially solve the issues?
Solution
As hinted by #Max (below) the last Microsoft KBs fixes the Jscript Workflow issues.
The only comment in KB that reflects jscript is
Addresses an issue with applications and printer drivers that utilize
the Windows JavaScript engine (jscript.dll) for processing print jobs.
So apparently the fix in common jscript handling
The summary of KBs that fix the issue
Server 2019 1809: KB4519338
Server 2019 1903: KB4517389
Server 2016: KB4519998
Server 2012 R2 : KB4520005
Server 2012: KB4520007
Server 2008 R2: KB4519976
Server 2008 (with SP2): KB4520002
Not required to uninstall previous KB updates.
See that previous monthly Rollup (October 3) is no longer present in Window Updates after installing the new one.
While I didn't manage to isolate the main "First chance exception 0xC0000005" from the Workflows:
After applying the October 8 KB the issue 0xC0000005 is no longer there in the previously failing Workflows
The isolated second chance exceptions are also not reproduced, see ntdl Access Violation case, Server Object Creation case
My isolated Regex samples with test data from the failing workflow didn't reproduce the Exception, so I cannot confirm that the regex use directly caused the issues
However VBScript proved not to be affected, as pointed out by #Lee
This appears to be an unfortunate issue in the basket, while fixing memory corruption creating a several ones.
We also experienced these same errors related to CVE-2019-1367 and classic ASP. We narrowed the scope of the errors to a few places we were using JScript instead of VBScript to do JSON conversions, then we narrowed it further to were we were using regex. We worked around the errors by rewriting the functionality that was in JScript code in VBScript.
I found this article that refers to CVE-2019-13670 with a very similar number and very similar wording: Google Chrome could allow a remote attacker to execute arbitrary code on the system, caused by a V8 memory corruption in regex..
CVE-2019-1367 is specific to Internet Explorer and updated C\Windows\system32\JScript.dll. From that, I'm guessing IE's javascript engine and classic ASP JScript engine are both handled by JScript.dll? Wild guess. CVE-2019-13670 is specific to Chrome (which I assume doesn't use JScript.dll), but it mentioned regex and we found our problems to be specific to regex usage in JScript.
My group is also experiencing these issues. Our entire legacy system is written in ASP
with JScript. The KB4522007 update was installed on 2019-09-25, at which point we noticed the errors. In addition to the error mentioned in the original post, we are experiencing other errors as well:
A trappable error (C0000005) occurred in an external object. The script cannot continue running.
The function returned |.
Function expected
Variable uses an Automation type not supported in JScript
Object doesn't support this property or method
These are all errors happening in unmodified files, and were never observed prior to the update. The errors are periodic, and not systematically reproducible...
Removing the KB4522007 updated caused the errors to go away.
it seems the latest updates from Microsoft fixes the problem.
I'm on a QA environment, and can talk to the BizTalk Admin tomorrow. Apparently, I'm the first person to call a BizRule from an orchestration, and the orchestration is giving error:
Inner exception: The type initializer for 'Microsoft.RuleEngine.RuleEngineCache' threw an exception.
I checked, and the Rules Engine is configured on this machine. But the "Rule Engine Update Service" was not running. I tried to start it, and get this error in the "Services" tool:
"The Rule Engine Update Service service on Local Computer started and
then stopped. Some services stop automatically if they are not in use
by other services or programs."
I checked the event log, and for each time I tried to start it, I see this:
Service could not be started. :
System.Reflection.TargetInvocationException: Exception has been thrown
by the target of an invocation. System.InvalidCastException: Specified
cast is not valid.
Any ideas what I can do? I have admin privileges on the machine (BizTalk 2013/R2).
Our BizTalk Admin corrected the CacheEntries using RegEdit. The one on the right was the other server in the group that was working.
We have Websphere Message Broker v8.0.0.3 on Redhat Linux env, and we encounterd the following runtime error message:
BIP2057
Execution Group <insert_1> could not be started: broker name <insert_2>; UUID <insert_3>; label <insert_4>; Pub-Sub server <insert_5> (1=Yes, 0=No).
The situation is that one of the execution groups stopped (without any trace/log about it) and the wmb could not auto start with the BIP2057 error.
the weird part occurs when another EG restarted (unlinked reason to the first problem, because of some applicative errors) and then the first EG "took its place", started successfully, and now the second EG could not start for the same reason.
We have checked for user permissions or the logs without any success in identifying the problem.
Any help would be much appreciated.
Run the following service IBM MQSeries if it's not running already:
C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqlsvc.exe
If it gives an
exception of Access Denied, your user must not be having rights to
log on.
Another possible resolution can be by restarting the
DataFlowEngine.exe process.
If none of the above scenarios fulfill the need, restart the system.
I have an odd BizTalk issue that only occurs in the Production environment. It started over the weekend. The error message from Event Viewer is at the end of this post.
I researched this online and all the possible solutions proposed by the gurus did not do the trick. On a lark, I exported the MSI from that Production BizTalk server and installed that MSI on a development machine. That Development BizTalk worked beautifully with the same file against a different SQL Server database (development DB).
Next, I went ahead and uninstalled and deleted that BizTalk application from the Production BizTalk server and re-imported & re-installed the same MSI (that was used on the aforementioned Development BizTalk machine). The same error occurred.
That leads me to ask what is probably a stupid question... would rebooting a production SQL Server (which more or less is verboten unless there are critical upgrades that need to be applied) solve this? The reason I ask this is because if the exact same MSI from the production BizTalk server worked elsewhere, but does not work on that one production BizTalk server.
Any ideas? Without further ado, here's the error message.
xlang/s engine event log entry: Uncaught exception (see the 'inner exception' below) has suspended an instance of service 'Xpress.BizTalk.P837.EDI_837P_5010_to_WorkTable_And_4010(dee9b36c-4d44-20a1-01b0-efc184ec3efb)'.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: d768edda-35ba-4bb1-a57d-347cc54ddf49
Shape name: Send 837P 4010
ShapeId: dcb098dc-9c6b-41b0-8511-c9a250de52b2
Exception thrown from: segment 1, progress 94
Inner exception: Exception occurred when persisting state to the database.
Exception type: PersistenceException
Source: Microsoft.XLANGs.BizTalk.Engine
Target Site: Void Commit()
The following is a stack trace that identifies the location where the exception occured
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.Commit()
at Microsoft.XLANGs.Core.Service.Persist(Boolean dehydrate, Context ctx, Boolean idleRequired, Boolean finalPersist, Boolean bypassCommit, Boolean terminate)
at Microsoft.XLANGs.Core.ServiceContext.PendingCommit(Boolean ignore, XMessage msg)
at Microsoft.XLANGs.Core.ExceptionHandlingContext.PendingCommit(Boolean ignoreCommit, XMessage msg)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXPortBase.SendMessage(Int32 iOperation, XLANGMessage msg, Correlation[] initCorrelations, Correlation[] followCorrelations, Context cxt, Segment seg, ActivityFlags flags)
at Xpress.BizTalk.P837.EDI_837P_5010_to_WorkTable_And_4010.segment1(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)
Additional error information:
A batch item failed persistence Item-ID 433c9f39-f2c7-4ac3-ae29-32b990b20bdd OperationType MAIO_CommitBatch Status -1061151992 ErrorInfo The operation could not be performed because the message on which it was acting was not found in the database. .
I'm not quite sure what's going on, but several issues are occurring on our website supported by Tridion 2011 SP1 that I can only think is being caused by the broker db
1, Loading the website results in the error with the Ambient Data Module:
[NullReferenceException: Object reference not set to an instance of an object.]
Tridion.ContentDelivery.AmbientData.HttpModule.OnRequestStart(Object sender, EventArgs e) +292
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +79
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +269
2, Audience manager profile synchronisation between the broker and subscription management databases is failing, receiving the following errors from the logs:
Error occurred while fetching a synchronization batch from the presentation system with url: [URL] Profilesync.aspx. Message class java.sql.SQLException No Data Access Object for AudienceManagerProfile java.sql.SQLException: No Data Access Object for AudienceManagerProfile
3, Publishing the website publication repeatedly fails at the transport stage, returning:
Transport failed: Could not transport tcm_0-10689-66560.Content.zip using HTTPS
I know these sound like several issues happening at once, but my site was running fine up until another web publication was added into blueprinting. Since then we've been getting these errors, and like to think it's all related to an issue with the broker db.
Anyone come across something like this?
UPDATE: Should also add the fact that dynamic linking has stopped working as well, which strengthens my belief there is an issue either connecting to or with the broker database
That's full of loaded questions, oh.. boy.. let's take one by one.
Did you check your license file is valid and did not expire recently.
I would rule the license file out first. If that's not the case then see below.
First, Adding publication blueprinting publication does not have any impact on CDA site. So, I would totally rule this out.
Second, Your site up and running until publication added.. This can't be happening. There should be some configurations changed/added on your web site. Sometimes, the configurations updated but the APPPool did not recycled so you don't see the impact. At later time, when the AppPool restarted/recycled you will notice the errors and might have the impression that the site is breaking all of sudden.
Did you double check your configs and dlls?
Are you able to connect to your database directly with broker user
id/password?
Any firewall changes happened recently, try connecting to DB from
CDA Server
For #3, I had this same problem... is it possible that adding the new publication caused the package size to increase from less than 30mb to more than 30mb? If so, check IIS Request Filtering >> Edit Feature Settings and look at Maximum allowed content length. This defaults to 30MB and for us increasing this size to something larger than our package size solved the Transport Failed error (since IIS would just reject our transports due to the size being greater than 30mb).
About issue #1: Could it be you have the Tridion.ContentDelivery.AmbientData.HttpModule configured in the Web.config, but are lacking the java installation? Do you have a cd_ambient_conf.xml in your config folder? And a cd_ambient.jar in the lib?
About issue # 2: you seem to be missing the audience manager DAOs (Data Access Objects). Do you have the following in your cd_storage_conf.xml?
<StorageBindings>
<Bundle src="AudienceManagerDAOBundle.xml"/>
</StorageBindings>
About issue # 3: do you have more information in the cd_transport.log or windows event viewer?
I agree with Ram that the added publication couldn't have caused this. It looks like your CD installation was changed somehow. Did you do an upgrade or something?