BizTalk 2013 R2 XLANG - Exception occurred when persisting state to the database - biztalk

Question

I am working on BizTalk 2013 R2. I have an orchestration where I am calling the helper class to call to Web service (for lower latency). The class is Serializable class. The Orchestration builds successfully. And when I tested the solution from the POSTMan it works fine.
But for every success response, I am also getting below exception in the event log.
xlang/s engine event log entry: An unrecoverable exception (see the 'inner exception' below) has occurred.
Service Name: Orchestration Name
Service Id: df34c579-a9e6-5696-e322-b80210d1f723
Instance Id: 0512ae45-c89c-4ffc-9bd2-332b854d4965
An exception occurred when persisting state to the database.
Exception type: PersistenceException
Source: Microsoft.XLANGs.BizTalk.Engine
Target Site: Void Commit()
The following is a stack trace that identifies the location where the
exception occurred
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.Commit()
at
Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.ScheduleComplete(Boolean terminate)
at Microsoft.XLANGs.Core.Service.Persist(Boolean dehydrate, Context ctx, Boolean idleRequired, Boolean finalPersist, Boolean bypassCommit, Boolean terminate)
at Microsoft.XLANGs.Core.ServiceContext.OnCommit()
at "OrchestrationInstanceName".segment0(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)
Additional error information:
A batch item failed persistence Item-ID 9b508582-438f-442d-a320-3ff7d9dd87ed OperationType MAIO_CommitBatch Status -1061151938 ErrorInfo A database failure occurred due to an unexpected failure. .
Exception type: PersistenceItemException
There was the database error accompanied with this persistence error.
The following stored procedure call failed: " { call [dbo]. [bts_UpdateMsgbox_XXXXXX]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}". SQL Server returned error string: "Warning: The join order has been enforced because a local join hint is used.;Warning: The join order has been enforced because a local join hint is used.;Duplicate key was ignored.".
I have added system.diagnostics.trace in shape calling .net helper class. Then I noticed the orchestration running twice on the first run it throws the Xlang - Persistence error and database - "Duplicate key was ignored" the error and then the orchestration runs for the second time and it completes successfully with the response.
It is evident that my orchestration runs twice. Then I added scope around the shape calling helper class. The transaction type of the scope is "None".
Then the Xlang Persistence error issue was resolved.
Why do I need to put extra scope around the calls to make this work without error?

Answer 1

Errors like that usually indicate that your BizTalkMsgBoxDb suffers from bad performance.
Initially I recommend running BizTalk Health Monitor which will point out the most obvious reasons. Most likely there are plenty of zombie/orphaned messages which this tool also can help to clean out.
Also take a look at the database optimization:
incorrect database settings
large index fragmentation
large amount of suspended instances - these should be terminated regularly
(please refer to https://msdn.microsoft.com/en-us/library/cc296892%28v=bts.10%29.aspx for details)
Consider also:
Reducing time for tracking history https://msdn.microsoft.com/en-us/library/aa558715.aspx
Reducing amount of tracked artifacts and events - turn off non-essential (Best practice for production is not have any at all)
If event that won't help, you'll need to evaluate your database server disk performance. You might need to migrate to faster storage