Scenario:
A biztalk application is deployed with a receive port, orchestration and send port. Messages flow correctly.
At some point, a bug is found in the orchestration, causing messages to suspend. The orchestration must be fixed and redeployed.
Question:
Because you can't redploy an orchestration with suspended instances, how would you go about retaining those messages, terminating the instances, redeploying and then resending those messages through the fixed orchestration? Is there a process or tool for this?
If the bug doesn`t require major modification - i.e no new orchestrations, no new schemas, no new promoted fields etc, then a short term 'hack' is possible, viz by simply reinstalling the fixed MSIs (and GAC) on your servers, and restarting the host instances (using NLB if applicable) (i.e. without importing the MSI's into BizTalk).
You should then be able to resume any suspended (resumable) orchs. Then schedule some downtime at a less busy time, put your app into partially stopped to prevent new orchs starting, wait for all running orchs to complete, and then import the fixed MSI (consider bumping up the buggy orch assembly version with the hotfix)
Building a custom tool with the ability to audit all messages going in and out of Biztalk is useful, so you can replay them. This will allow you to terminate orchs, reinstall, and then replay.
You can as well fix the orchstration and while building it, increment the version of the assembly. This way you can have parallel deployment of the orchestrations. You can unenlist the existing one after deploying the newer versioned orchstration.
Related
We are using BizTalk (2013 R2 CU 6) EDI functionality to batch EDI files. This uses the Microsoft.BizTalk.Edi.BatchingOrchestration.BatchingService Orchestration which is always running in a waiting state (for lack of a better term), dehydrated most of the time. While running the orchestration builds up instances of the BTXTimerMessages in the Queued (awaiting processing) state. These messages are never removed or processed, that I can tell. This eventually causes us to pass the 50k message threshold and start throttling.
As far as I can tell there is no way to setup a reoccurring schedule for the batcher, it must always run or be manually started. If we leave the batcher off there we get routing errors.
Currently the only way we have to eliminate these messages is to terminate the EDI batcher for each party, then restart it.
Is there a better way to purge these messages from the system, or stop them from being generated all together?
Had an orchestration following the parallel convoy pattern to receive and wait a bunch of messages before kicking off the flow. Question is we are to move to an high variability environment with multiple active-active biztalk servers, would this be a problem? will the messagebox smart enough to figure out which host instance it should push the message to?
It's no problem at all. Don't even think about it.
An Orchestration can run an any available Host Computer but a specific instance of an Orchestration would only run on one at a time.
It doesn't matter which one.
Requirement: Updating BizTalk application to a new version
Problem: The MSI import does not go through if there are running/suspended instances. Termination would result in loss of messages
What did I try:
I had about 100+ messages in messagebox some active, some with suspended resumable status.
I took the back up of BizTalkMsgBoxDb, I then terminated all instances from BTAdmin console and then restored the BizTalkMsgBoxDb.
I was expecting the messages to be back in BizTalkMsgBoxDb but when I queried from BiztalkAdmin console I don't find any of the message back.
Did I miss anything?
if your changes do not contain any changes to ports etc. try and replace the assemblies in GAC and then restart your host instances.
Doing a backup of just one of the BizTalk databases and restoring it is a very dangerous practice and I would strongly advise against it as it can cause some very nasty side effects.
The normal process of a deployment would be to switch of the receive locations and allow any running processes to finish and to resume or terminate any messages/orchestrations as appropriate.
Once there were no longer any suspended and running processes/messages would you unenlist all the Orchestrations and do the deployment.
If there are some long running processes that cannot be completed or terminated inside the deployment window then you would have to look at doing a side-by-side deployment. That involves changing the version number of all the DLLs, deploying this and then switching of the receive locations of the old version and switching on the new one.
When the old version has finished you stop that and un-deploy it.
I'm seeing a really strange error that I'm having a difficult time
tracking down. I think its related to my configuration of Rhino ESB, though I'm not sure
if RSB is actually causing it, so I figured I'd ask and see if
anyone else has come across this in any other usages of MSMQ.
I'm using RSB as a client in a web app (ASP.NET, the client runs in the background). The client talks to a windows service via the MSMQ binding for RSB. Restarting the service never appears to have an effect on MSMQ, neither does restarting IIS by hand. However, whenever I actually restart the computer itself, MSMQ always refuses to start back up, claiming that a "queue is in an inconsistent state". Attempting to start MSMQ manually results in the same error, effectively rendering the MSMQ install completely useless. The only way to solve it is to actually remove then reinstall MSMQ.
The only information I've found via the almighty Google are references to a problem in MSMQ 2.0 (this problem is occurring in MSMQ 4.0). I've verified that Dispose is being called on on the bus at shutdown, in both the service and the web site.
Does anyone have any idea why this could be occurring? Thanks!
I faced the same issue on Window 2008 Server (Virtual Machine). Although the environment was not related to rhino tools.
The error in the event log:
"The Message Queuing service cannot start because a queue is in an inconsistent state. For more information, see Microsoft Knowledge Base article 827493 at support.microsoft.com."
As Roy pointed out, this is happening every 2-3 days. Every time we would follow the steps below to recover - instead re-installing the MSMQ.
1) Stop all applications and services that uses MSMQ.
2) Kill the mqsvc.exe from the Task Manager
3) Go to C:\Windows\System32\msmq\storage and delete any .mq files
4) Start the MSMQ Service
4) Start your application
In my scenario I've been able to fix "queue is in an inconsistent state" error after MSMQ service restart.
Turns out the computer name was too long, so changing computer name to a name with less than 15 characters fixed the issue.
My team is experiencing a similar issue, with MSMQ getting called by NSB 2.5. The issue came up recently after Infrastructure moved our VM to another physical server and for some reason lowered available RAM. We think the issue may be memory-related.
EDIT
After a week of no more issues with this, I can confidently say that raising RAM on the server solved our MSMQ's "Inconsistent state" issue. Mind you, we did have to re-install MSMQ first -- but the issue never came back, and before the RAM update the issue popped up every 2 days.
Regularly on Windows 2008RC2, MSMQ cannot start after reboot.
The two regular issues for me are:
"The Message Queuing service cannot start because a queue is in an inconsistent state"
and
"The dependency service does not exist or has been marked for deletion"
Sometimes, the following has helped (although we are seeking a more solid answer)
rename msmq folder to msmq_old
net stop wuauserv
net stop bits
Delete “%windir%\softwaredistribution” directory
Reboot
This has occurred 5 times this year, and each time, a variety of the above with plenty of reboots.
Sometimes, we revert to Remove Feature / Add Feature, however you may get yourself in a loop. As it boots up, a rollback occurs in the windows update service, so the Feature is never uninstalled, and the problem is never repaired.
Following the steps above can help with that.
I have an application with a file receive location. After the host instance has been running for a few hours the receive location fails to identify new files dropped into the folder that it is monitoring. It doesn't forget about them altogether, it's just that performance grinds to a crawl. The receive location is configured to poll the target folder every 60 seconds but after host instance has been running for an hour or so, then it seems that the target folder is being polled only every thirty minutes. If I restart the host instance then the files waiting in the target folder are collected right away and performance is fine for the next hour or so.
The same application runs fine in a different environment.
There are now obvious entries in the event log related to the problem.
All the BizTalk SQL jobs are running fine except for Backup BizTalk Server (BizTalkMgmtDb).
Any suggestions gratefully received.
Thanks
Rob
Here are some additional tools which may help you identify and diagnose BizTalk database issues.
BizTalk MsgBox Viewer
Here is a tool to repair identified errors:
Terminator
Use at your own risk... read the glogs and docs. Start with the message box viewer and let us know our results.
Without more details, the biggest tell is that your Backup Job is failing. If the backup job is failing, it may not be properly configured. If it is properly configured and still failing, then you've got other issues. Can you give us some more information about your BizTalk install.
What version are you running?
What are our database sizes?
What are your purge and archive settings like?
Is there any long running blocks in your SQL Server DB coming from BizTalk?
Another thing to consider is the user accounts the send, receive and orchestration hosts are running under. Please check the BizTalk Administration Console. If they are all running the same account, sometimes the orchestrations can starve the send and receive processes of CPU time. I believe priority is given to orchestrations then receive, then send. Even if you are just developing, it is useful to use separate accounts for this. This also improves security.
The Wrox BizTalk Server 2006 will also supply tuning advice.
What other things are going on with the server? Is BizTalk pegged otherwise or is it idle?
You mention that the solution does not have any problems in another environment, so it's likely that there is a configuration problem.
Check the following:
** On SQL Server, set some upper memory limit for SQL Server. By default, SQL Server uses whatever it can get and then hangs onto it, so set a reasonable limit so that your system can operate without spending a lot of time paging memory onto and from your hard drive(s).
** Ensure that you have available disk space - maybe you are running low - this can lead to all kinds of strange problems.
** Try to split up the system's paging file among its physical drives (if you have more than one drive on the system). Also consider using a faster drive, or if you have lots of cash laying around, get a SAN.
** In BizTalk, is tracking enabled? If so, are you also tracking message bodies? Disable tacking or message body tracking and see if there is a difference.
** Start performance monitor and monitor the following counters when running your solution
Object: BizTalk Messaging
Instance: (select the receiving host) %%
Counter: Documents Received/Sec
Object: BizTalk Messaging
Instance: (select the transmitting host) %%
Counter: Documents Sent/Sec
Object: XLANG/s Orchestrations
Instance: (select the processing host) %%
Counter: Orchestrations Completed/Sec.
%% You may have only one host, so just use it. Since BizTalk configurations vary, I am using generic names for hosts.
The preceding counters monitor the most basic aspects of your server, but may help to narrow down places to look further. You can, of course, add CPU and Memory too. If you have time (days...maybe weeks) you could monitor for processes that allocate memory and never release it. Use the following counter...
Object: Memory
Counter: Pool Nonpaged Bytes
Slow decline of this counter indicates that a process is not releasing memory, which affects everything on the system.
Let us know how things turn out!
I had the same problem with, when my orchestration was idle for some time it took a long time to process the first msg. A article of EvYoung helped me solve this problem.
"This is caused by application domain unloading within the BizTalk host process. If an AppDomain is shutdown after idle, the next message that comes needs to wait for the Orchestration to compile again. Depending on the complexity of your design, this can be a noticeable wait. To prevent this in low latency requirement scenario, you can modify the BTSNTSVC.EXE.config file and set SecondsIdleBeforeShutdown property to -1. This will prevent AppDomain shutdown due to idle."
You can find the article in here:
http://blogs.msdn.com/b/biztalkcpr/archive/2008/05/08/thoughts-on-orchestration-performance.aspx
It took me to long to respond but i thought i might help someone. cheers :)
Some good suggestions from others. I will add :
Do you have any custom receive pipeline components on the receive location ? If so perhaps one is leaking memory, calling some external component eg database which is taking a long time ?
How big are the files you are receiving ?
On the File transport properties of your receive location, set "file renaming" on, do the files get renamed within 60s.