I have a BizTalk solution that would call a stored procedure that will query and update table. Apparently I'm encountering this error each time I test my solution in a load balanced servers.
The adapter failed to transmit message going to send port SQLAdapter_Sendport" with URL "mssql://server//StoredProc?". It will be retransmitted after the retry interval specified for this Send Port.
Details: System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed.
System.Runtime.InteropServices.COMException: The MSDTC transaction manager was unable to push the transaction to the destination transaction manager due to communication problems. Possible causes are: a firewall is present and it doesn't have an exception for the MSDTC process, the two machines cannot find each other by their NetBIOS names, or the support for network transactions is not enabled for one of the two transaction managers.
Exception from HRESULT: 0x8004D02A)
at System.Transactions.Oletx.ITransactionShim.Export(UInt32 whereaboutsSize, Byte[] whereabouts, Int32& cookieIndex, UInt32& cookieSize, CoTaskMemHandle& cookieBuffer)
at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
--- End of inner exception stack trace ---
I've read some articles that I should set the DTC properties as below. These I've already set in both BizTalk and SQL Servers.
Also the DTC must be added in the Windows Firewall.
And lastly if the useAmbientTransaction in the Send Port properties is set to True I keep encountering the said error above.
But if I set the useAmbientTransaction to False the message is succeeding in SQL.
I've read in some article that Microsoft doesn't recommend setting the useAmbientTransaction to False especially if there's Update and Delete involve in the stored procedure. But my solution uses delete and update in the stored procedure that's why I need to set it the useAmbientTransaction to True. But if I do that I keep encountering the error "MSDTC transaction manager was unable to push the transaction to the destination..."
Please suggest any possible solution for this.
If you are connecting to a database in a AlwaysOn Availability Group then you need to switch off useAmbientTransaction as the AlwaysOn do not support MSDTC except for SQL Server 2016.
The same article for SQL Server 2014 Cross-Database Transactions Not Supported For Database Mirroring or AlwaysOn Availability Groups (SQL Server)
vs
SQL Server 2016 Cross-Database Transactions and Distributed Transactions for Always On Availability Groups and Database Mirroring (SQL Server)
Related
I am using Gremlin Java driver to connect to a local gremlin server.
Simple code flow
Creating client
Cluster cluster = Cluster.build().addContactPoint(<endp>).port(<port>).enableSsl(false).create()
Client client = cluster.connect();
Submit Script
client.submit("g.V().count()");
If when i submit the first time the Gremlin server is down, on subsequent retries after bringing back gremlin server, connection still fails to create.
Exception First attempt when Gremlin Server is down:
org.apache.tinkerpop.gremlin.driver.exception.NoHostAvailableException: All hosts are considered unavailable due to previous exceptions. Check the error log to find the actual reason
Exception After Gremlin server is brought back up:
tinkerpop.gremlin.driver.exception.NoHostAvailableException: All hosts are considered unavailable due to previous exceptions
One thing to note is i do not create client on retry just do
Submit Script
client.submit("g.V().count()");
It is quite possible that Gremlin server may go down anytime, how to recover in such circumstances. Fundamentally is
NoHostAvailableExceptio
recoverable?
Normally, the Client should attempt to reconnect to a host that is previously marked unavailable. By default, it should be retrying the host at 1 second intervals as governed by this configuration: connectionPool.reconnectInterval. In your case, however I think you've uncovered a bug where the reconnect attempts are not started because the Client was never able to reach the host in the first place. As of 3.4.11, you can only remedy this by recreating the Client as you noted in your comments. I've created an issue to track this problem here: TINKERPOP-2569
In datapower, the operational state of queue manager object is pending. The information provided for this operational state is as follows : "This message indicates that the configuration of the object has changed, but has not been committed and has yet to take effect. No user intervention is required." What is exactly causing this problem and how can this be resolved?
If it is "pending" for a MQ QM object it means that DataPower is trying to figure out if it has a connection to it or not.
Normally if a QM object is in "pending" for a while, more than 20 seconds, it would mean that it didn't get the connection.
Check the System log and you'll probably see a ton of connection errors to the QM server.
First go to Troubleshooting from the Control Panel and do a TCP test to make sure you have a connection to the MQ server using the IP and port of the listener on the QM.
If you get a connection then check the MQ logs for any authentication issues, eg. user and/or auth-records. You need a Server-Connection channel for DataPower!
If you don't get a connection in TCP test then check your firewalls and also make sure that the DataPower network is setup correctly if you have multiple network cards (NIC) setup and set a static route for the MQ on the correct NIC.
I have an WebAPI service using ODP.NET to make connections to several oracle databases. Normally the web service would be hit several times a second and will never have long periods on inactivity. In our test site however, we did not use it for 2-3 days. This morning, we hit the service and got "connection request timeout" exceptions from ODP.NET, suggesting that the connection pool was out of available connections. We are closing the connections after use. The service was working fine before the period, but today the very first query got the timeout exception. Our app pool in IIS is configured to never reset.
My question then is, what can cause the connection pool to fill with bad connections after a period of inactivity, where these connections are not cleaned up in the usual 3 minute cycle? It only happened to 2 out of the 3 of our databases, and Validate Connection=true is set for all of them.
EDIT
So after talking to the DBA, there is some different between a connection/session being killed manually or by timeout and the database server severing the TCP connections. In this case, the TCP connection was severed as part of a regular backup (why is not important for this). I guess this happens when the whole database server goes offline at once. The basis of the question still applies I think though: why is ODP.NET unable to cleanup severed connections overtime? There is a performance counter that refers to "Stasis" connections, could those connections be stuck in that state? I would think that it should be able to see that a connection is no longer active (Validate Connection=True), kill it and not return it to the pool.
Granted, this problem can be solved by just resetting the app pool everything the database goes down. I would still like to configure ODP.NET connection pooling to be more fault tolerant.
I have run into this same issue, and the only solution I have found is to use the Connection Lifetime connection string parameter in conjunction with Validate Connection.
In my particular case, the connection timeout was set at the server and the connections in the pool would timeout, but not be sniped out of the pool, resulting in errors.
Setting both the Connection Lifetime and the Validate Connection parameters has resolved the issue.
Make sure the Connection Lifetime value that you choose is less than the server connection inactivity timeout.
The recommended solution is to use ODP.NET Fast Connection Failover (FCF). FCF will automatically remove invalid connections from the pool such that you don't need to use Validate Connection, Connection Lifetime, nor clear the pool.
To use FCF, set "HA events=true", use connection pooling, and have your DBA set up Fast Application Notification (FAN) on the server side. FAN is what alerts the ODP.NET pool when a DB service or node goes down or rebooted. Upon receiving the message, ODP.NET knows which connections to remove from the pool and removes them, leaving all other valid connections untouched.
Something else is going on here. Min Pool Size and some of the other settings help when the connection is severed from things like DBA configured idle timeouts and firewall tcp idle timeouts, 'connection request timeout' occurs when created a new connection.
This could be simple network problem. There could be something interfering with dns resolution of the servers. Another case is not having fully qualified entries in tnsnames. I've been bit by the latter a couple of times.
The other issue is the one you've already recognized - full pool.
Double check that you don't have a connection leak somewhere. A missing .Close is one thing but if you're not using a 'using' statement, a try/finally is required as an unhandled exception could be thrown prior to the .Close.
I would use perfmon to monitor some of the connection statistics to start - NumberOfPooledConnections, NumberOfActiveConnections, etc:
I am having problems testing the MQSC Adapter in BizTalk to communicate with a Queue on Z/OS Host.
The test senario: While sending messages though Biztalk I (Force)stop and start the Host channel, to mimic a HOST IPL.
In this scenario I experienced the following outcomes:
Sometimes there are messages left uncomitted on the Host MQ.
Even after the channel are started again. This way I don't know if the messages have been processed by Biztalk.
Question Is there a way to make sure that messages are always committed on the Host MQ or still available on the MQ if not committed
Once I experienced duplicate messages and once that messages was lost
Since these things happened only once each. I hope to find problems in my test applications / procedure.
Not really a question, but if others have experienced the same errors I would like to know.
The test Setup
Biztalk 2009 on 2 nodes
WebSphere MQ client 7.5.0.2 (Transactions supported)
MSDTC - XA Transactions are enabled (Edited to reflect comments)
MS Biztalk Adapters for Host Systems 2.0
Port Configuration
Receive Location
Send Port
If you are using MQ transactions, that is just MQPMO_SYNCPOINT and MQCMIT, then the disconnection (either explicitly or implicitly due to STOP CHANNEL(chl-name) MODE(FORCE)) will cause any outstanding transaction to be completed (committed on an explicit disconnection, rolled-back on an implicit one).
If you are using 2-PC transactions with a Transaction Manager (TM), then the transaction will need to be dealt with by the TM, depending on how far through it is, is it prepared already for example.
I have a Windows 2003 server where MSDTC is running. I have set it to No Authentication Mode, with allow inbound-outbound settings. My MSDTC works but it fails for the 1st transaction of the day. On immediate another transaction it will start working.
Error is : The transaction has already been implicitly or explicitly committed or
aborted (Exception from HRESULT: 0x8004D00E).
So I started MSDTC tracing, in trace file it shows this :
pid=2144;tid=2528;time=12/02/2011-10:49:39.140;seq=531;eventid=TRACING_STARTED;;"MSDTC is resuming the tracing of long -lived transactions"
pid=2144;tid=2528;time=12/02/2011-10:49:39.140;seq=532; eventid=TRANSACTION_BEGUN; tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441;"transaction got begun, description : ''"
pid=2144;tid=3288;time=12/02/2011-10:49:39.140;seq=533 ; eventid=RM_ENLISTED_IN_TRANSACTION; tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441 ;"resource manager #1002 enlisted as transaction enlistment #1. RM guid = '4e45a393-b02a-42bf-8f66-62bcb17fee8e'"
pid=2144;tid=4164;time=12/02/2011-0:49:58.390;seq=534; eventid=TRANSACTION_PROPOGATION_FAILED_CONNECTION_DOWN_FROM_REMOTE_TM ;tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441 ;"failed to propogate transaction to child node 'DBSERVER' because the connection with the remote transaction manager went down"
pid=2144;tid=4164;time=12/02/2011-10:49:58.390;seq=535; eventid=TRANSACTION_ABORTING;tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441;"transaction is aborting"
pid=2144;tid=4164;time=12/02/2011-10:49:58.390;seq=536; eventid=RM_ISSUED_ABORT;tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441 ;"abort request issued to resource manager #1002 for transaction enlistment #1"
pid=2144;tid=2528;time=12/02/2011-10:49:58.422;seq=537; eventid=RM_ACKNOWLEDGED_ABORT;tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441;"received acknowledgement of abort request from the resource manager #1002 for transaction enlistment #1"
pid=2144;tid=2528;time=12/02/2011-10:49:58.422;seq=538; eventid=TRANSACTION_ABORTED;tx_guid=4df1b0cf-26a0-43ba-8f41-965d80f92441 ;"transaction has been aborted"
pid=2144;tid=3640;time=12/02/2011-10:50:29.437;seq=539;eventid=TRACING_STOPPED;;"MSDTC is suspending the tracing of long - lived transactions due to lack of activity"
I have applied hack of Davy Brion from here, http://davybrion.com/blog/2010/03/msdtc-woes-with-nservicebus-and-nhibernate/
Also set timeout interval to 10 minutes in Transaction Options.
If the server remains idle for a while again the transaction will fail.
Thanks in advance..
Here is the KB article that helped: 922430
If the MS DTC transaction trace log file contains this data, follow these steps:
Click Start, click Run, type regedit, and then click OK.
Locate the following registry subkey:
HKEY_LOCAL_MACHINE\Software\Microsoft\MSDTC
Right-click MSDTC, point to New, and then click DWORD Value.
Type CmMaxNumberBindRetries, and then press ENTER.
Right-click CmMaxNumberBindRetries, and then click Modify.
Click Decimal.
In the Value data box, type 60.
This value increases the length of time that the client computer waits for the bind packet response from the server computer. This value is double the number of seconds before the client computer stops the transaction if the client computer does not receive the bind packet response. For example, a value of 60 equals 30 seconds.
Note The value of 60 is only a recommended value. Additional testing on your configuration may be required.
Click OK.
Restart MS DTC.
Note For the slow response scenario, make sure that the ports that are required by Kerberos authentication (UDP 88 and TCP 88) are open when a firewall is involved in the Perimeter Network. The ports UDP 389 and TCP 389 (both for LDAP to find KDC) must also be open.