My development team is experiencing numerous ORA-12571: TNS:packet writer failure errors using ASP.NET 3.5 and 4.0 against Oracle 11g. These errors are inconsistent as to when they occur, and are generated by numerous applications. This exception happens while calling random stored procedures, packets, and inline SQL statements. The Oracle 11 client is installed on the web server. Some applications use Microsoft System.Data.OracleClient to connect to Oracle, and some use the .NET components provided by oracle (ODP.NET). Both data access objects come up with the same error.
There are other non .NET applications that run on a different web server, but use the same database server. The apps do not have any such issues. My initial thinking is that there is something configured incorrectly on the web server with the Oracle client.
Has anyone else received this error? What did you do to fix it?
ORA-12571: TNS:packet writer failure
Stack Trace:
at System.Data.OracleClient.OracleConnection.CheckError(OciErrorHandle errorHandle, Int32 rc)
at System.Data.OracleClient.OracleCommand.Execute(OciStatementHandle statementHandle, CommandBehavior behavior, Boolean needRowid, OciRowidDescriptor& rowidDescriptor, ArrayList& resultParameterOrdinals)
at System.Data.OracleClient.OracleCommand.Execute(OciStatementHandle statementHandle, CommandBehavior behavior, ArrayList& resultParameterOrdinals)
at System.Data.OracleClient.OracleCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.OracleClient.OracleCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, String srcTable)
Another possible solution is that the firewall between you and the Oracle database thinks your connection is dead and closes it underneath you. You will only find out when you try to execute a query and get the ORA-12571 error.
This is caused by having connections open for a long time with no activity.
The solution is to add the SQLNET.EXPIRE_TIME to the sqlnet.ora file on the server and set it to some interval (10). This will cause the connections to be pinged every 10 minutes to ensure they are still alive.
The result of this is that your firewall will see network activity and not close the connection.
SQLNET.EXPIRE_TIME=10
ORA-12571: TNS:packet writer failure - One of the hardest problems I've had to resolve
I think this is a bug in Oracle. I came across many issues with the DBDataAdapter.Fill method where the Oracle Client would choke on a memory error. This was resolved for me by using the 11.2.0.2 client with patch 6 applied.
If you search Oracle's Support site you will see many issues like this.
Check also on "Read Protected Memory" issues with the 11g1 / 11g2 clients.
After I installed the elmah module and could analyze the exceptions, I tried to:
Change the connection configuration.
Remove and/or update the server firewall rules.
Update the Oracle client on the server machine.
Any of the options above resolved the problem, but I was forgetting the obsolete provider (System.Data.OracleClient) the we were using. After I replaced it with the last version of the ODP.NET (Oracle.DataAccess), everthing started to work flawlessly.
Obs: Based on your exception description you are currently using the obsolete provider.
Related
We have a backup system in our company, those backups go to 3 different locations. For some reason one location called "Hades" has an error sometimes.
I would be happy if anybody knows what could be the problem
Error while backuping (name of the program that we want to make a backup for) on Qnap Hades: System.IO.IOException: The network is busy. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost) at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite) at ERPBackUp.Program.<>c__DisplayClass5_0.b__1()
As the error message states there is a network error and that can always happen.
The only option usually is to retry it again after some time.
Another reason might be that it's just the QNAP server which is not answering, i.e. because it's busy with some stuff. Then the problem might be that the server should get the running software reduced, down to the important things. I know QNAP offers a bunch of things to run on the servers and that can even be increased by some software packages. In this case it might be advisable to uninstall some things.
Also increasing the swap partition and increasing memory might help.
We're using Azure KeyVault since a few weeks and last week we suddenly had a few services fail with this error message:
System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
The funky thing is that we have multiple web apps running on the same server. For some it works and for some it doesn't. Also, it was working for all of them for some time. And some came back, others not.
Any idea what this could be? I don't think our IP is blocked or WebSockets are not configured, otherwise it would not work for all apps. The only difference is that the currently working apps are .NET and the non-working ones .NET Core. But as I said, even .NET Core was working until Friday. All services do get the same secrets (E.g. same Database login data) and every runs on Azure.
I've seen some other posts about this issue, but the only solution was that they built a caching layer on top of it. Which seems more like a bandaid and not a solution.
For that issue I've opened a ticket at the Microsoft support.
The final recommendation they gave me is downgrading the library Microsoft.Extensions.Configuration.AzureKeyVault to version 2.2.0. Before we used 3.1.3.
That worked well so far. We hadn't have that exception anymore.
I would appreciate, if anyone could share her/his experience, whether newer versions of that lib are working properly as well.
The issue is still there with the latest version 3.1.8 of library Microsoft.Extensions.Configuration.AzureKeyVault. Downgrading to 2.2.0 worked for me as well.
I have a Windows 2008 R2 server that hosts many back end NServiceBus endpoints. All of the services that rely on the NServiceBus.Host.exe host (installed as Windows Services) are able to interact with MSDTC perfectly, averaging a small handful of concurrent distributed transactions throughout the day. There are 2 small Web.API applications, however, that self host NServiceBus endpoints (as publishers) that constantly receive the following error when trying to process subscription requests:
NServiceBus.Transports.Msmq.MsmqDequeueStrategy Error in receiving
messages. System.Transactions.TransactionAbortedException: The
transaction has aborted. --->
System.Transactions.TransactionManagerCommunicationException:
Communication with the underlying transaction manager has failed. --->
System.Runtime.InteropServices.COMException: The Transaction Manager
is not available. (Exception from HRESULT: 0x8004D01B) at
System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String
nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier,
Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle&
whereaboutsBuffer, IResourceManagerShim& resourceManagerShim) at
System.Transactions.Oletx.DtcTransactionManager.Initialize() ---
End of inner exception stack trace --- at
System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException
comException) at
System.Transactions.Oletx.DtcTransactionManager.Initialize() at
System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at
System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions
properties) at
System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction
tx) --- End of inner exception stack trace --- at
System.Transactions.TransactionStateAborted.CheckForFinishedTransaction(InternalTransaction
tx) at System.Transactions.Transaction.Promote() at
System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction
transaction) at
System.Transactions.TransactionInterop.GetDtcTransaction(Transaction
transaction) at
System.Messaging.MessageQueue.StaleSafeReceiveMessage(UInt32 timeout,
Int32 action, MQPROPS properties, NativeOverlapped* overlapped,
ReceiveCallback receiveCallback, CursorHandle cursorHandle, IntPtr
transaction) at
System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32
action, CursorHandle cursor, MessagePropertyFilter filter,
MessageQueueTransaction internalTransaction,
MessageQueueTransactionType transactionType) at
System.Messaging.MessageQueue.Receive(TimeSpan timeout,
MessageQueueTransactionType transactionType) at
NServiceBus.Transports.Msmq.MsmqDequeueStrategy.ReceiveMessage(Func`1
receive) in
c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line
313
Some other notes:
Both the erroring ApplicationPools' identities and the Windows
Services' Log On users are the same.
This actually worked well before
a recent reboot, as the Web.API services were able to successfully
process subscription requests, and are able to publish messages just
fine (though publishing does not automatically use MSDTC, and we are
not using a TransactionScope explicitly). Since the local reboot, we
simply get the above error if a subscription request message sits
in either of the Web.API publisher's input queue.
I've used both procmon.exe and MSDTC tracing and have found nothing of interest. The typical event viewer logs also do not provide any information.
All endpoints are running .NET 4.5 and NServiceBus 4.6
We cannot
recreate this in any other environment.
Additional notes from below conversations
The thread which throws the exception is pure NServiceBus subscription management where none of "my" code is involved. When the application pool starts the w3wp.exe worker process on demand, NSB is spawning a worker thread unbeknownst to the application to process subscription requests. It should only ever work across the publisher's input queue and the subscription storage, which I'm using MSMQ for that as well, in a queue right beside the other (i.e. no other server is involved to my knowledge).
The "code" of the website didn't change across reboots, and the application pool stopped and restarted several times before the reboot without issue.
Not really an answer, but too long for a comment.
What part of your operation requires DTC? A Distributed Transaction gets enlisted automatically when needed, usually when you are talking to two different DTC-supporting bits of infrastructure (e.g. MSMQ and a database).
You said you tested via DTC tracing--do you mean DTC Ping? Did you test by having it run on both machines (or all machines if there are more than two involved in the transaction)? The DTC tool is pretty esoteric, and its output can be confusing.
Also, if it did work before the reboot, is it possible the reboot reset firewall settings? Firewalls are a common cause of DTC problems.
Also, I assume you checked and rechecked your DTC settings on the local machine? Did you ensure that your MSMQ queues are set up to be transactional?
From your comments:
Note that this particular failure occurs when attempting to dequeue a
message from a local private MSMQ queue [...]
The stack trace makes it appear that that's all it's doing, but I suspect that as it is attempting dequeue it is also trying to enlist the transaction between multiple servers. See below.
Why MSDTC? It's the original way to support exactly-once messaging in
NServiceBus (see here).
Right, but what I'm asking is why the particular operation requires a distributed transaction. If all a handler is doing is reading from a queue and (for example) writing output to the console, MSDTC will never be enlisted, even though the handler is wrapped in a transaction scope. It will simply use a local transaction to read from the queue. The escalation to a distributed transaction is automatic, and only happens when it is needed to support multiple bits of infrastructure.
So if you recently deployed code in a handler that writes data to a new database server, you may be getting a failure because you are now enlisting a transaction that includes the new server, which may be where the failure is happening.
So determining all the servers involved in the distributed transaction is the first step. The next step would be to check the DTC settings on all involved servers. If DTC settings aren't the problem, I'd recommend testing communication between the servers using DTCPing. The NServiceBus documentation has some good instructions for using DTCPing.
What "fixed" this for us in the production environment was adding the application pool identity user to the local Administrators group on the server. Unfortunately we don't have time to determine what setting required that security setup, as this isn't a required configuration in other similar servers. Also, this isn't the most desirable solution from a security perspective, but in our particular situation, we're willing to live with it.
I develop in the last days a web page (http://www.srtbox.com/) for testing my architecture, more info here. With NoRM, MongoLab or MongoHQ for DB hosting. And I having a a lot of errors with NoRM. All with the Norm.BSON.BsonDeserializer class. I could fix one, but now im getting some errors in the connection. Error:
System.Net.Sockets.SocketException
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at System.IO.BinaryReader.ReadBytes(Int32 count)
at Norm.BSON.BsonDeserializer.Deserialize[T](BinaryReader stream, Int32 length)
The truth is that this NoRM driver seems really unstable... The official driver does not offer Linq support and I did not find one example with a POCO object. Some one have a good experience with and NoSQL database and asp.net?? Scalable like MongoDB? RavenDB seems a nice option, but MongoDB have more success stories, of course without .Net .
So the big question is: Which NoSQL Scalable DB will you recommend for ASP.Net? Do you have any success story?
PD: I will be thankful if you visit my site: (http://www.srtbox.com/) for testing.
That looks like a network error, not a problem with the driver. If the database isn't local you will see that if you have a bad connection or didn't set up the firewall right - not much you can do differently in code to change it. Trying to keep a single connection open for too long can also cause connection errors.
Most of the NoSQL databases available work well with .NET so you can choose based on functional requirements rather than .NET compatibility. However, you shouldn't expect it to work just like SQL or have that many examples - most of them are used primarily on other platforms and since they all have different ways of running queries, LINQ isn't always a good fit anyway.
Also, what do you mean by POCO? The serialization attributes can make the classes look complicated, but they are just regular objects, not the lazy loaded self updating objects you get from a typical ORM.
I've got a number of ASP.Net websites (.Net v3.5) running on a server with a SQL 2000 database backend. For several months, I've been receiving seemingly random InvalidOperationExceptions with the message "Internal connection fatal error". Sometimes there's a few days in between, while other times there are multiple errors per day.
The exception is not limited to one site in particular, though they share business and data access assemblies. The error seems to always be thrown from SqlClient.TdsParser.Run(). It sometimes is thrown from old-school direct SqlCommand.Execute() calls, while other times it is thrown from Linq2Sql code.
I've been assured by the network guys that there are no errors or packets lost on their end. Has anyone else experienced this? Could it be a driver problem? We have been unable as of yet to pinpoint a specific trigger for this exception.
We're running II6 on Windows Server 2003.
After a few months of ignoring this issue, it started to reach a critical mass as traffic gradually increased. Under heavy load, including some crawlers, things got crazy and these errors poured in nonstop.
Through trial and error, we eventually tracked down a handful of SqlCommand or LINQ queries whose SqlConnection wasn't closed immediately after use. Instead, through some sloppy programming originating from a misunderstanding of LINQ connections, the DataContext objects were disposed (and connections closed) only at the end of a request rather than immediately.
Once we refactored these methods to immediately close the connection with a C# "using" block (freeing up that pool for the next request), we received no more errors. While we still don't know the underlying reason that a connection pool would get so mixed up, we were able to cease all errors of this type. This problem was resolved in conjunction with another similar error I posted, found here: Why is my SqlCommand returning a string when it should be an int?
Sounds like the database connection is getting dropped or timing out.
We recently had similar issues moving to IIS 6 from IIS 5 connecting to SQL 2000. Our issue was solved by increasing number of ephemeral ports available.
Look at the usage of the ephemeral ports by the IIS server. The default max no. of ports available is normally 4000. You might want to consider increasing this if the sites on your server are particularly busy or your application is making a lot of database calls.
You can monitor these first to see if going over max limit.
Search Microsoft Knowledge base for "MaxUserPort" and "TcpTimedWaitDelay" and make necessary registry changes. Make sure you back up registry or snapshot server before making the changes. Will need to reboot for changes to take effect.
You should double check your database and recordset connection are being closed after use. Not closing will use up this port range unnecessarily.
Check the efficiency of your stored procedures anyway as they might be taking longer than they need too.
"If you rapidly open and close 4000 sockets in less than four minutes, you will reach the default maximum setting for client anonymous ports, and new socket connection attempts fail until the existing set of TIME_WAIT sockets times out." - from http://support.microsoft.com/kb/328476
Check your server's LOG folder (\program files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG or similar) for files named SqlDump*.mdmp and SqlDump*.txt. If you do find any you'll have to take it to Product Support.
I was creating a new EF Core project and was trying to create the database to an external Linux server instead of a Windows Server or local one. After hours of searching I found out that I am using MySQL instead of the Microsoft SQL server.
I found it weird that everyone was using 1433 instead of the usual 3306. So to fix my 'Internal connection fatal error' I had to set up a docker instance of SQL Server bound to its default port of 1433.
It literally was that simple. In the docker repo look for "microsoft-mssql-server" and run the image as described neatly in the description below. Everything works now and I am able to push my database from my EF Core project to an external server.