I'm occasionaly getting this exception on our production server:
System.ArgumentNullException: Value cannot be null.
at System.Threading.Monitor.Enter(Object obj)
at System.Data.ProviderBase.DbConnectionPool.TransactedConnectionPool.TransactionEnded(Transaction transaction, DbConnectionInternal transactedObject)
at System.Data.SqlClient.SqlDelegatedTransaction.SinglePhaseCommit(SinglePhaseEnlistment enlistment)
at System.Transactions.TransactionStateDelegatedCommitting.EnterState(InternalTransaction tx)
at System.Transactions.CommittableTransaction.Commit()
at System.Transactions.TransactionScope.InternalDispose()
at System.Transactions.TransactionScope.Dispose()
//... continues here with references to my DAL code
What is the reason for this exception to happen?
I have already did some research on this but with no conrete success yet. I also read this questions here:
Intermittent System.ArgumentNullException using TransactionScope
TransactionScope automatically escalating to MSDTC on some machines?
And now I know that if I could avoid escalating my transactions to DTC I would get rid of this problem. But what if I could not? I have multiple databases to update or read from in one transaction so I have to use DTC. I'm getting this error ocassionaly on actions that usually works well.
Technical background
It is ASP MVC2 and LINQ2SQL application on .NET 3.5
We have three virtuals with load balanacing based on IP address, each having IIS7
Single virutal with SQL server 2008 - it is shared by web servers
I should point out that I was not able to reproduce this exception on my development machine (development server + SQL express 2008) and on our testing machine (virtual with single IIS7 and SQL server 2008 together) either.
I'm suspecting our production servers configuration that there is some threading/processing issue (like two processes are trying to use the same connection).
UPDATE
I have found another link. It is stating that ado.net connection dispose bug is probably back. But it is a pity there is no resolution in the end and I have found nobody else describing similar issue.
http://social.msdn.microsoft.com/Forums/nl-BE/adodotnetdataproviders/thread/388a7965-9385-4f5c-a261-1894aa73c16e
According to
http://support.microsoft.com/kb/960754, there is an issue with 2.50727.4016 version of System.Data.dll.
If your server has this older version, I would try to get the updated one from Microsoft.
It looks like a bug, as it's .NET internal code, nothing do do with your own code.
If you take a look with reflector (or any other IL tool) on the internal TransactedConnectionPool.TransactionEnded method, you will see its implementation has changed between .NET 3 and .NET 4... I suppose it was not thread-safe back then. You could try to report it to Microsoft Connect.
According to the doco MSDN System.Transactions.TransactionScope the method is synchronous and therefore it's using the monitor. The doco doesn't say that the method is threadsafe, so I think it's likely that you are somehow calling dispose on the same Transaction scope object from more than one thread. You can use a static property of the transactionscope object System.Transactions.Transaction.Current to find out which transaction you are referring to. Perhaps a log message prior to disposing your transaction scope might expose where this is occurring...
If it's not a threading issue then odds on you've found a corner case that trips a bug in .Net. I've found the behaviour of MSDTC when things go wrong to be unpleasant at best.
Related
Ahoy,
We have two BizTalk applcations in BizTalk 2013 R2 that seem to be having random issues. Both applications follow the same process.
Pull data from a WCF endpoint.
Delete data from a database via a stored procedure.
Insert the new data that was pulled via WCF-SQL call.
Both applications worked great during our testing for quite a while. But, over time, we've had a few issues crop up with the insert via the WCF-SQL call.
A fatal error occurred while reading the input stream from the network. The session will be terminated (input error: 64, output error: 0).
This error showed up in the Sql Server logs. We had this one for about a day and then it just went away. Everything else continued to work fine on that target sql server. It was only BizTalk that had issues.
Our latest error is where the request to the WCF-SQL insert happens ( the data is actually inserted ), but there never is a response. So, the Send Port continues to try and send for it's retries and the Orchestration just dehydrates.
We tinkered with every setting throughout the application to try and solve this, but only a delete of the application and a redeploy fixed this ( for now at least ).
So, I guess my question is whether or not anyone else has had these sorts of issues with BizTalk having "random" errors like this where it'll work great and then go downhill like we've seen?
I'd really prefer to have something stable that is minimal maintenance. This is an enterprise product after all.
I've issues similar to this happen when moving between environments where there were data differences, e.g. a column full of NULLs in QA and a column full of actual data in PROD. There are a few things you can try.
Use SQL Sever Profiler to capture the RPC call coming from BizTalk, and try running it directly on the SQL Server BizTalk is calling remotely (wrap it in a transaction you roll back at the end if this is production). Does it take longer than expected to run? Debug the procedure to find the pain points and optimize if possible. I've written a blog about how to do this here: http://blog.tallan.com/2015/01/09/capturing-and-debugging-a-sql-stored-procedure-call-from-biztalk/
Up the timeout settings in the binding configuration for the send port to ensure that it is not timing out before SQL can finish doing its work.
Up the System.Transactions timeout in Machine.config to ensure that MSDTC isn't causing issues: http://blogs.msdn.com/b/madhuponduru/archive/2005/12/16/how-to-change-system-transactions-timeout.aspx and http://blog.brandt-lassen.dk/2012/11/overriding-default-10-minutes.html
If possible, do a data compare between the TEST/QA and PROD databases. Look for significant differences, especially in columns that you are using in JOIN conditions and WHERE clauses.
How would you go about and handle lost data from a sql connection loss on a ASP.NET application.
Lets say your running an algorithm adding and removing certain roles. A midst it, the connection to the SQL database is lost. And because the connection is not there, wont even be able to backtrack the steps done. The whole state is lost, leaving the database in an error nous condition.
Would you set the IIS Rapid Fail Protection to shut the site down upon 1 exception and manually force the function to run again (after connection have been fixed).
Or how is the professional way of handling it, i am quite new to it. Maybe there's something i do not know of it (such as iis maybe trying to rerun it/caching)
(Using entity framework)
This is not a coding problem in its own way, it is more of a question of best practice handling data loss with sql database on asp.net
You need to do batch SQL Operations inside a SQL Transaction. So that whatever the error, a rollback happens. This is a built-in SQL feature and nothing special needs to be done.
Once you start a SQL Transaction, a Commit is issued only when all operations succeed. The default behavior is normally Rollback in case of all other non-success scenarios.
If you're encountering issues with any specific logic, post the code snippet and we're glad to help.
I'm using web api self host inside a windows service and I've encountered a problem and after googling for couple of hours haven't found a reasonable answer.
One of the api controllers serves large stream of data (not really that large, couple of tens of MB). It takes some time to produce data so I've decided to use TransferMode.StreamedResponse to minimize the time client has to wait for response. I've also added a CompressHandler and custom CompressedContent (derived from HttpContent) mostly based on a following answer.
The controller returns an instance of IDataReader, which is then serialized by custom formatter which is lastly compressed inside CompressedContent that I've mentioned. The whole data passing is streamed so while the client receives data, a data reader on server side may still be reading rows from database. Everything works fine when client is acting nicely.
The problem occurs when a client drops connection while the data is still being serialized to the underlying network stream. I've tried to watch for IsFaulted task inside of ContinueWith delegate (in CompressedContent from the link) and dispose underlying network Stream. Unfortunately the CommunicationException (The specified network name is no longer available) is still being thrown when the control leaves my code. From the stacktrace it looks like the exception is thrown when the Web Api tries to close (end) the underlying network stream (http channel?). As it happens with unobserved exceptions it brings entire windows service down.
I've mitigated the problem by setting windows service recovery options but I would like to know if this failure can be handled in code.
Is there a way to setup a custom error handler (IErrorHandler presumably) inside web api self hosting service mode to prevent this kind of error?
I'm using Beta version, I will try to reproduce this error on RC but I somehow doubt that setting up this kind of error handler would change in any way
We had the same issue. I was able to submit a fix to MS and they have in turn released a nightly build that fixes this. They are looking at back porting the fix to the RTM. You can see the pull release here: http://aspnetwebstack.codeplex.com/SourceControl/network/forks/rdean79/issue284/contribution/3329
Hi i got a web application(asp.net) where we just started getting "System.InvalidOperationException: Timeout expired." when trying to get a new sql connection.
So my guess some where in the code a connection is created but never disposed, but how would i go about to find where this happens? sadly most of the database communication does not use a datalayer it works directly with the sql data types...
Is there any thing i could enable in the web.config to trace what connections are open for longer then x seconds and where they where opened?
find everywhere your sqlconnection is used. Ensure it is in a using() block to automatically dispose of it. There is nothing built in to the web.config with this unfortunately. You may be able to try out ants memory profiler to help track this down. for ex:
http://www.developerfusion.com/review/59466/ants-memory-profiler-51/
Ok i found a way to track them down by using
EXEC SP_WHO2
DBCC INPUTBUFFER(SPID)
SP_WHO2 gives you information about the connections and by using DBCC INPUTBUFFER you can find out what command they ran last time.
.NET framework will not solve your architecture and logging issues so you will have to find the problem yourselves. What can help you:
Performance counters can show you utilization of connection pool
SQL Server Management Studio can show you connections in use, their activity and last executed statement. In Object Explorer select the server and in context menu go to Reports > Standard Reports > Activity xxx
Another approach is turning off the pool and using some profiler to collect information about your application and track not disposed connections.
Just curious if anyone else has got this particular error and know how to solve it?
The scenario is as follow...
We have an ASP.NET web application using Enterprise Library running on Windows Server 2008 IIS farm connecting to a SQL Server 2008 cluster back end.
MSDTC is turned on. DB connections are pooled.
My suspicion is that somewhere along the line there is a failed MSDTC transaction, the connection got returned to the pool and the next query on a different page is picking up the misbehaving connection and got this particular error. Funny thing is we got this error on a query that has no need whatsoever with distributed transaction (committing to two database, etc.). We were only doing select query (no transaction) when we got the error.
We did SQL Profiling and the query got ran on the SQL Server, but never came back (since the MSDTC transaction was already aborted in the connection).
Some other related errors to accompany this are:
New request is not allowed to start
because it should come with valid
transaction descriptor.
Internal .Net Framework Data Provider error 60.
MSDTC has default 90 seconds timeout, if one query execute exceed this time limit, you will encounter this error when the transaction is trying to commit.
A bounty may help get the answer you seek, but you're probably going to get better answers if you give some code samples and give a better description of when the error occurs.
Does the error only intermittently occur? It sounds like it from your description.
Are you enclosing the close that you want to be done as a transaction in a using TransactionScope block as Microsoft recommends? This should help avoid weird transaction behavior. Recall that a using block makes sure that the object is always disposed regardless of exceptions thrown. See here: http://msdn.microsoft.com/en-us/library/ms172152.aspx
If you're using TransactionScope there is an argument System.TransactionScopeOption.RequiresNew that tells the framework to always create a new transaction for this block of code:
Using ts As New Transactions.TransactionScope(Transactions.TransactionScopeOption.RequiresNew)
' Do Stuff
End Using
Also, if you're suspicious that a connection is getting faulted and then put back into the connection pool, the likely solution is to enclose the code that may fault the connection in a Try-Catch block and Dispose the connection in the catch block.
Old question ... but ran into this issue past few days.
Could not find a good answer until now. Just wanted to share what I found out.
My scenario contains multiple sessions being opened by multiple session factories. I had to correctly rollback and wait and make sure the other transactions were no longer active. It seems that just rolling back one of them will rollback everything.
But after adding the Thread.Sleep() between rollbacks, it doesn't do the other and continues fine with the rollback. Subsequent hits that trigger the method don't result in the "New request is not allowed to start because it should come with valid transaction descriptor." error.
https://gist.github.com/josephvano/5766488
I have seen this before and the cause was exactly what you thought. As Rice suggested, make sure that you are correctly disposing of the db related objects to avoid this problem.