I have a custom workflow in CRM 2011 which is manually triggered against custom entity records. When a single record is selected for processing, the workflow is always successful. However when selecting more than one record, at least one will fail. The error(s) provided seem to vary from one attempt to the next, even though the same data is being used.
Errors I've encountered so far are:
ValidateOpen - Encountered disposed CrmDbConnection when it should
not be disposed You cannot create a SqlExecutionContext from another
SqlExecutionContext on which OnBeginRequest has not been called
And after restarting the DB server:
Invalid Pointer
All of these occur when calling the Update method of the IOrganizationService. The Invalid Pointer error seems to be the more common error since having restarted the DB server (it was suggested to me that it might be an issue with an overly-busy DB server). I've also deleted any backlog of asynchronous tasks in case this was the issue, but it hasn't had any effect.
Does anyone know what this error means, why I might be getting it or how I can get around the issue?
Many thanks!
It seems this error message is a round-about way of saying, 'tried to open another connection to CRM when the previous one was open', and the reason it tries to do this is very much relevant to the fact it only fails when processing more than one record.
Reference this blog:
It seems that when a workflow is run against multiple records, it uses the same instance of the class, which means that class-level variables won't be re-instantiated between executions. So, when subsequent executions come to the code which sets the class-level service variable to an instance of an IOrganisationService, it finds the variables already has one and that it's open.
The solution I've found easiest to implement it to have the service variable within the Execute function, rather than class-level. This has solved the problem everywhere that I've tried it since.
According to the CRM 2011 Web Service Error Codes...
InvalidPointer - The object is disposed.
Related
I am using an Axon Event Tracking processor. Sometimes events take longer that 10 seconds to process.
This seems to cause the message to be processed again and this appears in the log "Releasing claim of token X/0 failed. It was owned by another node."
If I up the number of segments it does not log this BUT the event is still processed twice so I think this might be misleading. (I think I was mistaken about this)
I have tried adjusting the fetchDelay, cleanupDelay and tokenClaimInterval. None of which has fixed this. Is there a property or something that I am missing?
Edit
The scenario taking longer than 10 seconds is making a HTTP request to an external service.
I'm using axon 4.1.2 with all default configuration when using with Spring auto configuration. I cannot see the Releasing claim on token and preparing for retry in [timeout]s log.
I was having this issue with a single segment and 2 instances of the application. I realised I hadn't increased the number of segments like I thought I had.
After further investigation I have discovered that adding an additional segment seems to have stopped this. Even if I have for example 2 segments and 6 applications it still doesn't reappear, however I'm not sure how this is different to my original scenario of 1 segment and 2 application?
I didn't realise it would be possible for multiple threads to grab the same tracking token and process the same event. It sounds like the best action would be to put an idem-potency check before the HTTP call?
The Releasing claim of token [event-processor-name]/[segment-id] failed. It was owned by another node. message can only occur in three scenarios:
You are performing a merge operation of two segments which fails because the given thread doesn't own both segments.
The main event processing loop of the TrackingEventProcessor is stopped, but releasing the token claim fails because the token is already claimed by another thread.
The main event processing loop has caught an Exception, making it retry with a exponential back-off, and it tries to release the claim (which might fail with the given message).
I am guessing it's not options 1 and 2, so that would leave us with option 3. This should also mean you are seeing other WARN level messages, like:
Releasing claim on token and preparing for retry in [timeout]s
Would you be able to share whether that's the case? That way we can pinpoint a little better what the exact problem is you are encountering.
By the way, very likely you have several processes (event handling threads of the TrackingEventProcessor) stealing the TrackingToken from one another. As they're stealing an un-updated token, both (or more) will handled the same event. Hence why you see the event handler being invoked twice.
Obviously undesirable behavior and something we should resolve for you. I would like to ask you to provide answers to my comments under the question, as right now I have to little to go on. Let us figure this out #Dan!
Update
Thanks for updating your question #dan, that's very helpful.
From what you've shared, I am fairly confident that both instances are stealing the token from one another. This does depend though on whether both are using the same database for the token_entry table (although I am assuming they are).
If they are using the same table, then they should "nicely" share their work, unless one of them takes to long. If it takes to long, the token will be claimed by another process. This other process in this case is the thread of the TEP of your other application instance. The "claim timeout" is defaulted to 10 seconds, which also corresponds with the long running event handling process.
This claimTimeout is adjustable though, by invoking the Builder of the JpaTokenStore/JdbcTokenStore (depending on which you are using / auto wiring) and calling the JpaTokenStore.Builder#claimTimeout(TemporalAmount) method. And, I think this would be required on your end, giving the fact you have a long running operation.
There are of course different ways of tackling this. Like, making sure the TEP is only ran on a single instance (not really fault tolerant though), or offloading this long running operation to a schedule task which is triggered by the event.
But, I think we've found the issue at least, so I'd suggest to tweak the claimTimeout and see if the problem persists.
Let us know if this resolves the problem on your end #dan!
I'm doing an update for multiple nodes and then a batch publish (a lot of data) using the command "Activate" to publish the nodes programmatically but sometimes I don't know why I will get an error sometimes. It only happened for few times so far so I find it hard to figure out what's the one causing the issue.
Here is the error thrown.
Caused by: org.apache.jackrabbit.core.state.StaleItemStateException: 8b3ce2d4-eb7b-4838-901c-413aa9eeee84/{http://www.magnolia.info/jcr/mgnl}activationStatus has been modified externally
at org.apache.jackrabbit.core.ItemSaveOperation.removeTransientItems(ItemSaveOperation.java:723)
at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:228)
Have anyone encountered a similar issue? Or any idea where to look at this issue?
Thanks! :)
Basically update operation internally cause Magnolia to modify the activation state (which is correct). When this operation is ongoing the update operation is taking over and attempting to change the state. In this case if you are unlucky you might hit the exception you got. Reasoning being is that those commands run async but touch the same nodes.
Potential Solution:
1) You may simply give some time for the first operation to complete and execute the second one after.
2) This needs some development effort but here we go:
You may use LockManager to lock the nodes being used by the first operation and unlock it when the operation is done completely. Second operation meanwhile will check if the nodes are locked and if yes will not continue but wait till its unlocked from the first process.
Having deployed a new build of an ASP.NET site in a production environment, I am logging dozens of data errors every second, almost always with the error "Cannot find table 0." We use datasets and frequently refer to Table[0], and while I understand the defensive coding practice of checking the dataset for tables before accessing Table[0], it's never been a problem in the past. A certain page will load fine one second, and then be missing one of its data-driven components the next. Just seeing if this rings a bell for anyone.
More detail: I used a different build server this time, and while I imagine the compiler settings are the same on both, I have a hard time thinking that there's a switch that makes 50% of my database calls come back with no tables. I also switched the project to VS 2008, but then reverted all of those changes when I switched back to VS 2005. I notice that the built assembly has a new MyLibrary.XmlSerializers.dll, where it didn't used to, but I also can't imagine that that's causing all the trouble. (It also doesn't fall down on calls to MyLibrary, or at least no more than any other time.)
Updated to add: I've discovered that the troublesome build is a "Release" build, where the working build was compiled as "Debug". Could that explain it?
Rolling back to the build before these changes fixed it. (Rebooting the SQL Server, the step we tried before that, did not.)
The trouble also seems to be load-based - this cruised through our integration and QA environments without a problem, and even our smoke test environment - the one that points to production data - is fine under light load.
Does this have the distinguishing characteristics of anything you might have seen in the past?
Bumping this old question because we have encountered the same issue and perhaps our solution would give more insight in what causes this.
Essentially this problem occurs in a production environment that is under very heavy load in a Windows service that uses multiple threads to process several jobs simultaneously (100 users use the same DB via ASP.NET web app and there are about 60 transactions/second on older hardware with SQL Server 2000).
No variables are shared, that is connections are opened anew, transaction is started, operations executed, transaction committed and connection closes.
Under heavy load sometimes one of the following exceptions occurs:
NullReferenceException: Object reference not set to an instance of an
object.
at System.Data.SqlClient.SqlInternalConnectionTds.get_IsLockedForBulkCopy()
or
System.Data.SqlClient.SqlException:
The server failed to resume the transaction. Desc:3400000178
or
New request is not allowed to start because it should come with valid transaction descriptor
or
This SqlTransaction has completed; it is no longer usable
It seems somehow the connection that is within the pool becomes corrupted and remains associated with previously used transactions. Furthermore, if such connection is retrieved from pool then sqlAdapter.Fill(dataset) results in an empty dataset, causing "Cannot find table 0". Because our service would retry the operation (reading job list) on failure and it would always get the same corrupt connection from the pool it would fail with this error until restarted.
We removed the issue by using SqlConnection.ClearPool(connection) on exception to make sure this connection is discarded from the pool and restructuring the application so less threads access the same resources simultaneously.
I have no clue who exactly caused this issue so I am not sure we have really fixed that, maybe just made it so rare it had not occurred again yet.
I've fought precisely this error message before. The key is that an underlying data method is swallowing a timeout exception.
You're probably doing something like this:
var table = GetEmployeeDataSet().Tables[0];
GetEmployeeDataSet is swallowing an exception, probably a timeout exception, which is why it only happens sporadically - it happens under load. You need to do the following to fix it:
Modify the underlying code to not swallow the exception, but rather let it bubble up to the next level so you can identify it properly.
Identify the query(s) causing the problem, and then rewrite, reindex, denormalize or throw hardware at the problem. See this for more info: System.Data.SqlClient.SqlException: Timeout expired
I've seen something similar. I believe our problem had to do with failed sessions being re-used (once the session object failed it went into a poor state and could not recover.) We fixed it by increasing the memory for the session pool and increasing the frequency of the web application recycling.
It also was "caused" by a new version that at first blush did not seem to have any change to cause such an effect. However, eventually it became clear that the logic of the program was opening and closing a lot more connections (maybe 20% more) than it used to. This small change pushed the limit of our prior configuration.
You might check the SQL Server logs for errors. Or, the Web server event log. It sounds like your connection pool could be out of open connections or your db could be out.
Which database calls changed between versions?
The error is obviously telling you one of your database calls isn't returning any data on occasion; I can't think of any cases where a code/assembly issue would cause it.
I have seen something like this when doing something with nHibernate Sessions in a non-thread-safe manner. That would explain why you only see it under load. Would need to see your code to guess at what isn't thread-safe though.
Just curious if anyone else has got this particular error and know how to solve it?
The scenario is as follow...
We have an ASP.NET web application using Enterprise Library running on Windows Server 2008 IIS farm connecting to a SQL Server 2008 cluster back end.
MSDTC is turned on. DB connections are pooled.
My suspicion is that somewhere along the line there is a failed MSDTC transaction, the connection got returned to the pool and the next query on a different page is picking up the misbehaving connection and got this particular error. Funny thing is we got this error on a query that has no need whatsoever with distributed transaction (committing to two database, etc.). We were only doing select query (no transaction) when we got the error.
We did SQL Profiling and the query got ran on the SQL Server, but never came back (since the MSDTC transaction was already aborted in the connection).
Some other related errors to accompany this are:
New request is not allowed to start
because it should come with valid
transaction descriptor.
Internal .Net Framework Data Provider error 60.
MSDTC has default 90 seconds timeout, if one query execute exceed this time limit, you will encounter this error when the transaction is trying to commit.
A bounty may help get the answer you seek, but you're probably going to get better answers if you give some code samples and give a better description of when the error occurs.
Does the error only intermittently occur? It sounds like it from your description.
Are you enclosing the close that you want to be done as a transaction in a using TransactionScope block as Microsoft recommends? This should help avoid weird transaction behavior. Recall that a using block makes sure that the object is always disposed regardless of exceptions thrown. See here: http://msdn.microsoft.com/en-us/library/ms172152.aspx
If you're using TransactionScope there is an argument System.TransactionScopeOption.RequiresNew that tells the framework to always create a new transaction for this block of code:
Using ts As New Transactions.TransactionScope(Transactions.TransactionScopeOption.RequiresNew)
' Do Stuff
End Using
Also, if you're suspicious that a connection is getting faulted and then put back into the connection pool, the likely solution is to enclose the code that may fault the connection in a Try-Catch block and Dispose the connection in the catch block.
Old question ... but ran into this issue past few days.
Could not find a good answer until now. Just wanted to share what I found out.
My scenario contains multiple sessions being opened by multiple session factories. I had to correctly rollback and wait and make sure the other transactions were no longer active. It seems that just rolling back one of them will rollback everything.
But after adding the Thread.Sleep() between rollbacks, it doesn't do the other and continues fine with the rollback. Subsequent hits that trigger the method don't result in the "New request is not allowed to start because it should come with valid transaction descriptor." error.
https://gist.github.com/josephvano/5766488
I have seen this before and the cause was exactly what you thought. As Rice suggested, make sure that you are correctly disposing of the db related objects to avoid this problem.
For Asp.net web applications, is it best to:
trap errors within sql stored procedures and test for a return value in the code or
just let the error occur in sql (dont handle it) and rely on ado.net raising the errors within the code.
What are the best practises here?
A general rule that applies here is to catch the error as near the source as possible. SQL Server now has "try ... catch ..." error trapping syntax. So use it. The overhead of the little bit of extra code is insignificant, and if you have multiple statements in your SP, you can adapt the string in RAISERROR to help localize the problem.
In the interface, it shouldn't be difficult to trap the SP error event and handle it the same way you handle other error trapping in your procedural code.
This is one of the more neglected "best practices" in stored procedures, and it's even more important than in "regular" code because it's trickier to use a step=through debugger.
One useful pattern is to handle this in your SP the same way you it expect it to be handled in any other opaque SDK library.
According this article, this is a type of "boneheaded" exception - i.e. if an error occurs in the SP, this means that there's something wrong with the SP or the data itself.
My advise would be to trap the error in aps.net, as there you have much more possibilities to log the error, as well as all the parameters passed to the SP in order to investigate the problem.
I use try/catch on all potentially error generating calls to libraries or other servers including database queries. I'm primarily a developer not a DBA so I handle the error in the language that I'm most proficient in, C# not SQL. I'm sure that'll vary for every programmer. Not handling the error is never fine in my book.
I prefer both -
the stored procedure calls RAISEERROR, which manifests as an exception in ADO.NET
the stored procedure returns an error code, in case one sproc is calling another
in general, I always make db calls inside a try-catch
The most likely failure mode in the data layer is bad user input vales such as requesting a record that doesn't exist. Most other errors are caused by defective code and usually are caught right away when testing. I like to catch the db error and and always throw a custom error with more information about the data and context of the request. Then the error bubbles back up the call stack and if it was a user correctable error I can alert them to that fact. Otherwise the central error page handler for the application will be called when the error reaches the top of the stack. The worst thing to do is to catch an error and not let it bubble back up. Return codes are outdated constructs unless talking to com components or foreign systems.
Ideally, your web application isn't aware of the back-end or data access. So it shouldn't be handling sql-related errors - those should be handled by the data access layer. But the web application needs to handle failure gracefully, by providing the end-user with a friendly message.
I would say it depends on what you are doing in your stored proc and what you want to do with certain errors that occur. Sometimes I handle it in the sp but other times i let it raise up to the data layer code.
Bear in mind, sometimes you can miss errors when using sql server 2005 try/catch, see my post on this. Whereas, in code (C# in my case) you can access all the errors in the SqlErrorCollection object.
Just be absolutely certain that your error handling in the stored procedure is well thought out and any uncertainties are left to be passed up to the data layer code where you must log everything.
The answer is that you need to do both. Some errors are actually triggered downline from the Database engine. Their source varies, of course, according to the actual way you connect to the database (OLBC, OLEDB etc). You've got to find a way of dealing with these. Some errors such as Deadlock errors out to be handled at the application level too.
As well as errors, it is a good idea to receive and deal with messages from the SQL Server Database Engine. These are very similar to errors and can give the application a lot of useful diagnostic information. If you’re using System.Data.SQLClient, you’ll need to create a SqlInfoMessageEventHandler delegate, identifying the method that handles the event, to listen for the InfoMessage event on the SqlConnection class. You’ll find that message-context information such as severity and state are passed as arguments to the callback, because from the system perspective, these messages are just like errors.
I hope this helps!