I want to use Datastore from Cloud Compute through Java and I am following Getting started with Google Cloud Datastore.
My use case is quite standard - read one entity (lookup), modify it and save the new version. I want to do it in a transaction so that if two processes do this, the second one won't overwrite the changes made by the first one.
I managed to issue a transaction and it works. However I don't know what would happen if the transaction fails:
How to identify a failed transaction? Probably a DatastoreException with some specific code or name will be thrown?
Should I issue a rollback explicitly? Can I assume that if a transaction fails, nothing from it will be written?
Should I retry?
Is there any documentation on that?
How to identify a failed transaction? Probably a DatastoreException
with some specific code or name will be thrown?
Your code should always ensure that a transaction is either successfully committed or rolled back. Here's an example:
// Begin the transaction.
BeginTransactionRequest begin = BeginTransactionRequest.newBuilder()
.build();
ByteString txn = datastore.beginTransaction(begin)
.getTransaction();
try {
// Zero or more transactional lookup()s or runQuerys().
// ...
// Followed by a commit().
CommitRequest commit = CommitRequest.newBuilder()
.setTransaction(txn)
.addMutation(...)
.build();
datastore.commit(commit);
} catch (Exception e) {
// If a transactional operation fails for any reason,
// attempt to roll back.
RollbackRequest rollback = RollbackRequest.newBuilder()
.setTransaction(txn);
.build();
try {
datastore.rollback(rollback);
} catch (DatastoreException de) {
// Rollback may fail due to a transient error or if
// the transaction was already committed.
}
// Propagate original exception.
throw e;
}
An exception might be thrown by commit() or by another lookup() or runQuery() call inside the try block. In each case, it's important to clean up the transaction.
Should I issue a rollback explicitly? Can I assume that if a
transaction fails, nothing from it will be written?
Unless you're sure that the commit() succeeded, you should explicitly issue a rollback() request. However, a failed commit() does not necessarily mean that no data was written. See the note on this page.
Should I retry?
You can retry using exponential backoff. However, frequent transaction failures may indicate that you are attempting to write too frequently to an entity group.
Is there any documentation on that?
https://cloud.google.com/datastore/docs/concepts/transactions
Related
So I am attempting to use the Cloud Firestore offline cache ONLY as an API for my instrumentation tests, to avoid having to read and write from the server database during my integration tests.
First, in my test setup method, I call this method
protected fun setFirestoreToOfflineMode() {
Tasks.await(FirebaseFirestore.getInstance().disableNetwork())
}
Then, at the beginning of each relevant test, I use
fun givenHasTrips(vararg trips: Trip) {
GlobalScope.launch(Dispatchers.Default) {
trips.forEach {
firestoreTripApi.put(it)
}
}
}
In that put method, I have the following code:
try {
Tasks.await(tripCollection().document(tripData.id).set(tripData)),
firestoreApiTimeoutSeconds, TimeUnit.SECONDS)
Either.Right(Unit)
} catch (e: Throwable) {
Either.Left(Failure.ServerError)
}
I am calling the set() method and am waiting for a successful result in order to be able to return that the operation was a success, to update my UI afterward.
What happens is the cache DB is written correctly BUT the "set()" function times out because the database is in offline mode. I have read that Firestore only confirms a success if the Server DB was correctly written. If that is the case, I do not know if it is possible to have this call not time-out when operating strictly in offline-cache mode.
Is there a solution to have Firestore act as if the local cache database was the source of truth and return successes if placed in offline mode, just for tests?
The Task returned by the methods that modify the database (set, update, delete) only issues a callback when the data is fully committed to the cloud. There is no way to change this behavior.
What you can do instead is set up a listener to the document(s) that are expected to change, and wait for the listener to trigger. The listener will trigger even while offline.
I've a Model Product that fires retrieved event ProductRetrieved and a listener CheckProductValidity that throws an exception depending on the API path (if-else condition).
Also, I've a update query that I have implemented in Console\Kernal.php that runs everyday at 00:00 Hours.
Problem: CheckProductValidity throws an exception for scheduled tasks. How do I make an exception in listener to allow retrieval of model Product data when it is done by scheduler.
Possible solution: Use unsetEventDispatcher and setEventDispatcher but at times this update query may take more than usual time. Also, cron also sends notifications and processes jobs (all depending on Product) so that might cause problems.
Not really a solution but this is how I fixed it.
// fix to catch if artisan schedule:run has intiated this check;
$parameters = app('request')->server->get('argv');
$allowed_commands = ['schedule:run', 'migrate:refresh', 'db:seed', 'queue:work'];
if ($parameters[0]==='artisan'
&& in_array($parameters[1], $allowed_commands))
return true;
In the listener I added this code which would check if the request was a result of artisan command or a route.
All,
I am using Change Feed Processor Library.Want to know the best way to handle service failure along with the exceptions/errors scenario's in ProcessChangesAsync method. Below are the events am referring to.
1) Service failure - Service having the processor library crashed in the middle of some operation. How to start the process from the same document(doc on failure instance)? is there any inbuilt mechanism where change feed will start with the last failed documents? E.g. Let assume,in current batch we have 10 docs.5 processed successfully and then service breaks because of network failure or by some other reasons.Will my process starts with 6th document once service is re-started? How to achieve this?
2) Exception and Errors- Any errors in ProcessChangesAsync method can be handle using try catch at the global level but how to persist those failure records and make them available for the next batch? Again,looking for any available inbuilt mechanism in change feed process.
1) The Processor Library, by default, checkpoints after a successful run of ProcessChangesAsync. In the latest library version, you can customize the Checkpointer to do manual checkpoints in case you need it. If for some reason the processor shuts down before checkpointing, then it will start processing next from the the last successful checkpoint stored in the Leases collection. In your case, it will start with the first document again, so you will never lose a change but you could experience double processing (this is an "at least once" model).
2) There is no built-in mechanism that you can leverage, handling exceptions within the ProcessChangesAsync is your responsibility. You could not only add a global try/catch but, in the case you are looping over the documents, add a try/catch inside the loop, to handle a failing document (maybe send it to queue for later analysis/post-process) without losing the batch. If you require logging for those errors (I'm assuming that's what you mean by persisting errors?), then the latest version is compatible with LibLog, so plugging your own custom logging is as simple as:
using Microsoft.Azure.Documents.ChangeFeedProcessor.Logging;
var hostName = "SampleHost";
var tracelogProvider = new TraceLogProvider(); //You can use any provider supported by LibLog
using (tracelogProvider.OpenNestedContext(hostName))
{
LogProvider.SetCurrentLogProvider(tracelogProvider);
// After this, create IChangeFeedProcessor instance and start/stop it.
}
Source
Extra info for the comments
To avoid exceptions halting the batch or causing a batch to be reprocessed, you can have handling like this:
public async Task ProcessChangesAsync(IChangeFeedObserverContext context, IReadOnlyList<Document> documents, CancellationToken cancellationToken)
{
try
{
foreach(var document in documents)
{
try
{
// Do your work for the document
}
catch(Exception ex)
{
// Something happened with the current document, handle it, send it to a queue / another storage to analyze, log it. This catch will make the loop continue with the next.
}
}
}
catch(Exception ex)
{
// Something unhandled happened, log it and avoid throwing it again so the next batch is processed
}
}
We are using the SQLite.NET PCL in a Xamarin application.
When putting the database under pressure by doing inserts into multiple tables we are seeing BUSY exceptions being thrown.
Can anyone explain what the difference is between BUSY and LOCKED? And what causes the database to be BUSY?
Our code uses a single connection to the database created using the following code:
var connectionString = new SQLiteConnectionString(GetDefaultConnectionString(),
_databaseConfiguration.StoreTimeAsTicks);
var connectionWithLock = new SQLiteConnectionWithLock(new SQLitePlatformAndroid(), connectionString);
return new SQLiteAsyncConnection (() => { return connectionWithLock; });
So our problem turned out to be that although we had ensured within the class we'd written that it only created a single connection to the database we hadn't ensured that this class was a singleton, therefore we were still creating multiple connections to the database. Once we ensured it was a singleton then the busy errors stopped
What I've take from this is:
Locked means you have multiple threads trying to access the database, the code is inherently not thread safe.
Busy means you have a thread waiting on another thread to complete, your code is thread safe but you are seeing contention in using the database.
...current operation cannot proceed because the required resources are locked...
I am assuming that you are using async-style inserts and are on different threads and thus an insert is timing out waiting for the lock of a different insert to complete. You can use synchronous inserts to avoid this condition. I personally avoid this, when needed, by creating a FIFO queue and consuming that queue synchronously on a dedicated thread. You could also handle the condition by retrying your transaction X number of times before letting the Exception ripple up.
SQLiteBusyException is a special exception that is thrown whenever SQLite returns SQLITE_BUSY or SQLITE_IOERR_BLOCKED error code. These codes mean that the current operation cannot proceed because the required resources are locked.
When a timeout is set via SQLiteConnection.setBusyTimeout(long), SQLite will attempt to get the lock during the specified timeout before returning this error.
Ref: http://www.sqlite.org/lockingv3.html
Ref: http://sqlite.org/capi3ref.html#sqlite3_busy_timeout
I have applied the following solution which works in my case(mobile app).
Use sqlitepclraw.bundle_green nugget package with SqlitePCL.
Try to use the single connection throughout the app.
After creating the SQLiteConnection.
Apply busytime out using following call.
var connection = new SQLiteConnection(databasePath: path);
SQLite3.BusyTimeout(connection.Handle, 5000); // 5000 millisecond.
I need to execute two things on update():
commit entity to the database
send the entity through JMS
Because the object is quite large the send through JMS has to be outside the database transaction. Problem is that Seam adds the transaction based on JSF phases and so the database transaction is already active as soon as my own overridden update() is called.
Adding a call-back to the update like afterUpdate() would be nice but this does not seem to be possible.
Question:
How can I commit the entity and after that execute code outside the transaction?
Thanks for any help!
I found that the transaction used is a Spring transaction. That allows for something like:
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
#Override
public void afterCompletion(int status) {
switch (status) {
case STATUS_COMMITTED:
LOGGER.debug("update::afterCompletion");
afterCompletionCallback();
break;
case STATUS_ROLLED_BACK:
break;
case STATUS_UNKNOWN:
default:
break;
}
}
});
The transaction is then still available but the database is not locked anymore and any timeouts don't effect the Seam transaction.