Firestore documentation says:
"In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data."
I am using the cloud_firestore package and I noticed that doing
final TransactionHandler transaction = (Transaction tx) async {
DocumentSnapshot ds = await tx.get(userAccountsCollection.document(id));
return ds.data;
};
return await runTransaction(transaction).then((data){
return data;
});
the transaction may run multiple times but always return after the first transaction. Now in case of concurrent edits, the first transaction data may be incorrect so this is a problem for me.
How can I wait for the transaction to actually finish even if it will run multiple times and not return after the first one finished?
Your transaction code doesn't make any sense. It's not getting the contents of any documents. You only need to use a transaction if you intend to read, modify, and write at least one document.
The transaction function might only be run once anyway. There is only need for it to run multiple times if the server sees that there are a lot of other transactions occurring on documents, and it's having trouble keeping up with them all.
Related
Firestore documentation does not seem to specify when or how security rules are evaluated in a transaction and how that interacts with retries and optimistic locking.
My use case is straightforward, I have a lastUpdatedAt field on my documents and I'm using a transaction to ensure that I grab the latest document and check that the lastUpdatedAt field so I can resolve any conflicts before issuing an update.
In pseudocode the pattern is this
async function saveDocument(data: MyType, docRef: DocumentReference)
await firebase.firestore().runTransaction(async (transaction) => {
const latestDocRef = await transaction.get(docRef)
const latestDoc = latestDocRef.data()
if(latestDoc.lastUpdatedAt > data.lastUpdatedAt){
data = resolveConflicts(data, latestDoc)
data.lastUpdatedAt = latestDoc.lastUpdatedAt
}
await transaction.update(docRef, data)
}
}
Then in security rules I check to ensure that only updates with the latest or later lastUpdatedAt are allowed
function hasNoLastUpdatedAtConflict(){
return (!("lastUpdatedAt" in resource.data) ||
resource.data.lastUpdatedAt == null ||
request.resource.data.lastUpdatedAt >= resource.data.lastUpdatedAt);
}
//in individualRules
allow update: if hasNoLastUpdatedAtConflict() && someOtherConditionsEtc();
The docs say
In the Mobile/Web SDKs, a transaction keeps track of all the documents you read inside the transaction. The transaction completes its write operations only if none of those documents changed during the transaction's execution. If any document did change, the transaction handler retries the transaction. If the transaction can't get a clean result after a few retries, the transaction fails due to data contention.
However they don't specify how that behavior interacts with security rules. My transaction above is failing somtimes with a security rule violation. It only fails on a live Firestore environment, I haven't been able to make it fail in the emulator. I suspect what's happening is:
Transaction starts, sends doc to client
A concurrent write happens which changes lastUpdatedAt
The client does not see the new write so it can't resolve the conflict and it issues the update as-is
The security rule now fails because of the concurrent write when it would have otherwise succeeded
Firestore fails the whole transaction with permission-denied instead of retrying due to dirty data
I suppose I could implement client side retry in the case a transaction is rejected by a security rule violation but it's very surprising behavior if this is indeed what is happening.
Does anybody have insight into the actual behavior of security rules and Firestore transactions with optimistic locking?
After thorough testing I have confirmed that a firestore transaction using optimistic locking from the iOS/android library can be rejected with a permission-denied error due to a concurrent write happening after a document was read in the transaction.
The following scenario happens:
Begin transaction
Read a doc such as {id: 1, lastUpdatedAt: 1, data: "foo"}
Before writing to that doc in the transaction a cloud firestore trigger updates the doc to {id: 1, lastUpdatedAt: 2, data: "foo"}
Update doc in transaction .update({lastUpdatedAt: 1, data: "bar"})
Transaction throws exception firestore/permission-denied because this update violates a security rule of request.resource.data.lastUpdatedAt >= resource.data.lastUpdatedAt
The transaction does not get retried as the docs suggest, even though it performed a read of stale data. This means that users of the firestore libraries cannot rely on transactions being retried if it is possible that a concurrent write could cause the transaction to violate a security rule.
This is surprising and undocumented behavior!
As per the documentation and code sample, you can ensure that related documents are always updated atomically and always as part of a transaction or batch write using the getAfter() security rule function. It can be used to access and validate the state of a document after a set of operations completes but before Cloud Firestore commits the operations. Like get(), the getAfter() function takes a fully specified document path. You can use getAfter() to define sets of writes that must take place together as a transaction or batch.
These are some access call limits, which you may want to have a look at.
The documentation suggests that getAfter is useful to examine the contents of the database after the entire transaction's state would be recorded (in a sort of "staging" environment in memory), but before the transaction actually changes the database, visible to everyone. This is different from get(), because get() only looks at the actual contents of the database, before the transaction is finally committed. In short, getAfter() uses the entire staged write of the entire transaction or batch, while get uses the actual existing contents of the database.
getAfter() is useful when you need to examine other documents that may have been changed in the transaction or batch, and still have a chance to reject the entire transaction or batch by failing the rule. So, for example, if two documents being written in a single transaction must have some field value in common in order to be consistent, you need to use getAfter() to verify the equality between the two.
Point of note : The security rules for writes kick in before anything in the database has been changed by that write. That's how the security rules are able to safely and efficiently reject access, without having to roll back any writes that already happened.
If a transaction reads 3 docs and then updates 2 documents successfully but something after this causes the transaction to fail... will I be charged for the 3 reads and 2 writes that were made even though they are rolled back?
Edit---
Also will the get() below only cost 1 read? Where col2 is a subcollection of doc1.
db.collection('col1').doc('doc1').collection('col2').doc('doc2').get();
Edit 2
The firebase website states the following
For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data.
So say my transaction performs 10 reads on 10 different documents. If this gets called and during exectution some of the same documents are updated by other users, which will make the transaction retry, am I going to be hit with 10 * Number of retries for my reads?
Edit 3
I have read more about the transactions here https://firebase.google.com/docs/firestore/transaction-data-contention and it states that the server side transactions will lock the documents and wait for the transaction to finish.
q1) As the transaction is locking and not retrying over and over... will multiple concurrent calls to the firebase function that has a transaction not cost any extra reads/writes and will the functions just take longer to execute because of the lock?
q2) The webpage also has a banner at the bottom stating
Note: Only the server client libraries support transactions with read operations after write operations. For transactions in the mobile/web SDKs, document reads must come before document writes.
I just tried this on my firebase function and recieve the following error...
Error: Firestore transactions require all reads to be executed before all writes.
at Transaction.get (/srv/node_modules/#google-cloud/firestore/build/src/transaction.js:76:19)
I am using firebase admin version "^8.8.0", is performing reads after writes a feature that has been added in newer versions?
If transactions fail, will I still be charged?
Yes. A read was completed so you are charged for it. (I am unsure if there are any "rollback charges" - as the change now needs to be reversed.)
What is the cost of a sub-collection document read?
The doc1 was not read - so it would not be charged. You are charged for only one read.
I couldn't find a clear text in the documentation and these answers are from my personal usage for Firebase for over a couple years. A Firebasers confirmation would be helpful.
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
I have a Firebase Cloud Function that I call directly from my app. This cloud function fetches a collection of Firestore documents, iterating over each, then returns a result.
My question is, would it be best to keep the result of that fetch/get in memory (on the node server), refreshed with .onSnapshot? It seems this would improve performance as my cloud function would not have to wait for the Firestore response (it would already have the collection in memory). How would I do this? Simple as populating a global variable? How to do .onSnaphot realtime listener with cloud functions?
it might depend how large these snapshots are and how many of them may be cached ...
because, it is a RAM disk and without house-keeping it might only work for a limited time.
Always delete temporary files
Local disk storage in the temporary directory is an in-memory file-system. Files that you write consume memory available to your function, and sometimes persist between invocations. Failing to explicitly delete these files may eventually lead to an out-of-memory error and a subsequent cold start.
Source: Cloud Functions - Tips & Tricks.
It does not tell there, what exactly the hard-limit would be - and caching elsewhere might not improve access time that much. it says 2048mb per function, per default - while one can raise the quotas with IAM & admin. it all depends, if the quota per function can be raised far enough to handle the cache.
here's an example for the .onShapshot() event:
// for a single document:
var doc = db.collection('cities').doc('SF');
// this also works for multiple documents:
// var docs = db.collection('cities').where('state', '==', 'CA');
var observer = doc.onSnapshot(docSnapshot => {
console.log(`Received doc snapshot: ${docSnapshot}`);
}, err => {
console.log(`Encountered error: ${err}`);
});
// unsubscribe, to stop listening for changes:
var unsub = db.collection('cities').onSnapshot(() => {});
unsub();
Source: Get realtime updates with Cloud Firestore.
Cloud Firestore Triggers might be another option.
I want to use Datastore from Cloud Compute through Java and I am following Getting started with Google Cloud Datastore.
My use case is quite standard - read one entity (lookup), modify it and save the new version. I want to do it in a transaction so that if two processes do this, the second one won't overwrite the changes made by the first one.
I managed to issue a transaction and it works. However I don't know what would happen if the transaction fails:
How to identify a failed transaction? Probably a DatastoreException with some specific code or name will be thrown?
Should I issue a rollback explicitly? Can I assume that if a transaction fails, nothing from it will be written?
Should I retry?
Is there any documentation on that?
How to identify a failed transaction? Probably a DatastoreException
with some specific code or name will be thrown?
Your code should always ensure that a transaction is either successfully committed or rolled back. Here's an example:
// Begin the transaction.
BeginTransactionRequest begin = BeginTransactionRequest.newBuilder()
.build();
ByteString txn = datastore.beginTransaction(begin)
.getTransaction();
try {
// Zero or more transactional lookup()s or runQuerys().
// ...
// Followed by a commit().
CommitRequest commit = CommitRequest.newBuilder()
.setTransaction(txn)
.addMutation(...)
.build();
datastore.commit(commit);
} catch (Exception e) {
// If a transactional operation fails for any reason,
// attempt to roll back.
RollbackRequest rollback = RollbackRequest.newBuilder()
.setTransaction(txn);
.build();
try {
datastore.rollback(rollback);
} catch (DatastoreException de) {
// Rollback may fail due to a transient error or if
// the transaction was already committed.
}
// Propagate original exception.
throw e;
}
An exception might be thrown by commit() or by another lookup() or runQuery() call inside the try block. In each case, it's important to clean up the transaction.
Should I issue a rollback explicitly? Can I assume that if a
transaction fails, nothing from it will be written?
Unless you're sure that the commit() succeeded, you should explicitly issue a rollback() request. However, a failed commit() does not necessarily mean that no data was written. See the note on this page.
Should I retry?
You can retry using exponential backoff. However, frequent transaction failures may indicate that you are attempting to write too frequently to an entity group.
Is there any documentation on that?
https://cloud.google.com/datastore/docs/concepts/transactions