If a transaction reads 3 docs and then updates 2 documents successfully but something after this causes the transaction to fail... will I be charged for the 3 reads and 2 writes that were made even though they are rolled back?
Edit---
Also will the get() below only cost 1 read? Where col2 is a subcollection of doc1.
db.collection('col1').doc('doc1').collection('col2').doc('doc2').get();
Edit 2
The firebase website states the following
For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data.
So say my transaction performs 10 reads on 10 different documents. If this gets called and during exectution some of the same documents are updated by other users, which will make the transaction retry, am I going to be hit with 10 * Number of retries for my reads?
Edit 3
I have read more about the transactions here https://firebase.google.com/docs/firestore/transaction-data-contention and it states that the server side transactions will lock the documents and wait for the transaction to finish.
q1) As the transaction is locking and not retrying over and over... will multiple concurrent calls to the firebase function that has a transaction not cost any extra reads/writes and will the functions just take longer to execute because of the lock?
q2) The webpage also has a banner at the bottom stating
Note: Only the server client libraries support transactions with read operations after write operations. For transactions in the mobile/web SDKs, document reads must come before document writes.
I just tried this on my firebase function and recieve the following error...
Error: Firestore transactions require all reads to be executed before all writes.
at Transaction.get (/srv/node_modules/#google-cloud/firestore/build/src/transaction.js:76:19)
I am using firebase admin version "^8.8.0", is performing reads after writes a feature that has been added in newer versions?
If transactions fail, will I still be charged?
Yes. A read was completed so you are charged for it. (I am unsure if there are any "rollback charges" - as the change now needs to be reversed.)
What is the cost of a sub-collection document read?
The doc1 was not read - so it would not be charged. You are charged for only one read.
I couldn't find a clear text in the documentation and these answers are from my personal usage for Firebase for over a couple years. A Firebasers confirmation would be helpful.
Related
Firestore documentation does not seem to specify when or how security rules are evaluated in a transaction and how that interacts with retries and optimistic locking.
My use case is straightforward, I have a lastUpdatedAt field on my documents and I'm using a transaction to ensure that I grab the latest document and check that the lastUpdatedAt field so I can resolve any conflicts before issuing an update.
In pseudocode the pattern is this
async function saveDocument(data: MyType, docRef: DocumentReference)
await firebase.firestore().runTransaction(async (transaction) => {
const latestDocRef = await transaction.get(docRef)
const latestDoc = latestDocRef.data()
if(latestDoc.lastUpdatedAt > data.lastUpdatedAt){
data = resolveConflicts(data, latestDoc)
data.lastUpdatedAt = latestDoc.lastUpdatedAt
}
await transaction.update(docRef, data)
}
}
Then in security rules I check to ensure that only updates with the latest or later lastUpdatedAt are allowed
function hasNoLastUpdatedAtConflict(){
return (!("lastUpdatedAt" in resource.data) ||
resource.data.lastUpdatedAt == null ||
request.resource.data.lastUpdatedAt >= resource.data.lastUpdatedAt);
}
//in individualRules
allow update: if hasNoLastUpdatedAtConflict() && someOtherConditionsEtc();
The docs say
In the Mobile/Web SDKs, a transaction keeps track of all the documents you read inside the transaction. The transaction completes its write operations only if none of those documents changed during the transaction's execution. If any document did change, the transaction handler retries the transaction. If the transaction can't get a clean result after a few retries, the transaction fails due to data contention.
However they don't specify how that behavior interacts with security rules. My transaction above is failing somtimes with a security rule violation. It only fails on a live Firestore environment, I haven't been able to make it fail in the emulator. I suspect what's happening is:
Transaction starts, sends doc to client
A concurrent write happens which changes lastUpdatedAt
The client does not see the new write so it can't resolve the conflict and it issues the update as-is
The security rule now fails because of the concurrent write when it would have otherwise succeeded
Firestore fails the whole transaction with permission-denied instead of retrying due to dirty data
I suppose I could implement client side retry in the case a transaction is rejected by a security rule violation but it's very surprising behavior if this is indeed what is happening.
Does anybody have insight into the actual behavior of security rules and Firestore transactions with optimistic locking?
After thorough testing I have confirmed that a firestore transaction using optimistic locking from the iOS/android library can be rejected with a permission-denied error due to a concurrent write happening after a document was read in the transaction.
The following scenario happens:
Begin transaction
Read a doc such as {id: 1, lastUpdatedAt: 1, data: "foo"}
Before writing to that doc in the transaction a cloud firestore trigger updates the doc to {id: 1, lastUpdatedAt: 2, data: "foo"}
Update doc in transaction .update({lastUpdatedAt: 1, data: "bar"})
Transaction throws exception firestore/permission-denied because this update violates a security rule of request.resource.data.lastUpdatedAt >= resource.data.lastUpdatedAt
The transaction does not get retried as the docs suggest, even though it performed a read of stale data. This means that users of the firestore libraries cannot rely on transactions being retried if it is possible that a concurrent write could cause the transaction to violate a security rule.
This is surprising and undocumented behavior!
As per the documentation and code sample, you can ensure that related documents are always updated atomically and always as part of a transaction or batch write using the getAfter() security rule function. It can be used to access and validate the state of a document after a set of operations completes but before Cloud Firestore commits the operations. Like get(), the getAfter() function takes a fully specified document path. You can use getAfter() to define sets of writes that must take place together as a transaction or batch.
These are some access call limits, which you may want to have a look at.
The documentation suggests that getAfter is useful to examine the contents of the database after the entire transaction's state would be recorded (in a sort of "staging" environment in memory), but before the transaction actually changes the database, visible to everyone. This is different from get(), because get() only looks at the actual contents of the database, before the transaction is finally committed. In short, getAfter() uses the entire staged write of the entire transaction or batch, while get uses the actual existing contents of the database.
getAfter() is useful when you need to examine other documents that may have been changed in the transaction or batch, and still have a chance to reject the entire transaction or batch by failing the rule. So, for example, if two documents being written in a single transaction must have some field value in common in order to be consistent, you need to use getAfter() to verify the equality between the two.
Point of note : The security rules for writes kick in before anything in the database has been changed by that write. That's how the security rules are able to safely and efficiently reject access, without having to roll back any writes that already happened.
I have just added a new feature to an app I'm building. It uses the same working Cosmos/Table storage code that other features use to query and pump results segments from the Cosmos DB Emulator via the Tables API.
The emulator is running with:
/EnableTableEndpoint /PartitionCount=50
This is because I read that the emulator defaults to 5 unlimited containers and/or 25 limited and since this is a Tables API app, the table containers are created as unlimited.
The table being queried is the 6th to be created and contains just 1 document.
It either takes around 30 seconds to run a simple query and "trips" my Too Many Requests error handling/retry in the process, or hangs seemingly forever and no results are returned, the emulator has to be shut down.
My understanding is that with 50 partitions I can make 10 unlimited tables, collections since each is "worth" 5. See documentation.
I have tried with rate limiting on and off, and jacked the RU/s to 10,000 on the table. It always fails to query this one table. The data, including the files on disk, has been cleared many times.
It seems like a bug in the emulator. Note that the "Sorry..." error that I would expect to see upon creation of the 6th unlimited table, as per the docs, is never encountered.
After switching to a real Cosmos DB instance on Azure, this is looking like a problem with my dodgy code.
Confirmed: my dodgy code.
Stand down everyone. As you were.
We are using Firebase Functions with a few different HTTP functions .
One of the functions runs via a manual trigger from our website. It then pulls in a lot of data from an external resource and saves it into our Firestore database. Our function resources are Node.js 10, 1 GB of Memory and 540s before it times out.
However, when we have large datasets that we need to pull in, e.g. 5 000 - 10 000 records to write to the database, we start running into issues. We receive an error on large data sets of:
8 RESOURCE_EXHAUSTED: Bandwidth exhausted
The full error on Firebase Functions Health Dashboard logs looks like this:
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
at Object.callErrorFromStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client.js:176:52)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:342:141)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:305:181)
at Http2CallStream.outputStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:117:74)
at Http2CallStream.maybeOutputStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:156:22)
at Http2CallStream.endCall (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:142:18)
at ClientHttp2Stream.stream.on (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:420:22)
at ClientHttp2Stream.emit (events.js:198:13)
at ClientHttp2Stream.EventEmitter.emit (domain.js:466:23)
Our Firebase project is on the blaze plan and also, on GCP connected to an active billing account.
Upon inspection on GCP, it seems like we are NOT exceeding our WRITES per minute quote, as previously thought, however, we are exceeding our Cloud Build limit. We are also using batched writes when we save data to firestore from within the function, which seems to also make the amount of db writes less. e.g.
We don't use Cloud Build, so I assume that Firebase Functions uses Cloud Build in the back end to run the functions or something, but I can't find any documentation on the matter. We also have a few firestore database functions that run when documents are created. Not sure if that uses Cloud build in the back end or not.
Any idea why this would happen ? Whenever this happens, our function gets terminated with that error which causes us to only import half of our data. The data import works flawlessly with smaller amounts of data.
See our usage here for this particular project:
Cloud Build is used during the deployment of Cloud Functions. If you check this documentation you can see that:
Deployments work by uploading an archive containing your function's source code to a Google Cloud Storage bucket. Once the source code has been uploaded, Cloud Build automatically builds your code into a container image and pushes that image to Container Registry. Cloud Functions uses that image to create the container that executes your function.
This by itself is not enough to justify the charges you are seeing, but if you check the container image documentation it says:
Because the entire build process takes place within the context of your project, the project is subject to the pricing of the included resources:
For Cloud Build pricing, see the Pricing page. This process uses the default instance size of Cloud Build, as these instances are pre-warmed and are available more quickly. Cloud Build does provide a free tier: please review the pricing document for further details.
So with that information in mind, I would make an educated guess that your website is triggering the HTTP function enough times to make Cloud Functions scale up this particular function with new intances of it, which triggers a build process for the container that hosts the function and charges you as a Cloud Build charge. So to keep doing what you doing you are going to have to increase your Cloud Build Quota to meet this demand of your website.
There was a Firestore trigger that was triggering on new records of the same type I was importing.
So in short, I was creating thousands of records in a collection, and for every one of those, the firestore rule (function) triggered, but what I did not know at the time, is that it created a new build process in the background for each firestore trigger that ran, which is not documented anywhere.
I am working on a flutter app that fetches 341 documents from the firestore, after 2 days of analysis I found out that my read requests are increasing too much. So I made a chart on the stackdriver metrics explorer from which I get to know that my app is just reading 341 docs a single time, it's the firebase console which is increasing my reads.
Now, comes to what are the questions that are bothering me,
1)How reads are considered when we see data on the console and how can I reduce my read requests? Basically there are 341 docs but it is showing more than 600 reads whenever I refresh my console.
2)As you can see in the picture there are two types of document reads 'LOOKUP' and 'QUERY', what's the exact difference between them?
3)I am getting data from the firestore with a single instance and when I open my app the chart shows 1 active client which is cool but in the next 5 minutes, the number of active clients starts to increase.
Can anybody please explain to me why this is happening?
For the last question, I tried to disable all the service accounts and then again opened my app but got the same thing again.
Firestore.instance.collection("Lectures").snapshots(includeMetadataChanges: true).listen((d){
print(d.metadata.isFromCache);//prints false everytime
print(d.documents.length);// 341
print(d.documentChanges.length);//341
});
This is the snippet I am using. When the app starts it runs only once.
I will try to answer your questions:
How reads are considered when we see data on the console and how can I
reduce my read requests? Basically there are 341 docs but it is
showing more than 600 reads whenever I refresh my console.
Reads are considered depending on your how you query your Firestore database in addition to your access to this database from the console so using of the Firebase console will incur reads and even if you leave the console open to do other stuff, when new changes to database occured these changes will incur reads also, automatically.and any document read from the server is going to be billed. It doesn't matter where the read came from. The console should be included in that.
Check this official documentation under the "Manage data" title you can see there is a note : "Note: Read, write, and delete operations performed in the console count towards your Cloud Firestore usage."
Saying that if you think there is an issue with this, you can contact Firebase support directly to have more detailed answers.
However, If you check the free plan of Firebase you can see that you have 50K free reads per day.
A workaround that I found for this (thanks to Dependar Sethi)
Bookmarking the Usage tab of the Firestore page. (So you basically
'Skip' the Data Tab)
Adding a dummy collection in a certain way that ensures it is the
first collection(alphabetically) which gets loaded by default on
the Firestore page.
you can find his full solution here.
Also, you can optimise your queries however you want to retreive only the data that you want using where() method and pagination with Firebase
As you can see in the picture there are two types of document reads
'LOOKUP' and 'QUERY', what's the exact difference between them?
I guess there are no important difference between them but "QUERY" is getting the actual data(when you call data() method) while "LOOKUP" is getting a reference of these data(without calling data() method).
I am getting data from the firestore with a single instance and when I
open my app the chart shows 1 active client which is cool but in the
next 5 minutes, the number of active clients starts to increase.
For this question, considering the metrics that you are choosing in Stackdriver I can see 3 connected clients. and as per the decription of "connected client" metric:
The number of active connections. Each mobile client will have one connection. Each listener in admin SDK will be one connection. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
So please check: how many mobiles are connected to this instance and how many listeners do you have in your app. The sum of all of them is the actual number of connected clients that you are seeing in Stackdriver.
Firestore documentation says:
"In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data."
I am using the cloud_firestore package and I noticed that doing
final TransactionHandler transaction = (Transaction tx) async {
DocumentSnapshot ds = await tx.get(userAccountsCollection.document(id));
return ds.data;
};
return await runTransaction(transaction).then((data){
return data;
});
the transaction may run multiple times but always return after the first transaction. Now in case of concurrent edits, the first transaction data may be incorrect so this is a problem for me.
How can I wait for the transaction to actually finish even if it will run multiple times and not return after the first one finished?
Your transaction code doesn't make any sense. It's not getting the contents of any documents. You only need to use a transaction if you intend to read, modify, and write at least one document.
The transaction function might only be run once anyway. There is only need for it to run multiple times if the server sees that there are a lot of other transactions occurring on documents, and it's having trouble keeping up with them all.