Is there any way to check the read or write count depending on the call made by the code. For instance, let's say I have
final ref = Firestore.instance.document('users/uid');
ref.get();
Now that this is 1 read operation, and I can use
ref.delete()
or
ref.setData(data);
etc.
This is easy for single operations, how about I am listening for a Stream, how do I know how many read/write operations are taking place by my function calls?
It doesn't matter if it's a stream, or iterating an array, or any other form of consumption. Once the query hits the server, you are charged for the number of documents returned by that query, regardless of how you choose to iterate them. They show up in the client app all at the same time, and you get to choose how to deal with them.
For realtime listeners, you are charged a read for each document that gets delivered to your snapshot listener. Again, it doesn't matter what you do with the documents after that - the cost is already paid.
Related
I am using a collection in Firebase Firestore to log some activities but I don't want this log collection to grow forever. Is there a way to set a limit to the number of documents in a collection or a size limit for the whole collection or get a notification if it passes a limit?
OR is there a way to automatically delete old documents in a collection just by settings and not writing some cron job or scheduled function?
Alternatively, what options are there to create a rotational logging system for client activities in Firebase?
I don't want this log collection to grow forever.
Why not? There are no downsides. In Firestore the performance depends on the number of documents you request and not on the number of documents you search. So it doesn't really matter if you search 10 documents in a collection of 100 documents or in a collection of 100 MIL documents, the response time will always be the same. As you can see, the number of documents within a collection is irrelevant.
Is there a way to set a limit to the number of documents in a collection or a size limit for the whole collection or get a notification if it passes a limit?
There is no built-in mechanism for that. However, you can create one mechanism yourself in a very simple way. Meaning, that you can create a document in which you can increment/decrement a numeric value, each time a document is added or deleted from the collection. Once you hit the limit, you can restrict the addition of documents in that particular collection.
OR is there a way to automatically delete old documents in a collection just by settings and not writing some cron job or scheduled function?
There is also no automatic operation that can help you achieve that. You can either use the solution above and once you hit the limit + 1, you can delete the oldest document. Or you can use a Cloud Function for Firebase to achieve the same thing. I cannot see any reason why you should use a cron job. You can use a Cloud Scheduler to perform some operation at a specific time, but as I understand you want it to happen automatically when you hit the limit.
Alternatively, what options are there to create a rotational logging system for client activities in Firebase?
If you still don't want to have larger collections, maybe you can export the data into a file and add that file to Cloud Storage for Firebase.
I'm a firebase user recently diving into rtdb, and just found a limit docs explaining write limit for a single db instance, saying the quote below:
The limit on write operations per second on a single database. While not a hard limit, if you sustain more than 1,000 writes per second, your write activity may be rate-limited.
In firestore's security rules for example, delete operation is in the category of write operation, and i guess such concept would be applicable to other firebase services. So i want to exactly know if delete operation is subject to write limit for rtdb instance.
FYI, i'm planning to use the latest node js admin sdk with cloud functions to operate a huge number of deletes, using this link's method for huge number of different paths.
So, if the delete op is subject to rtdb write operation, it seems to be a critical mistake to deploy this function even if only few number of users are likely to trigger this function concurrently. And even few concurrent invocations would soon max out the per-second write limit, considering that firebase admin sdk is good at iterating those ops really quickly.
Since i have to specify the id(key) of path for each removal(-so that no nested data would be deleted unintentionally), simply deleting parent path is not applicable to this situation, and even really dangerous..
If delete op is not subject to write limit, then i also want to know if there is truly no single limit for delete operations for rtdb!! Hope this question reach to firebase gurus in the community! Comments are welcomed and appreciate! Thank you in advance [:
A delete operation does count as a write operation. If you run 20K delete operations i.e. 20K separate .remove() operations simultaneously using Promise.all(), they all will be counted as unique operation and you'll be rate limited. Those additional delete requests over the limit will take time to succeed.
Instead if you are using a Cloud function you can create a single object including all paths to be deleted and use update() to remove all those nodes in a single write operation. Let's say you have a root node users and each user node has a points node and you want to remove it from all the users.
const remObject = {
"user_id_1/points": null,
"user_id_2/points": null
}
await admin.database().ref("users").update(remObject)
Although you would need to know IDs of all users, this will remove points node from all users in a single operation and hence you won't be rate limited. Another benefit of doing this would be all those nodes will be deleted for sure unlike executing individual requests where some of them may fail.
If you run different `remove()` operation for each user as shown below, then it'll count as N writes where N is number of operations.
const userIDs = []
const removeRequests = userIDs.map(u => admin.database().ref(`users/${u}/points`).remove())
await Promise.all(removeRequests)
// userIDs.length writes which will count towards that rate limit
I ran some test functions with above code and no surprise both adding and removing 20K nodes using distinct operations with Promise.all() took over 40 seconds while using a single update operation with an object took just 3.
Do note that using the single update method maybe limited by "Size of a single write request to the database" which is 16 MB for SDKs and 256 MB for REST API. In such cases, you may have to break down the object in smaller parts and use multiple update() operations.
How does firebase query works?
for example, if i write this query,
var collectionReference = FirebaseFirestore.instance
.collection('collection')
.where(cond)
.where(cond2)
.where(cond3);
So is this gonna return only the documents which fit the conditions?
AND I AM GOING TO GET CHARGED ONLY FOR THOSE DOCUMENT READS?
from docs TL:DR
Charges for reads have some nuances that you should keep in mind. The following sections explain these nuances in detail.
Listening to query results
Cloud Firestore allows you to listen to the results of a query and get realtime updates when the query results change.
When you listen to the results of a query, you are charged for a read each time a document in the result set is added or updated. You are also charged for a read when a document is removed from the result set because the document has changed. (In contrast, when a document is deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for example, if the user goes offline), you will be charged for reads as if you had issued a brand-new query.
Managing large result sets
Cloud Firestore has several features to help you manage queries that return a large number of results:
Cursors, which allow you to resume a long-running query.
Page tokens, which help you paginate the query results.
Limits, which specify how many results to retrieve
Offsets, which allow you to skip a fixed number of documents.
There are no additional costs for using cursors, page tokens, and limits. In fact, these features can help you save money by reading only the documents that you actually need.
However, when you send a query that includes an offset, you are charged a read for each skipped document. For example, if your query uses an offset of 10, and the query returns 1 document, you are charged for 11 reads. Because of this additional cost, you should use cursors instead of offsets whenever possible.
Queries other than document reads
For queries other than document reads, such as a request for a list of collection IDs, you are billed for one document read. If fetching the complete set of results requires more than one request (for example, if you are using pagination), you are billed once per request.
Minimum charge for queries
There is a minimum charge of one document read for each query that you perform, even if the query returns no results.
I know that whenever I turn on a listener to a QUERY, the minimum charge is one read, even if this query returns no documents (because there are no documents that fit the query, or even if there are but there were no changes since last time this listener was on).
According to documentation:
Minimum charge for queries
There is a minimum charge of one document read for each query that you perform, even if the query returns no results.
Stream<QuerySnapshot> stream = Firestore().collection("someCollection").where("someField", isGreaterThan: 42).snapshots();
stream.listen((querySnapshot) => doSomething());
Also, I know that I'm not charged if I use a simple get to read some document, as long as this document already exists in the offline cache,
and was not changed in Firestore (According to Doug Stevenson).
DocumentSnapshot doc = await Firestore().collection("someCollection").document("documentID").get();
My question is:
Suppose a document already exists in my offline cache, and was not changed in Firestore since.
If instead of a get I turn on a listener to this single document
(DocumentSnapshot, NOT a QuerySnapshot), like this:
Stream<DocumentSnapshot> stream = Firestore().collection("someCollection").document("documentID").snapshots();
stream.listen((documentSnapshot) => doSomething());
Will I be charged for one read when this listener is turned on?
Will I be charged for one read when this listener is turned on?
No. If you turn on a listener that is listening to a single document, you are not charged at all since the document already exists in the cache. Remember, you are charged with one document read when a query does not return any results but in your case, you aren't performing any query, you are just listening to a document change. It makes no sense to be charged with one document read each time you start listening for changes or each time you open the application. That's the beauty of Firestore, you have a local copy of your database so you cannot be charged for cached documents, unless the documents are changed in the back-end.
I am use Firestore and try to remove race condition in Flutter app by use transaction.
I have subcollection where add 2 document maximum.
Race condition mean more than 2 document may be add because client code is use setData. For example:
Firestore.instance.collection(‘collection').document('document').collection('subCollection’).document(subCollectionDocument2).setData({
‘document2’: documentName,
});
I am try use transaction to make sure maximum 2 document are add. So if collection has been change (For example new document add to collection) while transaction run, the transaction will fail.
But I am read docs and it seem transaction use more for race condition where set field in document, not add document in subcollection.
For example if try implement:
Firestore.instance.collection(‘collection').document('document').collection('subCollection').runTransaction((transaction) async {
}),
Give error:
error: The method 'runTransaction' isn't defined for the class 'CollectionReference'.
Can transaction be use for monitor change to subcollection?
Anyone know other solution?
Can transaction be use for monitor change to subcollection?
Transactions in Firestore work by a so-called compare-and-swap operation. In a transaction, you read a document from the database, determine its current state, and then set its new state based on that. When you've done that for the entire transaction, you send the whole package of current-state-and-new-state documents to the server. The server then checks whether the current state in the storage layer still matches what your client started with, and if so it commits the new state that you specified.
Knowing this, the only way it is possible to monitor an entire collection in a transaction is to read all documents in that collection into the transaction. While that is technically possible for small collections, it's likely to be very inefficient, and I've never seen it done in practice. Then again, for just the two documents in your collection it may be totally feasible to simply read them in the transaction.
Keep in mind though that a transaction only ensures consistent data, it doesn't necessarily limit what a malicious user can do. If you want to ensure there are never more than two documents in the collection, you should look at a server-side mechanism.
The simplest mechanism (infrastructure wise) is to use Firestore's server-side security rules, but I don't think those will work to limit the number of documents in a collection, as Doug explained in his answer to Limit a number of documents in a subcollection in firestore rules.
The most likely solution in that case is (as Doug also suggests) to use Cloud Functions to write the documents in the subcollection. That way you can simply reject direct writes from the client, and enforce any business logic you want in your Cloud Functions code, which runs in a trusted environment.