Firestore usage limits
Maximum write rate to a collection in which documents contain
sequential values in an indexed field
How does this relate to composite indexes? Suppose I have an index on a field occasionId(string) and a field date(timestamp) for a collection events. Would > 500 document creations in a second all with the current server timestamp as their date value cause the limit to be hit or only if they all had the same occasionId value?
I believe that you will still hit the limit since the per field default index of date field is sequential and can not be removed in the beta version.
For more details you can see the discussion on Google Groups and on StackOverflow.
Related
A question for the Firebase Firestore gurus out there.
I'm wanting to query a users collection for all documents where the bio field has a character length of <n
Is this possible?
I'm thinking this might have to be done post-query with some JS.
Firestore can only order/filter on values that are stored in the documents it returns. It cannot calculate any values, nor look values up.
So if you want to filter on the length of the bio, you'll have to store that value in a field in the document (e.g. bioLength) and update that each time you also update the bio.
With that field in place, you can then filter on it in a query.
I have some documents in Firestore with two Timestamp fields named lastUpdated and lastProcessed in addition to other fields. lastUpdated field is updated when a user updates the record's fields via web console. lastProcessed field is updated when the backend function processes the document (as a result of user clicking a button).
Following are the possible combinations of these 2 fields.
User has only updated the document, but yet to process (lastUpdated == some_timestamp, lastProcessed == '')
User has updated the document, and then processed (lastUpdated < lastProcessed)
User has updated the document, processed and re-updated (lastUpdated > lastProcessed)
My requirement is to execute a query to get a subset of these records (say top 10), ordered by its most recent timestamp. So when evaluating a record for the ordering, lastUpdated field should be considered for scenarios 1 and 3 above. But lastProcessed field should be considered for scenario 2.
Is this possible with Firestore?
When querying the Firestore database it is not possible to execute the logic you explain in your question (i.e. calculate on the fly the scenario to be applied and define which field shall be used in the query).
One classical solution is to add an extra field to the document which contains the value to be queried for. The value of this field can be calculated (according to the business logic) when you modify the document from your frontend, or via a Cloud Function triggered in the backend each time the doc is changed.
The main advantage of using a Cloud Function is to prevent users modifying the value of this field.
I have the following structure in a Firestore collection. The "ranks" collection is updated with documents named after the timestamps. In each document, I have the same fields and values. How can I query all documents for a specific field without parsing the entire document? I.e. I want all values in all documents where field is "aave"?
I am new to Firestore and I've been trying this for several weeks now. I tried limiting with where and considered using sub collection group queries but in my case data is not stored in sub collections. Sorry, for not being able to provide more context, since I couldn't get much closer.
Queries select specific values, or ranges of values, of a known field. There is no support for dynamic field names in a query in Firestore.
But if you want to get all documents where the field aave exists/has any value, you can make use of the fact that in the sort order of values they always start with null. So to get all documents where the field aave exists/has any value, you could do:
firebase.firebase().collection("ranks").where("aave", ">=", null)
I don't want to have automatic indexes created by firestore because I need to remove and add every five minutes 50-100 documents (each doc has +/-60 fields) to my subcollection. This causes of big volume for "Cloud Firestore Index Write Ops" (300k / day for only one user) and Cloud Storage. I don't need to sort, filtering that documents so I suppose I can turn off automatic indexes, right?
I know that I can add exemptions for fields, but I don't know how can I use it for documents in subcollections. What should I pass in Collection ID and Field path if the path for documents is like:
mainCollectionName/{id}/subcollectionName/{document=**}
and when should I select a collection checkbox and when collection group checkbox?
Unfortunately, it's not possible to disable indexes or create exemptions for documents to be indexed. As clarified in this similar post here, this cannot be achieved and there is even a limit of 200 exemptions of fields that can be done - you can check the limits here.
For your case, indeed, you would have to exempt the fields individually and besides that, to create the exemption, to set the collection you use its id and not the path. So, you would only need to set in the Collection ID field the subcollectionName and then the field to be exempted.
In addition to this, feel free to raise a Feature Request in Google's Issue Tracker, so they can check about implementing an exemption of documents in the future.
I was looking for a solution to Firestore's limitation of Sequential indexed fields which means the following from this doc.
"Sequential indexed fields" means any collection of documents that
contains a monotonically increasing or decreasing indexed field. In
many cases, this means a timestamp field, but any monotonically
increasing or decreasing field value can trigger the write limit of
500 writes per second.
As per the solution, I can add a shard field in my collection which will contain random value and create a composite index with the timestamp. I am trying to achieve this with the existing fields I have in my Document.
My document has the following fields:
{
users: string[],
createdDate: Firebase Timestamp
....
}
I already have a composite index created: users Arrays createdDate Descending. Also, I have created Exemptions for the fields field from Automatic index settings. The users field will contain a list of firebase auto-generated IDs so definitely its random. Now I am not sure whether the field users will do the job of field shard form the example doc. In this way we can avoid adding a new field and still increase the write rate. Can someone please help me with this?
While I don't have specific experience that says what you're trying to do definitely will or will not work the way you expect, I would assume that it works, based on the fact that the documentation says (emphasis mine):
Add a shard field alongside the timestamp field. Use 1..n distinct values for the shard field. This raises the write limit for the collection to 500*n, but you must aggregate n queries.
If each users array contains different and essentially random user IDs, then the array field values would be considered "distinct" (as two arrays are only equal if their elements are all equal to each other), and therefore suitable for sharding.