I was looking for a solution to Firestore's limitation of Sequential indexed fields which means the following from this doc.
"Sequential indexed fields" means any collection of documents that
contains a monotonically increasing or decreasing indexed field. In
many cases, this means a timestamp field, but any monotonically
increasing or decreasing field value can trigger the write limit of
500 writes per second.
As per the solution, I can add a shard field in my collection which will contain random value and create a composite index with the timestamp. I am trying to achieve this with the existing fields I have in my Document.
My document has the following fields:
{
users: string[],
createdDate: Firebase Timestamp
....
}
I already have a composite index created: users Arrays createdDate Descending. Also, I have created Exemptions for the fields field from Automatic index settings. The users field will contain a list of firebase auto-generated IDs so definitely its random. Now I am not sure whether the field users will do the job of field shard form the example doc. In this way we can avoid adding a new field and still increase the write rate. Can someone please help me with this?
While I don't have specific experience that says what you're trying to do definitely will or will not work the way you expect, I would assume that it works, based on the fact that the documentation says (emphasis mine):
Add a shard field alongside the timestamp field. Use 1..n distinct values for the shard field. This raises the write limit for the collection to 500*n, but you must aggregate n queries.
If each users array contains different and essentially random user IDs, then the array field values would be considered "distinct" (as two arrays are only equal if their elements are all equal to each other), and therefore suitable for sharding.
Related
I have the following structure in a Firestore collection. The "ranks" collection is updated with documents named after the timestamps. In each document, I have the same fields and values. How can I query all documents for a specific field without parsing the entire document? I.e. I want all values in all documents where field is "aave"?
I am new to Firestore and I've been trying this for several weeks now. I tried limiting with where and considered using sub collection group queries but in my case data is not stored in sub collections. Sorry, for not being able to provide more context, since I couldn't get much closer.
Queries select specific values, or ranges of values, of a known field. There is no support for dynamic field names in a query in Firestore.
But if you want to get all documents where the field aave exists/has any value, you can make use of the fact that in the sort order of values they always start with null. So to get all documents where the field aave exists/has any value, you could do:
firebase.firebase().collection("ranks").where("aave", ">=", null)
I don't want to have automatic indexes created by firestore because I need to remove and add every five minutes 50-100 documents (each doc has +/-60 fields) to my subcollection. This causes of big volume for "Cloud Firestore Index Write Ops" (300k / day for only one user) and Cloud Storage. I don't need to sort, filtering that documents so I suppose I can turn off automatic indexes, right?
I know that I can add exemptions for fields, but I don't know how can I use it for documents in subcollections. What should I pass in Collection ID and Field path if the path for documents is like:
mainCollectionName/{id}/subcollectionName/{document=**}
and when should I select a collection checkbox and when collection group checkbox?
Unfortunately, it's not possible to disable indexes or create exemptions for documents to be indexed. As clarified in this similar post here, this cannot be achieved and there is even a limit of 200 exemptions of fields that can be done - you can check the limits here.
For your case, indeed, you would have to exempt the fields individually and besides that, to create the exemption, to set the collection you use its id and not the path. So, you would only need to set in the Collection ID field the subcollectionName and then the field to be exempted.
In addition to this, feel free to raise a Feature Request in Google's Issue Tracker, so they can check about implementing an exemption of documents in the future.
I am using Firebase FireStore database for the first time and I have the following question.
I have created a calendar collection. This collection will contains document representing events that have to be shown into a Calendar implemented by an Angular application.
So I am defining the following fields for these documents:
id: int. It is a unique identifier of the specific document\event.
title: string. It is the event title.
start_date_time: string. It specifies the date and the time at which the event starts.
end_date_time: string. It specifies the date and the time at which the event ends.
And here I have some doubts:
Is the id field required? From what I know I will have the document UID that will ensure the uniqueness of the document. If not strongly required adopt an id field can be convenient have something like an auto increment field? (I know that I have to handle in some other way the auto increment because FireStore is not a relational DB and doesn't automatically handle it). For example I was thinking that can be useful to order my document from the first inserted one to the last inserted one.
It seems to me that FireStore doesn't handle DateTime field (as done for example by a traditional relational database). Is this assumption correct? How can I correctly handle my start_date_time and end_date_time fields? These field have to contains the date time used by my Angular application? So for example I was thinking that I can define it as string field and put into these fields values as 2020-07-20T07:00:00 representing a specific date and a specific time. It could be considered a valid approach to the problem or not?
Is the id field required?
No fields are required. Firestore is schema-less. The only thing a document requires is a string ID that is unique to the collection where it lives.
There is no autoincrement of IDs. That doesn't scale massively the way Firestore requires. If you need ordering, you will have to define that for yourself according to your needs.
In general, you are supposed to accept the randomly generated IDs that the Firebase client APIs will generate for you. Ordering is typically defined using a field in the document.
It seems to me that FireStore doesn't handle DateTime field
Firestore has a timestamp field type that stores moments in time to nanosecond precision. There is no need to store a formatted string, unless that's something you require for other reasons.
I have the following document structure in firebase:
{
typeId: number,
tripId: number,
locationId: string,
expenseId: number,
createtAt: timestamp
}
I want to query this collection using different 'where' statement everytime. Sometimes user wants to filter by type id and sometimes by locationId or maybe include all of the filters.
But it seems like I would need to create a compound index of each possible permutation? For example: typeId + expenseId, typeId + locationId, location + expenseId, etc, otherwise it doesn't work.
What if I have 20 fields and I want to make it possible to search across all of these?
Could you please help me to construct a query and indexes for the following requirement: Possibility to query across all fields, query can contain one, two, three, all or no fields included in where clause and always has to be ordered descending order by createdAt.
Cloud Firestore automatically creates indexes for the individual fields of your documents. So it can already filter on each field without you have to manually add these indexes.
In many cases it is able to combine these indexes to allow queries on field combinations, by performing a so-called zig-zag-merge-join.
Custom additional indexes are typically only needed once you add an ordering-clause to your query, in addition to filter clauses. If you have such a case, the Firestore client will log an error telling you exactly what index to create (with a link to the Firestore console that is prepopulated to created the index for you).
Firestore usage limits
Maximum write rate to a collection in which documents contain
sequential values in an indexed field
How does this relate to composite indexes? Suppose I have an index on a field occasionId(string) and a field date(timestamp) for a collection events. Would > 500 document creations in a second all with the current server timestamp as their date value cause the limit to be hit or only if they all had the same occasionId value?
I believe that you will still hit the limit since the per field default index of date field is sequential and can not be removed in the beta version.
For more details you can see the discussion on Google Groups and on StackOverflow.