I have the following document structure in firebase:
{
typeId: number,
tripId: number,
locationId: string,
expenseId: number,
createtAt: timestamp
}
I want to query this collection using different 'where' statement everytime. Sometimes user wants to filter by type id and sometimes by locationId or maybe include all of the filters.
But it seems like I would need to create a compound index of each possible permutation? For example: typeId + expenseId, typeId + locationId, location + expenseId, etc, otherwise it doesn't work.
What if I have 20 fields and I want to make it possible to search across all of these?
Could you please help me to construct a query and indexes for the following requirement: Possibility to query across all fields, query can contain one, two, three, all or no fields included in where clause and always has to be ordered descending order by createdAt.
Cloud Firestore automatically creates indexes for the individual fields of your documents. So it can already filter on each field without you have to manually add these indexes.
In many cases it is able to combine these indexes to allow queries on field combinations, by performing a so-called zig-zag-merge-join.
Custom additional indexes are typically only needed once you add an ordering-clause to your query, in addition to filter clauses. If you have such a case, the Firestore client will log an error telling you exactly what index to create (with a link to the Firestore console that is prepopulated to created the index for you).
Related
I have a collection where the documents are uniquely identified by a date, and I want to get the n most recent documents. My first thought was to use the date as a document ID, and then my query would sort by ID in descending order. Something like .orderBy(FieldPath.documentId, descending: true).limit(n). This does not work, because it requires an index, which can't be created because __name__ only indexes are not supported.
My next attempt was to use .limitToLast(n) with the default sort, which is documented here.
By default, Cloud Firestore retrieves all documents that satisfy the query in ascending order by document ID
According to that snippet from the docs, .limitToLast(n) should work. However, because I didn't specify a sort, it says I can't limit the results. To fix this, I tried .orderBy(FieldPath.documentId).limitToLast(n), which should be equivalent. This, for some reason, gives me an error saying I need an index. I can't create it for the same reason I couldn't create the previous one, but I don't think I should need to because they must already have an index like that in order to implement the default ordering.
Should I just give up and copy the document ID into the document as a field, so I can sort that way? I know it should be easy from an algorithms perspective to do what I'm trying to do, but I haven't been able to figure out how to do it using the API. Am I missing something?
Edit: I didn't realize this was important, but I'm using the flutterfire firestore library.
A few points. It is ALWAYS a good practice to use random, well distributed documentId's in firestore for scale and efficiency. Related to that, there is effectively NO WAY to query by documentId - and in the few circumstances you can use it (especially for a range, which is possible but VERY tricky, as it requires inequalities, and you can only do inequalities on one field). IF there's a reason to search on an ID, yes it is PERFECTLY appropriate to store in the document as well - in fact, my wrapper library always does this.
the correct notation, btw, would be FieldPath.documentId() (method, not constant) - alternatively, __name__ - but I believe this only works in Queries. The reason it requested a new index is without the () it assumed you had a field named FieldPath with a subfield named documentid.
Further: FieldPath.documentId() does NOT generate the documentId at the server - it generates the FULL PATH to the document - see Firestore collection group query on documentId for a more complete explanation.
So net:
=> documentId's should be as random as possible within a collection; it's generally best to let Firestore generate them for you.
=> a valid exception is when you have ONE AND ONLY ONE sub-document under another - for example, every "user" document might have one and only one "forms of Id" document as a subcollection. It is valid to use the SAME ID as the parent document in this exceptional case.
=> anything you want to query should be a FIELD in a document,and generally simple fields.
=> WORD TO THE WISE: Firestore "arrays" are ABSOLUTELY NOT ARRAYS. They are ORDERED LISTS, generally in the order they were added to the array. The SDK presents them to the CLIENT as arrays, but Firestore it self does not STORE them as ACTUAL ARRAYS - THE NUMBER YOU SEE IN THE CONSOLE is the order, not an index. matching elements in an array (arrayContains, e.g.) requires matching the WHOLE element - if you store an ordered list of objects, you CANNOT query the "array" on sub-elements.
From what I've found:
FieldPath.documentId does not match on the documentId, but on the refPath (which it gets automatically if passed a document reference).
As such, since the documents are to be sorted by timestamp, it would be more ideal to create a timestamp fieldvalue for createdAt rather than a human-readable string which is prone to string length sorting over the value of the string.
From there, you can simply sort by date and limit to last. You can keep the document ID's as you intend.
I have the following structure in a Firestore collection. The "ranks" collection is updated with documents named after the timestamps. In each document, I have the same fields and values. How can I query all documents for a specific field without parsing the entire document? I.e. I want all values in all documents where field is "aave"?
I am new to Firestore and I've been trying this for several weeks now. I tried limiting with where and considered using sub collection group queries but in my case data is not stored in sub collections. Sorry, for not being able to provide more context, since I couldn't get much closer.
Queries select specific values, or ranges of values, of a known field. There is no support for dynamic field names in a query in Firestore.
But if you want to get all documents where the field aave exists/has any value, you can make use of the fact that in the sort order of values they always start with null. So to get all documents where the field aave exists/has any value, you could do:
firebase.firebase().collection("ranks").where("aave", ">=", null)
I was looking for a solution to Firestore's limitation of Sequential indexed fields which means the following from this doc.
"Sequential indexed fields" means any collection of documents that
contains a monotonically increasing or decreasing indexed field. In
many cases, this means a timestamp field, but any monotonically
increasing or decreasing field value can trigger the write limit of
500 writes per second.
As per the solution, I can add a shard field in my collection which will contain random value and create a composite index with the timestamp. I am trying to achieve this with the existing fields I have in my Document.
My document has the following fields:
{
users: string[],
createdDate: Firebase Timestamp
....
}
I already have a composite index created: users Arrays createdDate Descending. Also, I have created Exemptions for the fields field from Automatic index settings. The users field will contain a list of firebase auto-generated IDs so definitely its random. Now I am not sure whether the field users will do the job of field shard form the example doc. In this way we can avoid adding a new field and still increase the write rate. Can someone please help me with this?
While I don't have specific experience that says what you're trying to do definitely will or will not work the way you expect, I would assume that it works, based on the fact that the documentation says (emphasis mine):
Add a shard field alongside the timestamp field. Use 1..n distinct values for the shard field. This raises the write limit for the collection to 500*n, but you must aggregate n queries.
If each users array contains different and essentially random user IDs, then the array field values would be considered "distinct" (as two arrays are only equal if their elements are all equal to each other), and therefore suitable for sharding.
I just watched the playlist of Get to know Cloud Firestore, and I just learned that every field of the document is indexed.
My question is, is there way for a certain fields to be excluded on by Firestore indexing? Something like fields that I am pretty sure that I will not be using as a query lookup.
Thanks.
As you correctly found, Firestore automatically indexes all individual fields of the documents in the collection. You can exclude certain fields from the single field indexes panel in the Firebase console.
From there:
Cloud Firestore creates the indexes defined by your automatic index settings for each field you add, enabling most simple queries by default. You can add exemptions to manually set how a specific field is indexed.
From there, you can enter the collection (or collection group), and the field name, and then select which indexes (ascending, descending, arrays) get auto-created or not.
I'm building an app using React + Redux + Firebase. In Firestore users collection, I have multiple fields including language, subject as map data. So for example,
// language that user can speak
language = {
english: true
}
// subjects that the user can teach
subject = {
math: true
science: true
}
In the main app, user can search other users by filtering subjects and languages. For example, user may look for the other users who can teach 'math' and 'science' using an 'english' language.
And I'm struggling with filtering multiple fields in map data. I could query for on field by doing ref.where('subject.${val}', '==' , true). But is there any way to query on multiple fields to get a document which contains a subject hash at any given number of subjects?
I tried to put data in array but currently firebase Array membership only support filtering one item in the array. So I guess it's a good start with storing in hash map. Any suggestion?
I tried to put data in array but currently firebase Array membership only support filtering one item in the array.
You're right, you can filter your items only by a single item that exist in an array. If you'll chain more than one function call, you'll get an error similar with this:
Invalid Query. Queries only support having a single array-contains filter.
And to answer your question:
But is there any way to query on multiple fields to get a document which contains a subject hash at any given number of subjects?
Unfortunately, there are no wildcards in Firestore. You have to identify the property names of your documents by their exact property name.
If you want to limit the results by filtering on multiple properties, you need to chain where() functions, like in the following example:
ref.where('subject.math', '==' , true)
.where('subject.science', '==' , true)
.where('language.english', '==' , true)