I have a collection where the documents are uniquely identified by a date, and I want to get the n most recent documents. My first thought was to use the date as a document ID, and then my query would sort by ID in descending order. Something like .orderBy(FieldPath.documentId, descending: true).limit(n). This does not work, because it requires an index, which can't be created because __name__ only indexes are not supported.
My next attempt was to use .limitToLast(n) with the default sort, which is documented here.
By default, Cloud Firestore retrieves all documents that satisfy the query in ascending order by document ID
According to that snippet from the docs, .limitToLast(n) should work. However, because I didn't specify a sort, it says I can't limit the results. To fix this, I tried .orderBy(FieldPath.documentId).limitToLast(n), which should be equivalent. This, for some reason, gives me an error saying I need an index. I can't create it for the same reason I couldn't create the previous one, but I don't think I should need to because they must already have an index like that in order to implement the default ordering.
Should I just give up and copy the document ID into the document as a field, so I can sort that way? I know it should be easy from an algorithms perspective to do what I'm trying to do, but I haven't been able to figure out how to do it using the API. Am I missing something?
Edit: I didn't realize this was important, but I'm using the flutterfire firestore library.
A few points. It is ALWAYS a good practice to use random, well distributed documentId's in firestore for scale and efficiency. Related to that, there is effectively NO WAY to query by documentId - and in the few circumstances you can use it (especially for a range, which is possible but VERY tricky, as it requires inequalities, and you can only do inequalities on one field). IF there's a reason to search on an ID, yes it is PERFECTLY appropriate to store in the document as well - in fact, my wrapper library always does this.
the correct notation, btw, would be FieldPath.documentId() (method, not constant) - alternatively, __name__ - but I believe this only works in Queries. The reason it requested a new index is without the () it assumed you had a field named FieldPath with a subfield named documentid.
Further: FieldPath.documentId() does NOT generate the documentId at the server - it generates the FULL PATH to the document - see Firestore collection group query on documentId for a more complete explanation.
So net:
=> documentId's should be as random as possible within a collection; it's generally best to let Firestore generate them for you.
=> a valid exception is when you have ONE AND ONLY ONE sub-document under another - for example, every "user" document might have one and only one "forms of Id" document as a subcollection. It is valid to use the SAME ID as the parent document in this exceptional case.
=> anything you want to query should be a FIELD in a document,and generally simple fields.
=> WORD TO THE WISE: Firestore "arrays" are ABSOLUTELY NOT ARRAYS. They are ORDERED LISTS, generally in the order they were added to the array. The SDK presents them to the CLIENT as arrays, but Firestore it self does not STORE them as ACTUAL ARRAYS - THE NUMBER YOU SEE IN THE CONSOLE is the order, not an index. matching elements in an array (arrayContains, e.g.) requires matching the WHOLE element - if you store an ordered list of objects, you CANNOT query the "array" on sub-elements.
From what I've found:
FieldPath.documentId does not match on the documentId, but on the refPath (which it gets automatically if passed a document reference).
As such, since the documents are to be sorted by timestamp, it would be more ideal to create a timestamp fieldvalue for createdAt rather than a human-readable string which is prone to string length sorting over the value of the string.
From there, you can simply sort by date and limit to last. You can keep the document ID's as you intend.
Related
I'm trying to find the id of a doc where a map value in an array of maps equals "x".
in the following case, I'm trying to find which rep owns the cause with code "hog"
I'll likely be going down the denormalizing route, but is this possible?
Firestore has an array-contains operator that you can use to query whether a certain item exists in an array field, but that operator only works if you specify the exact, complete value of the field. It can't test for a partial match.
The common approach to your use-case is to add an additional array field with just the values you want to query on, i.e.
cause-codes: ["hog"]
Once you modified your documents with this additional field, you can then use a query like:
repsRef.where('cause-codes:', 'array-contains', 'hof')
I want to query by date and status. I created an index for both the fields, but when querying it throws an error:
Cannot have inequality filters on multiple properties: [created_at, status]
const docQuery = db.collection('documents')
.where('created_at', '<=', new Date()) // ex. created_at later than 2 weeks ago
.where('status', '!=', 'processed') // works when == is used
.limit(10);
'status' is a string and can be multiple things.
I read about query limitations, https://firebase.google.com/docs/firestore/query-data/queries but that is such a simple thing for any database to do, what is a good solution for such a problem? Loading entire records is not an option.
The issue isn't that it's a "simple" thing to do. The issue is that it's an unscalable thing to do. Firestore is fast and cheap because of the limitations it places on queries. Limiting queries to a single inequality/range filter allows it to scale massively without requiring arbitrarily large amounts of memory to perform lots of data transformations. It can simply stream results directly to the client without loading them all into enough memory to hold all of the results. While it's not necessary to understand how this works, it's necessary to accept the limitations if you want to use Firestore effectively.
For your particular case, you could make your data more scalable to query by changing to your "status" field from a single string field to a set of boolean fields. For each one of the possible status values, you could instead have a boolean field that indicates the current status. So, for the query you show here, if you had a field called "isProcessed" with a boolean value true/false, you could find the first 10 unprocessed documents created before the current date like this:
const docQuery = db.collection('documents')
.where('created_at', '<=', new Date())
.where('processed', '==', false)
.limit(10);
Yes, this means you have to keep each field up to date when there is a change in status, but you now have a schema that's possible to query at massive scale.
I have some code like this:
...
const snapshot = firestore().collection("orders").orderBy("deliveryDate")
...
I want to access only the 100th order in the returned documents. So far, the only way I achieve this is to do firestore().collection("orders").orderBy("deliveryDate").limit(100) and this returns first 100 documents and I can access the last order. But, I end up fetching 99 unwanted documents and this could become quite slower if I want the 200th document or higher.
So, I basically want to know if there's a possible way of getting just the index I want after sorting.
As far as I know, startAt() and startAfter() only accept a doc reference or field values, not an index/offset
Firestore does not offer any way to offset by some numeric amount to web and mobile clients (and doing so would end up having the exact same cost as what you're doing now).
If you need to impose some sort of offset into your collection, you will need to maintain that in the document itself for querying, or use some other type of storage that gives you fast cheap access by index.
I want to get a collection of documents in descending order of their ids. How do I get that?
What should I put in place of ?? to make that happen?
firebase.firestore().collection("users").orderBy(??, "desc").get();
I think you're looking for
firebase.firestore.FieldPath.documentid()
Note that what this actually returns is not just the local documentID, but the fully qualified path all the way to root - that is actually what firestore indexes.
In your case, you'll only pull from collection "users", so effectively the documentid() will be "root/users/{documentid}"
(Yes, simple index lookups really do just append the collection to the documentID - it's part of Firestore's efficiency that there is "really" only one giant index)
https://firebase.google.com/docs/reference/js/firebase.firestore.FieldPath
Consider the following data in Firestore:
I also have a set of selected by the user he's interested in: ['python', 'sql', 'swift']
How to get all the documents from my collection where tags field contains at least one of tags selected by the user? I'm aware that Firestore doesn't have such operator out of the box. Probably I need to rethink the way I store data somehow? Please point me to the right direction.
With the way you have your data structured, you will need to make one array-contains type query for each of the tags the user is interested in, then merge the results of each of those queries in your app code.