I'm running into an issue with Firestore that is pretty unintuitive to me and so I'm wondering if anyone can help me understand why Firestore is giving this result.
Let's say I have a collection called "people" with fields "firstName" and "lastName". Let's say I have 20 documents in that collection that have the lastName "Quaid". I then have a field "inCanada" that is only present in a subset of those 20 documents, let's say 5. The other 15 documents don't have that field.
What is surprising is that an orderBy clause using that optional field is actually filtering the result set rather than just sorting it, which doesn't make sense to me when compared with other databases.
In the below example, I expected both result sets to have 20 elements, however the second set that has the orderBy only has those 5 documents where that field is present.
Could someone please explain why Firestore does this?
const Admin = require('firebase-admin');
(async function () {
const initFirebase = require('../initFirebase');
initFirebase();
const people = Admin.firestore().collection('people');
const quaids = people.where('lastName', '==', 'Quaid')
const quaids1 = await quaids.get();
const quaids2 = await quaids.orderBy('inCanada', 'desc').get();
console.log(quaids1._size); // 20
console.log(quaids2._size); // 5
})();
In a nutshell, this is because Firestore queries are based on indexes: each time you create a doc Firestore creates an index for each field of the doc.
So since the field "inCanada" is only present "in a subset of the 20 documents", only this subset of documents is present in the "inCanada" index, resulting in the situation you mention.
This official video from Todd Kerpelman from the Firebase team explains it very well: https://www.youtube.com/watch?v=Ofux_4c94FI#t=4m17s. (This link will open the video at 4m17s i.e. when the explanations on the index mechanism start. However the part on the impact of indexes on the querying possibilities is more around 6m22s!)
If you want to include the other documents in your query results you should write a value for the field "inCanada" for those documents, for example use a false value (and use true for the docs that are "inCanada").
Related
I've got a Collection message which contains both a document (representing the chat room) and a nested collection which contains all the messages of that conversation.
Now I'd like to request all the documents (chat rooms) in which my user is involved. So if one of the id1 or id2 fields in the users map is equal to my user id, I collect that document.
I've noticed that I can't use array queries as I'm using maps and not arrays.
So I don't know what would be the best approach to proceed to that query.
Stream<List<ChatRoom>> getChatRooms(String userId) {
final messageCollection = FirebaseFirestore.instance.collection('messages');
// var query = messagesCollection.where('users' in [userId])
See Frank's comment below: The best solution is to "use an array of user IDs (userIDs: ["ABCDEF", "GHIJKL"]) and an array-contains condition. In this use-case it seems that would save on the number of needed indexes (and thus on the cost of storage)".
If you really need to keep the map for other reasons, you can very well have the two fields in the doc. It's not a problem to duplicate the data.
If the value you assign to each userId in the map does not have to be meaningful, you can assign a Boolean value of true and then query as follows:
final messageCollection = FirebaseFirestore.instance.collection('messages');
messageCollection.where('users.id1', isEqualTo: true)
.get()
.then(...);
So, your map will look like:
users
id1: true
id2: true
Clarification:
The idea is to use the user ids as keys and have a value of true. Let's imagine two users with the following ids: "ABCDEF" and "GHIJKL"
Instead of having a map like
users
id1: "ABCDEF"
id2: "GHIJKL"
you could have it as follows:
users
ABCDEF: true
GHIJKL: true
Note that you could very well have the two maps in the doc, if for some reason you really need to keep the first map.
I am trying to fetch all documents whose sub-collection contain a specific document ID. Is there any way to do this?
For example, if the boxed document under 'enquiries' sub-collection exists, then I need the boxed document ID from 'books' collection. I couldn't figure out how to go backwards to get the parent document ID.
I make the assumption that all the sub-collections have the same name, i.e. enquiries. Then, you could do as follows:
Add a field docId in your enquiries document that contains the document ID.
Execute a Collection Group query in order to get all the documents with the desired docId value (Firestore.instance.collectionGroup("enquiries").where("docId", isEqualTo: "ykXB...").getDocuments()).
Then, you loop over the results of the query and for each DocumentReference you call twice the parent() methods (first time you will get the CollectionReference and second time you will get the DocumentReference of the parent document).
You just have to use the id property and you are done.
Try the following:
Firestore.instance.collection("books").where("author", isEqualTo: "Arumugam").getDocuments().then((value) {
value.documents.forEach((result) {
var id = result.documentID;
Firestore.instance.collection("books").document(id).collection("enquiries").getDocuments().then((querySnapshot) {
querySnapshot.documents.forEach((result) {
print(result.data);
});
First you need to retrieve the id under the books collection, to be able to do that you have to do a query for example where("author", isEqualTo: "Arumugam"). After retrieving the id you can then do a query to retrieve the documents inside the collection enquiries
For example, if the boxed document under 'enquiries' sub-collection exists, then I need the boxed document ID from 'books' collection.
There is no way you can do that in a single go.
I couldn't figure out how to go backwards to get the parent document ID.
There is no going back in Firestore as you probably were thinking. In Firebase Realtime Database we have a method named getParent(), which does exactly what you want but in Firestore we don't.
Queries in Firestore are shallow, meaning that it only get items from the collection that the query is run against. Firestore doesn't support queries across different collections in one go. A single query may only use the properties of documents in a single collection. So the solution to solving your problem is to perform two get() calls. The first one would be to check that document for existence in the enquiries subcollection, and if it exists, simply create another get() call to get the document from the books collection.
Renaud Tarnec's answer is great for fetching the IDs of the relevant books.
If you need to fetch more than the ID, there is a trick you could use in some scenarios. I imagine your goal is to show some sort of an index of all books associated with a particular enquiry ID. If the data you'd like to show in that index is not too long (can be serialized in less than 1500 bytes) and if it is not changing frequently, you could try to use the document ID as the placeholder for that data.
For example, let's say you wanted to display a list of book titles and authors corresponding to some enquiryId. You could create the book ID in the collection with something like so:
// Assuming admin SDK
const bookId = nanoid();
const author = 'Brandon Sanderson';
const title = 'Mistborn: The Final Empire';
// If title + author are not unique, you could add the bookId to the array
const uniquePayloadKey = Buffer.from(JSON.stringify([author, title])).toString('base64url');
booksColRef.doc(uniquePayloadKey).set({ bookId })
booksColRef.doc(uniquePayloadKey).collection('enquiries').doc(enquiryId).set({ enquiryId })
Then, after running the collection group query per Renaud Tarnec's answer, you could extract that serialized information with a regexp on the path, and deserialize. E.g.:
// Assuming Web 9 SDK
const books = query(collectionGroup(db, 'enquiries'), where('enquiryId', '==', enquiryId));
return getDocs(books).then(snapshot => {
const data = []
snapshot.forEach(doc => {
const payload = doc.ref.path.match(/books\/(.*)\/enquiries/)[1];
const [author, title] = JSON.parse(atob(details));
data.push({ author, title })
});
return data;
});
The "store payload in ID" trick can be used only to present some basic information for your child-driven search results. If your book document has a lot of information you'd like to display once the user clicks on one of the books returned by the enquiry, you may want to store this in separate documents whose IDs are the real bookIds. The bookId field added under the unique payload key allows such lookups when necessary.
You can reuse the same data structure for returning book results from different starting points, not just enquiries, without duplicating this structure. If you stored many authors per book, for example, you could add an authors sub-collection to search by. As long as the information you want to display in the resulting index page is the same and can be serialized within the 1500-byte limit, you should be good.
The (quite substantial) downside of this approach is that it is not possible to rename document IDs in Firestore. If some of the details in the payload change (e.g. an admin fixes a book titles), you will need to create all the sub-collections under it and delete the old data. This can be quite costly - at least 1 read, 1 write, and 1 delete for every document in every sub-collection. So keep in mind it may not be pragmatic for fast changing data.
The 1500-byte limit for key names is documented in Usage and Limits.
If you are concerned about potential hotspots this can generate per Best Practices for Cloud Firestore, I imagine that adding the bookId as a prefix to the uniquePayloadKey (with a delimiter that allows you to throw it away) would do the trick - but I am not certain.
so here i have
So I am making dating app, and now i come to the moment, where i need to create a list of people to match with for user.
So I need a firebase query to look in to gender and check if the people already matched, if yes it should not be included in the list.
I tried, filtering data by gender. How I should edit this query to check if they already matched?
Matches are displayed in users/{userID}/matches/{matchedUserID}
This is what i tried:
async createUserListAndNavigate() {
let genderTolook = 'female';
let data = firebase
.database()
.ref('users/')
.orderByChild('gender')
.equalTo(genderTolook);
data.once('value', snapshot => {
console.log(JSON.stringify(snapshot.val()));
});
}
You can use the array-contains-any query to check if the matches array already contains the user_id.
var users = db.collection("users");
users.where('tempuser.matches', 'array-contains-any', [<USER_ID_TO_MATCH>]);
Based on the result you can make further queries/operations. You could also combine this query with other queries.
See the documentation for further details.
I'm trying to figure out if there's a reasonable way of doing this:
My problem:
Exceeding my daily quota for reads in firestore pretty fast.
My database and what I do:
My database looks like this (simplified):
sessions: { // collection
sessionId: { // document
users: { // collection
userId: { // document
id: string
items: { // collection
itemId: trackObject
}
}
}
}
}
Now I want to retrieve from one session, all users and their items. Most sessions have 2-3 users but some users have around 3000 items. I basically want to retrieve an array like this:
[
{
userId,
items: [
...items
],
},
...users
]
How I go about it currently:
So I get all users:
const usersRef = db.collection(`sessions/${sessionId}/users`);
const userSnapshots = await usersRef.get();
const userDocs = userSnapshots.docs;
Then for each user I retrieve their items:
(I use a for-loop which can be discussed but anyhow)
const user = userDocs[i].data();
const itemsRef = usersRef.collection(`${user.id}/items`);
const itemSnapshots = await itemRef.get();
const items = itemSnapshots.docs
Finally I retrieve the actual items through a map:
user.items = items.map(doc => doc.data());
return user;
My theory:
So it looks like if I do this on a session where a user has 3000 items, the code will perform 3000 read operations on firestore. After just 17 runs I eat up my 50000 operations a day.
This reasoning is somewhat based on this answer.
My question:
Is there any other way of doing this? Like getting all tracks in one read-call? Should I see if I can fit all the items into an array-key in the user-object instead of storing as a collection? Is the free version of firestore simply not designed for this many documents being retrieved in one go?
If you're trying to reduce the number of document reads, you'll need to reduce the number of documents that you need to read to implement your use-case.
For example, it is fairly unlike that a user of your app will want to read the details of all 3000 items. So you might want to limit how many items you initially read, and load the additional items only on demand.
Also consider if each item needs to be its own document, or whether you could combine all items of a user into a single document. For example, if you never query the individual items, there is no need to store them as separate documents.
Another thing to consider if whether you can combine common items into a single document. An example of this is, even if you keep the items in a separate subcollection, to keep the names and ids of the most recent 30 items for a user in the user's document. This allows you to easily show a user and their 30 most recent items. Doing this you're essentially pre-rendering those 30 items of each user, significantly reducing the number of documents you need to read.
To learn more on data modeling considerations, see:
Cloud Firestore Payments
Going over read-quota in firebase firestore
the video series Getting to know Cloud Firestore, specifically What is a NoSQL Database? How is Cloud Firestore structured? and How to Structure Your Data
this article on NoSQL data modeling
Let's say I've got 100 documents in my collection. My goal is to get the documents between 5 and 33.
I tried startAt and endAt but it didn't work:
const db = firebase.firestore();
await db
.collection("pictures")
.startAt(start)
.endAt(end)
.get()
What you're trying to do isn't really possible. Documents don't have a natural index or position within a collection. To get ordering within a collection, you need to use at least one field on which you want to sort the documents. Only then do they have an order, and only then can you page through them.
The startAt and endAt methods on the query require that you define some order. You can see that in the example code provided in the API docs I linked to. Note the following statement in the docs for startAt:
Creates and returns a new Query that starts at the provided set of
field values relative to the order of the query. The order of the
provided values must match the order of the order by clauses of the
query.