Related
I am using a scheduled task in a Firebase Cloud Function to query an array which contains a number of objects that need to be updated if a matching condition exists. My current attempt is using the 'array-contains' method to get the objects, then loop over them to find a matching condition which will then batch update the items. This is my data structure:
I need to find an object that is <= the current time, and also if the 'active' value = false.
export const liveMeetingsTrigger = functions.runWith( { memory: '1GB' }).pubsub
.schedule('every 1 minutes').onRun(async context => {
const now = admin.firestore.Timestamp.now();
const liveMeetings = await admin.firestore().collection('fl_content').where('meeting', 'array-contains', 'liveMeetingDate').get();
const batch = admin.firestore().batch();
liveMeetings.forEach(doc => {
if(doc.data().liveMeetingDate <= now && doc.data().active == false){
batch.update(doc.ref,'active',true);
}
});
return await batch.commit();
});
I have also tried using an exact object in the query instead of just using 'liveMeetingDate', but still get no results back, any help would be great - thanks.
Debugging: As the array I am trying to reach is inside of the (map) object 'liveMeetings' i have tried the dot notation (liveMeetings.meeting) with no success. Also trying a new collection with the the 'meeting' array at top level has provided no success.
Simple logging in the console (liveMeetings.size) shows that nothing is being returned on the query, so therefore the logging does not even reach the loop in the code.
As explained in this anwser the following query will not work:
const liveMeetings = await admin.firestore().collection('fl_content').where('meeting', 'array-contains', 'liveMeetingDate').get();
because the meetings array contain some objects, instead of "simple" or primitive data (e.g. string, number...).
You could query it with the exact objects, like:
const obj = {active: false, liveMeetingDate: ..., meetingId: ..., ....};
const liveMeetings = await admin.firestore().collection('fl_content').where('meeting', 'array-contains', 'obj').get();
Another approach would be to create a new collection which contains the similar documents (same Document ID) but with a meeting Array that contains only the liveMeetingDate property.
Finally, note that since your Array is within a map, you need to do
await admin.firestore().collection('fl_content').where('liveMeetings.meeting', 'array-contains', ...).get();
(PS: I don't mark this question as duplicate since you expressly ask for more help in the comments of the duplicate question/answer)
I have a collection named 'test', which has multiple documents.
Each document has a unique name and a number field.
I have implemented pagination using query cursor. The query to get initial documents is as follows:
let lastDocument={};
ref.collection("test").orderBy('number', 'desc').limit(3).get()
.then(snapshot => {
lastDocument = snapshot.docs[snapshot.docs.length-1];
})
.catch(err => {
console.log("Error getting documents", err);
});
After getting the lastDocument, I paginate it as follows:
ref.collection("test").orderBy('number', 'desc').startAfter(lastDocument).limit(3); //lastDocument has the last document of the snapshot that was retrieved.
In my collection, if there are 2 or more documents which have same value of field 'number', then I don't get that document in the query result. Whereas, If I change the value of the field 'number', so as to make all the documents unique, I get all the results in batches(according to the limit clause) properly.
It's mentioned in the document as well, that, if multiple fields have same population value, the query won't work https://firebase.google.com/docs/firestore/query-data/query-cursors#paginate_a_query
How should I modify the query, so as to get all documents in the result, even if 2 or more documents have same values for the field used in orderBy?
I am using:
"react-native-firebase": "^5.5.6",
"react-native": "^0.59.8",
"firebase": "^6.6.1",
Update: I have tested the queries on JSbin and it returns proper
results. I have issues when I use it in react native app with
react-native-firebase library. The pattern that I have recognised so
far is as follows: let's say, there are documents like below
{id:a,num:1},{id:b,num:2},{id:c,num:2},{id:d,num:2},{id:e,num:3},{id:f,num:4}
Then if I query for the first set with orderBy('num','desc') and limit
as 2, I'll get [a,b]. Now, if I query with the startAfter last
document as 'b' and limit as 2, then it will skip all the documents
after b, which have num same as that of b (i.e 2). So, it'll skip doc
[c,d] and return me [e,f]. This works fine on jsbin, but not on react
native using react-native-firebase library.
Please find the exact output for the code here on react-native below:
"Got all results"
"wbXwyLJheRfYXXWlY46j => {\"index\":2,\"number\":2}"
"kGC5cYPN1nKnZCcAb9oQ => {\"index\":6,\"number\":2}"
"8Ek8iWCDQPPJ5s2n8PiQ => {\"index\":4,\"number\":2}"
"mr7MdAygvuheF6AUtWma => {\"index\":1,\"number\":1}"
"RCO5SvNn4fdoE49OKrIV => {\"index\":3,\"number\":1}"
"CvVG7VP1hXTtcfdUaeNl => {\"index\":5,\"number\":1}"
"Got first page"
"wbXwyLJheRfYXXWlY46j => {\"index\":2,\"number\":2}"
"kGC5cYPN1nKnZCcAb9oQ => {\"index\":6,\"number\":2}"
"Got second page"
"mr7MdAygvuheF6AUtWma => {\"index\":1,\"number\":1}"
"RCO5SvNn4fdoE49OKrIV => {\"index\":3,\"number\":1}"
Note that it has skipped : "8Ek8iWCDQPPJ5s2n8PiQ => {\"index\":4,\"number\":2}"
Encountered one more issue in react-native-firebase library:
If there is a map structure in the documents like below:
{
subscribedUsers:{
+1656666:true,
+1657878:false,
+1677676:true
}
}
Now if I fire a query like below:
db.collection("ownerUsers").where(subscribedUsers.${phonenumber} ,
'==' , true).limit(3).get();
The query returns 3 results.Now, if I try to get the next 3 results using startAfter clause like below, it works fine with firebase, but doesn't work with react-native-firebase library:
db.collection("ownerUsers").where(subscribedUsers.${phonenumber} , '==' , true).startAfter(last).limit(3).get()
then I get the below error:
[Info] 11-10 21:38:42.631 21908 21967 I ReactNativeJS: 'Error getting documents', { [Error: Firestore: Client specified an invalid argument. (firestore/invalid-argument).]
This is a bug in the react-native-firebase library. Check out the issue on Github to follow along. The issue is now marked as fixed, and this fix will be included in the next release (>5.5.6).
Old answer below...
The document you pass into startAfter needs to contain all the Firestore needs to determine where to start next. If the fields you filter on are not unique in the collection, the DocumentSnapshot will also need to contain the ID of the document. If your query is showing the first or second document matching the criteria, instead of the one after the one you're looking for, it's most likely because the ID is missing from lastDocument.
After your update, I tried reproducing the problem with this code:
var db = firebase.firestore();
var ref = db.collection("58783480").orderBy('number', 'desc');
ref.get().then(function(snapshot) {
console.log("Got all results");
snapshot.forEach(function(doc) {
console.log(doc.id+' => '+JSON.stringify(doc.data()));
});
var last;
ref.limit(2).get().then(function(snapshot) {
console.log("Got first page");
snapshot.forEach(function(doc) {
console.log(doc.id+' => '+JSON.stringify(doc.data()));
last = doc;
});
ref.startAfter(last).limit(2).get().then(function(snapshot) {
console.log("Got second page");
snapshot.forEach(function(doc) {
console.log(doc.id+' => '+JSON.stringify(doc.data()));
last = doc;
});
});
})
});
live example here
But the output matches exactly with what I'd expect:
Got all results
wbXwyLJheRfYXXWlY46j => {"index":2,"number":2}
kGC5cYPN1nKnZCcAb9oQ => {"index":6,"number":2}
8Ek8iWCDQPPJ5s2n8PiQ => {"index":4,"number":2}
mr7MdAygvuheF6AUtWma => {"index":1,"number":1}
RCO5SvNn4fdoE49OKrIV => {"index":3,"number":1}
CvVG7VP1hXTtcfdUaeNl => {"index":5,"number":1}
Got first page
wbXwyLJheRfYXXWlY46j => {"index":2,"number":2}
kGC5cYPN1nKnZCcAb9oQ => {"index":6,"number":2}
Got second page
8Ek8iWCDQPPJ5s2n8PiQ => {"index":4,"number":2}
mr7MdAygvuheF6AUtWma => {"index":1,"number":1}
I thought I read that you can query subcollections with the new Firebase Firestore, but I don't see any examples. For example I have my Firestore setup in the following way:
Dances [collection]
danceName
Songs [collection]
songName
How would I be able to query "Find all dances where songName == 'X'"
Update 2019-05-07
Today we released collection group queries, and these allow you to query across subcollections.
So, for example in the web SDK:
db.collectionGroup('Songs')
.where('songName', '==', 'X')
.get()
This would match documents in any collection where the last part of the collection path is 'Songs'.
Your original question was about finding dances where songName == 'X', and this still isn't possible directly, however, for each Song that matched you can load its parent.
Original answer
This is a feature which does not yet exist. It's called a "collection group query" and would allow you query all songs regardless of which dance contained them. This is something we intend to support but don't have a concrete timeline on when it's coming.
The alternative structure at this point is to make songs a top-level collection and make which dance the song is a part of a property of the song.
UPDATE
Now Firestore supports array-contains
Having these documents
{danceName: 'Danca name 1', songName: ['Title1','Title2']}
{danceName: 'Danca name 2', songName: ['Title3']}
do it this way
collection("Dances")
.where("songName", "array-contains", "Title1")
.get()...
#Nelson.b.austin Since firestore does not have that yet, I suggest you to have a flat structure, meaning:
Dances = {
danceName: 'Dance name 1',
songName_Title1: true,
songName_Title2: true,
songName_Title3: false
}
Having it in that way, you can get it done:
var songTitle = 'Title1';
var dances = db.collection("Dances");
var query = dances.where("songName_"+songTitle, "==", true);
I hope this helps.
UPDATE 2019
Firestore have released Collection Group Queries. See Gil's answer above or the official Collection Group Query Documentation
Previous Answer
As stated by Gil Gilbert, it seems as if collection group queries is currently in the works. In the mean time it is probably better to use root level collections and just link between these collection using the document UID's.
For those who don't already know, Jeff Delaney has some incredible guides and resources for anyone working with Firebase (and Angular) on AngularFirebase.
Firestore NoSQL Relational Data Modeling - Here he breaks down the basics of NoSQL and Firestore DB structuring
Advanced Data Modeling With Firestore by Example - These are more advanced techniques to keep in the back of your mind. A great read for those wanting to take their Firestore skills to the next level
What if you store songs as an object instead of as a collection? Each dance as, with songs as a field: type Object (not a collection)
{
danceName: "My Dance",
songs: {
"aNameOfASong": true,
"aNameOfAnotherSong": true,
}
}
then you could query for all dances with aNameOfASong:
db.collection('Dances')
.where('songs.aNameOfASong', '==', true)
.get()
.then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
console.log(doc.id, " => ", doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
NEW UPDATE July 8, 2019:
db.collectionGroup('Songs')
.where('songName', isEqualTo:'X')
.get()
I have found a solution.
Please check this.
var museums = Firestore.instance.collectionGroup('Songs').where('songName', isEqualTo: "X");
museums.getDocuments().then((querySnapshot) {
setState(() {
songCounts= querySnapshot.documents.length.toString();
});
});
And then you can see Data, Rules, Indexes, Usage tabs in your cloud firestore from console.firebase.google.com.
Finally, you should set indexes in the indexes tab.
Fill in collection ID and some field value here.
Then Select the collection group option.
Enjoy it. Thanks
You can always search like this:-
this.key$ = new BehaviorSubject(null);
return this.key$.switchMap(key =>
this.angFirestore
.collection("dances").doc("danceName").collections("songs", ref =>
ref
.where("songName", "==", X)
)
.snapshotChanges()
.map(actions => {
if (actions.toString()) {
return actions.map(a => {
const data = a.payload.doc.data() as Dance;
const id = a.payload.doc.id;
return { id, ...data };
});
} else {
return false;
}
})
);
Query limitations
Cloud Firestore does not support the following types of queries:
Queries with range filters on different fields.
Single queries across multiple collections or subcollections. Each query runs against a single collection of documents. For more
information about how your data structure affects your queries, see
Choose a Data Structure.
Logical OR queries. In this case, you should create a separate query for each OR condition and merge the query results in your app.
Queries with a != clause. In this case, you should split the query into a greater-than query and a less-than query. For example, although
the query clause where("age", "!=", "30") is not supported, you can
get the same result set by combining two queries, one with the clause
where("age", "<", "30") and one with the clause where("age", ">", 30).
I'm working with Observables here and the AngularFire wrapper but here's how I managed to do that.
It's kind of crazy, I'm still learning about observables and I possibly overdid it. But it was a nice exercise.
Some explanation (not an RxJS expert):
songId$ is an observable that will emit ids
dance$ is an observable that reads that id and then gets only the first value.
it then queries the collectionGroup of all songs to find all instances of it.
Based on the instances it traverses to the parent Dances and get their ids.
Now that we have all the Dance ids we need to query them to get their data. But I wanted it to perform well so instead of querying one by one I batch them in buckets of 10 (the maximum angular will take for an in query.
We end up with N buckets and need to do N queries on firestore to get their values.
once we do the queries on firestore we still need to actually parse the data from that.
and finally we can merge all the query results to get a single array with all the Dances in it.
type Song = {id: string, name: string};
type Dance = {id: string, name: string, songs: Song[]};
const songId$: Observable<Song> = new Observable();
const dance$ = songId$.pipe(
take(1), // Only take 1 song name
switchMap( v =>
// Query across collectionGroup to get all instances.
this.db.collectionGroup('songs', ref =>
ref.where('id', '==', v.id)).get()
),
switchMap( v => {
// map the Song to the parent Dance, return the Dance ids
const obs: string[] = [];
v.docs.forEach(docRef => {
// We invoke parent twice to go from doc->collection->doc
obs.push(docRef.ref.parent.parent.id);
});
// Because we return an array here this one emit becomes N
return obs;
}),
// Firebase IN support up to 10 values so we partition the data to query the Dances
bufferCount(10),
mergeMap( v => { // query every partition in parallel
return this.db.collection('dances', ref => {
return ref.where( firebase.firestore.FieldPath.documentId(), 'in', v);
}).get();
}),
switchMap( v => {
// Almost there now just need to extract the data from the QuerySnapshots
const obs: Dance[] = [];
v.docs.forEach(docRef => {
obs.push({
...docRef.data(),
id: docRef.id
} as Dance);
});
return of(obs);
}),
// And finally we reduce the docs fetched into a single array.
reduce((acc, value) => acc.concat(value), []),
);
const parentDances = await dance$.toPromise();
I copy pasted my code and changed the variable names to yours, not sure if there are any errors, but it worked fine for me. Let me know if you find any errors or can suggest a better way to test it with maybe some mock firestore.
var songs = []
db.collection('Dances')
.where('songs.aNameOfASong', '==', true)
.get()
.then(function(querySnapshot) {
var songLength = querySnapshot.size
var i=0;
querySnapshot.forEach(function(doc) {
songs.push(doc.data())
i ++;
if(songLength===i){
console.log(songs
}
console.log(doc.id, " => ", doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
It could be better to use a flat data structure.
The docs specify the pros and cons of different data structures on this page.
Specifically about the limitations of structures with sub-collections:
You can't easily delete subcollections, or perform compound queries across subcollections.
Contrasted with the purported advantages of a flat data structure:
Root-level collections offer the most flexibility and scalability, along with powerful querying within each collection.
I'm trying to dynamically generate queries.
I have a big object of things like location and price data. But I get that data from the request. How can I dynamically make use of that data if every query thing is a chained function?
Ideally I'd like to convert something like this...
const wheres = [
{ key: 'price', operator: '>=', value: '1000' },
{ key: 'price', operator: '<=', value: '2000' }
]
...to...
admin
.firestore()
.collection(`rentals`)
.where(`price`, `>=`, `1000`)
.where(`price`, `<=`, `2000`)
You don't have to chain everything directly with each other. The builder pattern used to build the query returns an instance of Query with each call to where() (and other filtering methods). The code you wrote is equivalent to this:
const collection = admin.firestore().collection('rentals')
var query = collection.where('price', '>=', '1000')
query = query.where('price', '<=', '2000')
You can keep working with query as much as you want like this. So, you should be able to keep appending more constraints to it in a loop or whatever suits your requirements.
I thought I read that you can query subcollections with the new Firebase Firestore, but I don't see any examples. For example I have my Firestore setup in the following way:
Dances [collection]
danceName
Songs [collection]
songName
How would I be able to query "Find all dances where songName == 'X'"
Update 2019-05-07
Today we released collection group queries, and these allow you to query across subcollections.
So, for example in the web SDK:
db.collectionGroup('Songs')
.where('songName', '==', 'X')
.get()
This would match documents in any collection where the last part of the collection path is 'Songs'.
Your original question was about finding dances where songName == 'X', and this still isn't possible directly, however, for each Song that matched you can load its parent.
Original answer
This is a feature which does not yet exist. It's called a "collection group query" and would allow you query all songs regardless of which dance contained them. This is something we intend to support but don't have a concrete timeline on when it's coming.
The alternative structure at this point is to make songs a top-level collection and make which dance the song is a part of a property of the song.
UPDATE
Now Firestore supports array-contains
Having these documents
{danceName: 'Danca name 1', songName: ['Title1','Title2']}
{danceName: 'Danca name 2', songName: ['Title3']}
do it this way
collection("Dances")
.where("songName", "array-contains", "Title1")
.get()...
#Nelson.b.austin Since firestore does not have that yet, I suggest you to have a flat structure, meaning:
Dances = {
danceName: 'Dance name 1',
songName_Title1: true,
songName_Title2: true,
songName_Title3: false
}
Having it in that way, you can get it done:
var songTitle = 'Title1';
var dances = db.collection("Dances");
var query = dances.where("songName_"+songTitle, "==", true);
I hope this helps.
UPDATE 2019
Firestore have released Collection Group Queries. See Gil's answer above or the official Collection Group Query Documentation
Previous Answer
As stated by Gil Gilbert, it seems as if collection group queries is currently in the works. In the mean time it is probably better to use root level collections and just link between these collection using the document UID's.
For those who don't already know, Jeff Delaney has some incredible guides and resources for anyone working with Firebase (and Angular) on AngularFirebase.
Firestore NoSQL Relational Data Modeling - Here he breaks down the basics of NoSQL and Firestore DB structuring
Advanced Data Modeling With Firestore by Example - These are more advanced techniques to keep in the back of your mind. A great read for those wanting to take their Firestore skills to the next level
What if you store songs as an object instead of as a collection? Each dance as, with songs as a field: type Object (not a collection)
{
danceName: "My Dance",
songs: {
"aNameOfASong": true,
"aNameOfAnotherSong": true,
}
}
then you could query for all dances with aNameOfASong:
db.collection('Dances')
.where('songs.aNameOfASong', '==', true)
.get()
.then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
console.log(doc.id, " => ", doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
NEW UPDATE July 8, 2019:
db.collectionGroup('Songs')
.where('songName', isEqualTo:'X')
.get()
I have found a solution.
Please check this.
var museums = Firestore.instance.collectionGroup('Songs').where('songName', isEqualTo: "X");
museums.getDocuments().then((querySnapshot) {
setState(() {
songCounts= querySnapshot.documents.length.toString();
});
});
And then you can see Data, Rules, Indexes, Usage tabs in your cloud firestore from console.firebase.google.com.
Finally, you should set indexes in the indexes tab.
Fill in collection ID and some field value here.
Then Select the collection group option.
Enjoy it. Thanks
You can always search like this:-
this.key$ = new BehaviorSubject(null);
return this.key$.switchMap(key =>
this.angFirestore
.collection("dances").doc("danceName").collections("songs", ref =>
ref
.where("songName", "==", X)
)
.snapshotChanges()
.map(actions => {
if (actions.toString()) {
return actions.map(a => {
const data = a.payload.doc.data() as Dance;
const id = a.payload.doc.id;
return { id, ...data };
});
} else {
return false;
}
})
);
Query limitations
Cloud Firestore does not support the following types of queries:
Queries with range filters on different fields.
Single queries across multiple collections or subcollections. Each query runs against a single collection of documents. For more
information about how your data structure affects your queries, see
Choose a Data Structure.
Logical OR queries. In this case, you should create a separate query for each OR condition and merge the query results in your app.
Queries with a != clause. In this case, you should split the query into a greater-than query and a less-than query. For example, although
the query clause where("age", "!=", "30") is not supported, you can
get the same result set by combining two queries, one with the clause
where("age", "<", "30") and one with the clause where("age", ">", 30).
I'm working with Observables here and the AngularFire wrapper but here's how I managed to do that.
It's kind of crazy, I'm still learning about observables and I possibly overdid it. But it was a nice exercise.
Some explanation (not an RxJS expert):
songId$ is an observable that will emit ids
dance$ is an observable that reads that id and then gets only the first value.
it then queries the collectionGroup of all songs to find all instances of it.
Based on the instances it traverses to the parent Dances and get their ids.
Now that we have all the Dance ids we need to query them to get their data. But I wanted it to perform well so instead of querying one by one I batch them in buckets of 10 (the maximum angular will take for an in query.
We end up with N buckets and need to do N queries on firestore to get their values.
once we do the queries on firestore we still need to actually parse the data from that.
and finally we can merge all the query results to get a single array with all the Dances in it.
type Song = {id: string, name: string};
type Dance = {id: string, name: string, songs: Song[]};
const songId$: Observable<Song> = new Observable();
const dance$ = songId$.pipe(
take(1), // Only take 1 song name
switchMap( v =>
// Query across collectionGroup to get all instances.
this.db.collectionGroup('songs', ref =>
ref.where('id', '==', v.id)).get()
),
switchMap( v => {
// map the Song to the parent Dance, return the Dance ids
const obs: string[] = [];
v.docs.forEach(docRef => {
// We invoke parent twice to go from doc->collection->doc
obs.push(docRef.ref.parent.parent.id);
});
// Because we return an array here this one emit becomes N
return obs;
}),
// Firebase IN support up to 10 values so we partition the data to query the Dances
bufferCount(10),
mergeMap( v => { // query every partition in parallel
return this.db.collection('dances', ref => {
return ref.where( firebase.firestore.FieldPath.documentId(), 'in', v);
}).get();
}),
switchMap( v => {
// Almost there now just need to extract the data from the QuerySnapshots
const obs: Dance[] = [];
v.docs.forEach(docRef => {
obs.push({
...docRef.data(),
id: docRef.id
} as Dance);
});
return of(obs);
}),
// And finally we reduce the docs fetched into a single array.
reduce((acc, value) => acc.concat(value), []),
);
const parentDances = await dance$.toPromise();
I copy pasted my code and changed the variable names to yours, not sure if there are any errors, but it worked fine for me. Let me know if you find any errors or can suggest a better way to test it with maybe some mock firestore.
var songs = []
db.collection('Dances')
.where('songs.aNameOfASong', '==', true)
.get()
.then(function(querySnapshot) {
var songLength = querySnapshot.size
var i=0;
querySnapshot.forEach(function(doc) {
songs.push(doc.data())
i ++;
if(songLength===i){
console.log(songs
}
console.log(doc.id, " => ", doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
It could be better to use a flat data structure.
The docs specify the pros and cons of different data structures on this page.
Specifically about the limitations of structures with sub-collections:
You can't easily delete subcollections, or perform compound queries across subcollections.
Contrasted with the purported advantages of a flat data structure:
Root-level collections offer the most flexibility and scalability, along with powerful querying within each collection.