I have a query for firebase that has an array of IDs that has a size > 10. Firebase has restrictions on the number of records to query in one session. Is there a way to query against more than 10 at a time?
[Unhandled promise rejection: FirebaseError: Invalid Query. 'in' filters support a maximum of 10 elements in the value array.]
https://cloud.google.com/firestore/docs/query-data/queries
let query = config.db
.collection(USER_COLLECTION_NAME)
.where("id", "in", matchesIdArray);
const users = await query.get();
(matchesIdArray.length needs to be unlimited)
I found this to work well for me without needing to make as many queries (loop and request in batches of 10).
export async function getContentById(ids, path) {
// don't run if there aren't any ids or a path for the collection
if (!ids || !ids.length || !path) return [];
const collectionPath = db.collection(path);
const batches = [];
while (ids.length) {
// firestore limits batches to 10
const batch = ids.splice(0, 10);
// add the batch request to to a queue
batches.push(
collectionPath
.where(
firebase.firestore.FieldPath.documentId(),
'in',
[...batch]
)
.get()
.then(results => results.docs.map(result => ({ /* id: result.id, */ ...result.data() }) ))
)
}
// after all of the data is fetched, return it
return Promise.all(batches)
.then(content => content.flat());
}
BEST METHOD
convert the list into a list that contains sublists of 10 items each.
then for loop through that second list and query through firebase in each loop.
EXAMPLE:
List<String> phoneNumbers = ['+12313','+2323','1323','32323','32323','3232', '1232']; //CAN BE UPTO 100 or more
CONVERT THE PHONE NUMBERS TO A LIST OF SUBLIST OF 10 ITEMS EACH
List<List<String>> subList = [];
for (var i = 0; i < phoneNumbers.length; i += 10) {
subList.add(
phoneNumbers.sublist(i, i + 10> phoneNumbers.length ? phoneNumbers.length : i + 10));
}
NOW RUN FIREBASE QUERY
subList.forEach((element) {
firestore
.collection('Stories')
.where('userPhone', whereIn: element)
.get()
.then((value) {
value.docs.forEach((snapshot) {
//handle the list
});
});
Your only workaround is to make one query for each item in the array that you would normally use with a single "in" query. Or, batch the requests in the array.
let query = config.db
.collection(USER_COLLECTION_NAME)
.where("id", "==", matchesIdArray[0]);
const users = await query.get();
You'd have to use the above code in a loop over the matchesIdArray array, and merge the results after they are all done.
One common way to work around this limitation is to retrieve the items in batches, and then either process the results from each query either sequentially or in parallel.
Another common workaround is to model your data in a way that doesn't require for you to read dozens of documents to handle an individual request from your user. It's hard to say how you could reduce that number, but it often involves duplicating the data that you need from those separate documents into a single aggregated document.
An example of this: if you have a news site and need to show the latest 10 article headlines for each of 5 categories, you can do:
Do 50 separate reads, one for each document.
Create a document with the headlines for the latest 10 articles, and then only need to read 5 documents (one for each category).
Create a document with the latest 10 headlines for all 5 categories, and then only need to read that one document.
In these last two scenarios you're making the code that writes to the database more complex, as it needs to now write the aggregated documents too. But in return you have much less data to read, which reduces the cost, and improves the performance of your app. This type trade-off is very common when using NoSQL databases, which tend to be used in scenarios that have massively more reads than writes of their data.
For more data modeling advice, I recommend:
NoSQL data modeling
Getting to know Cloud Firestore
I faced the same problem and my solution using typescript was :
for complete observables i used rxjs forkJoin
getPagesByIds(ids: string[]): Observable<Page[]> {
ids = [...ids];
if (ids.length) {
let observables: Observable<Page[]>[] = [];
while (ids.length) {
let observable = this.afs.collection<Page>(PAGE, ref => ref.where('id', 'in', ids.splice(0, 10))).get().pipe(map(pages => pages.docs.map(page => page.data())))
observables.push(observable)
}
return combineLatest(observables).pipe(map(pages => pages.flat(1)))
}
return of ([])
}
for uncomplete observables i used rxjs combineLatest
getPagesByIds(ids: string[]): Observable<Page[]> {
ids = [...ids];
if (ids.length) {
let observables: Observable<Page[]>[] = [];
while (ids.length) {
let observable = this.afs.collection<Page>(PAGE, ref => ref.where('id', 'in', ids.splice(0, 10))).get().pipe(map(pages => pages.docs.map(page => page.data())))
observables.push(observable)
}
return combineLatest(observables).pipe(map(pages => pages.flat(1)))
}
return of ([])
}
A very common scenario to need more than 10 elements to match an 'in' query is when displaying group chat conversations to all users involved and no one else. So while you may be limited with 10 "members" that can match with that chat ID, why not just reverse the logic and use the "array-contains" query since it is not limited.
Your pseudocode: create an array on every chat document called "usersInvolved" listing the IDs of every user involved and then compare that against the current user's ID.
db
.collection('chats')
.where('usersInvolved', 'array-contains', myUserID)
.onSnapshot(...)
That will display all the chat conversations for all users in that group conversation, without the limitation of 10 users.
As per the firebase documentation it support up to 10 ids only in the where field, For querying more than 10 elements either we need to query each document individually or split array to chunks of 10 ids.
For querying each items individually. Check the code below,
let usersPromise = [];
usersIds.map((id) => {
usersPromise.push(firestore.collection("users").doc(id).get());
});
Promise.all(usersPromise).then((docs) => {
const users = docs.map((doc) => doc.data());
// do your operations with users list
});
async getInBatch(query, key, arr) {
const promises = new Array(Math.ceil(arr.length / 10))
.fill('')
.map((_, i) => arr.slice(i * 10, (i + 1) * 10))
.map((i) => query.where(key, 'in', i).get());
const batches = await Promise.all(promises);
const docsData = [];
batches.forEach((snapshot) => {
snapshot.forEach((doc) => {
docsData.push([doc.id, doc.data()]);
});
});
return docsData;
}
const data = await this.getInBatch(db.collection('collection'),
'id',
[1,2,3]
);
For javascript, this was my implementation with promises, using a Promise.all
The first part of the code groups an array into in clauses with sets of 10, and then executes each set of 10 in the Promise.all. The resulting array is returned as a tuple with the docid in the first, and document data in the second part.
Adding my own solution here as I was not totally satisfied with the current answers:
docsSnap$(queryFn?: QueryFn): Observable<T[]> {
return this.firestore.collection<T>(`${this.basePath}`, queryFn)
.get()
.pipe(map((i: QuerySnapshot<T>) => i.docs
.map((d: QueryDocumentSnapshot<T>) => d.data())));
}
getIdChuncks(allIds: Array<string>): Array<Array<string>> {
const idBatches = [];
while (allIds.length > 0)
idBatches.push(allIds.splice(0, 10));
return idBatches;
}
processBatches(allIds: string[]) {
const batches$ = this.getIdChuncks(allIds)
.map(ids => this
.docsSnap$(ref => ref.where('id', 'in', ids)));
this.items$ = forkJoin(batches$)
.pipe(map(arr => arr.flat()));
}
Here is the Dart Implementation of Conrad Davis answer for Flutter to fix firestore Query "IN" (whereIn/whereNotIn) limited to 10
Future<List<QueryDocumentSnapshot<Map<String, dynamic>>>> getContentById(
{required List<Object?> ids,
required String path,
required String field,
bool whereIn = false}) {
var collectionPath = store.collection(path);
var batches = <Future<List<QueryDocumentSnapshot<Map<String, dynamic>>>>>[];
var batch = ids;
while (ids.length > 0) {
// firestore limits batches to 10
var end = 10;
if (ids.length < 10) end = ids.length;
batch = ids.sublist(0, end);
ids.removeWhere((element) => batch.contains(element));
if (whereIn) {
// add the batch request to to a queue for whereIn
batches.add(collectionPath
.where(field, whereIn: [...batch])
.get()
.then((results) => results.docs));
} else {
// add the batch request to to a queue for whereNotIn
batches.add(collectionPath
.where(field, whereNotIn: [...batch])
.get()
.then((results) => results.docs));
}
}
// after all of the data is fetched, return it
return Future.wait(batches)
.then((content) => content.expand((i) => i).toList());
}
Use like this:
getContentById(ids:["John", "Doe", "Mary"], path: "contacts", field: 'name', whereIn:true)
.then((value) async {
for (var doc in value) {
var d = doc.data();
//... Use your data Map<String, dynamic>
}
});
My problem is that I use wrong query to get the date.
const SaveDateBase = async ( e) => {
e.preventDefault()
await setDoc(doc(db, "Users", "Pompy", "Pompy", user.uid), {
displayName: user.displayName,
uid: user?.uid,
modulyPV}).then(()=>{
console.log("moduly", modulyPV)
})
};
useEffect(() => {
const getUsers = async (users) => {
const URC = query(collection(db, "Users").document("Pompy").collection("Pompy"), where("uid", "==", user?.uid));
const data = await getDocs(URC)
setModulyPV(data.docs.map((doc) => ({...doc.data(), id: doc.id})))
}
getUsers();
},[])
The date are saved in date base, and I can successfully update/delete them, but I do something wrong to fetch (read?) them.
I guess is problem with the code.
You can get the data in diff ways, first "Pompy" seems to be your document where you are storing a nested collection then you document "Pompy" So for retrieve that specific document should be something like:
let snapshot = await db
.collection('Users')
.doc('Pompy')
.collection('Pompy')
.get()
snapshot.forEach(doc =>{
console.log('data:', doc.data())
})
Then to query into the nested collection would be something like querying the nested collections.
https://cloud.google.com/firestore/docs/samples/firestore-data-get-sub-collections?hl=es-419#firestore_data_get_sub_collections-nodejs
You can also use collection groups.
https://firebase.google.com/docs/firestore/query-data/queries#collection-group-query
const pompys = query(collectionGroup(db, 'Pompy'), where("uid", "==", user?.uid));
I have create document like this in react native, I am using rnfirebase library
firestore()
.collection('WaterCanData')
.doc(EntryDate)
.collection('Entries')
.doc(values.customerName)
.set({
CustomerName: values.customerName,
CansOut: values.cansOut,
JarsOut: values.jarsOut,
EmptyCansIn: values.emptyCansIn,
JarsIn: values.jarsIn,
Bottles: values.bottles,
Ice: values.ice
})
.then(() => {
console.log('Entry added!!!!!!!!!');
})
When I try to retrieve EntryDate from WaterCanData Coellection I am not able to fetch it(Document name appears in italic font), So how should I retrive this document which contains a subcollection, Below I have attached my ss of data structure
Data structure
Data structuree
The reason your document appears in italics is because it doesn't currently exist. In Cloud Firestore, subcollections can exist without requiring their parent document to also exist.
Non-existant documents will not appear in queries or snapshots in the client SDKs as stated in the Firebase Console.
This document does not exist, it will not appear in queries or snapshots
If you want to be able to get your entry dates, you need to create the document (which can be empty).
firebase.firestore()
.collection('WaterCanData')
.doc(EntryDate)
.set({}); // an empty document
To create the document at the same time as an entry on it's subcollection, you can use a batched write like so:
const db = firebase.firestore();
const batch = db.batch();
// get references to the relevant locations
const entryDateRef = db
.collection('WaterCanData')
.doc(EntryDate);
const customerRef = entryDateRef
.collection('Entries')
.doc(values.customerName);
// queue the data to write
batch.set(entryDateRef, {});
batch.set(customerRef, {
CustomerName: values.customerName,
CansOut: values.cansOut,
JarsOut: values.jarsOut,
EmptyCansIn: values.emptyCansIn,
JarsIn: values.jarsIn,
Bottles: values.bottles,
Ice: values.ice
})
// make changes to database
batch.commit()
.then(() => {
console.log('Entry added!!!!!!!!!');
});
This will then allow you to list all of the entry dates in your database using something like:
firebase.firestore().collection('WaterCanData')
.get()
.then((querySnapshot) => {
querySnapshot.forEach(doc => {
const entryDate = doc.id;
// const customerEntriesRef = doc.ref.collection('Entries');
console.log('Entry date found: ' + entryDate);
}
});
If (as an example) you wanted to also find how many entries were linked to a given date, you would need to also query each subcollection (here the code gets a little more confusing).
firebase.firestore().collection('WaterCanData')
.get()
.then((querySnapshot) => {
const fetchSizePromises = [];
// for each entry date, get the size of it's "Entries" subcollection
querySnapshot.forEach(doc => {
const entryDate = doc.id;
const customerEntriesRef = doc.ref.collection('Entries');
// if this get() fails, just store the error rather than throw it.
const thisEntrySizePromise = customerEntriesRef.get()
.then(
(entriesQuerySnapshot) => {
return { date: entryDate, size: entriesQuerySnapshot.size }
},
(error) => {
return { date: entryDate, size: -1, error }
}
);
// add this promise to the queue
fetchSizePromises.push(thisEntrySizePromise)
}
// wait for all fetch operations and return their results
return Promise.all(fetchSizePromises);
})
.then((entryInfoResults) => {
// for each entry, log the result
entryInfoResults.forEach((entryInfo) => {
if (entryInfo.error) {
// this entry failed
console.log(`${entryInfo.date} has an unknown number of customers in its Entries subcollection due to an error`, entryInfo.error);
} else {
// got size successfully
console.log(`${entryInfo.date} has ${entryInfo.size} customers in its Entries subcollection`);
}
}
});
Using below code you can console every document id inside waterCanData collection. In your database you have only one document, then it will console your document id. (10042021)
firestore()
.collection('WaterCanData')
.get()
.then((querySnapshot) => {
querySnapshot.forEach((doc) => {
console.log(doc.id)
});
})
So, I don't really know how to write JS, I am developing a mobile app in Flutter, and I would be grateful for some help and clarifications regarding Future/Promises in JS.
I got a collection of posts for each user, and I want to create an .onCreate function which when a user posts a new post (a new document is created inside the 'posts/userId/user_posts' collection), then it gets all the user's followers (from a collection 'user_followers/userUid') and for each follower, it writes the postUid and postOwnerUid to that follower's newsFeed collection ('user_news_feed/followerId').
This is what I got right now, but I am walking blind, since I really don't know JS and I don't know how can I await a write function while inside a get function.
And how do I prevent Cloud Timeouts? If for instance the user has 1000 followers, how can I prevent Firebase from shutting down my function and making sure all the followers are notified?
exports.writeToUserNewsFeed = functions.firestore
.document('posts/{userId}/user_posts/{postId}')
.onCreate((snap, context) => {
const postData = snap.data();
const postUid = postData['post_uid'];
const userUid = postData['user_uid'];
const postCreationDate = postData['post_creation_date'];
var docRef = db.collection('user_followers').doc(userUid).collection('followers');
docRef.get().then((querySnapshot) => {
querySnapshot.forEach((doc) => {
db.collection('user_news_feed')
.doc(doc.data['uid'])
.collection('feed')
.document(postUid)
.set({
'post_uid': postUid,
'user_uid': userUid,
'post_uid': postCreationDate,
});
});
});
});
As explained in the doc, in a background Cloud Function like an onCreate() for Firestore, you need to return a Promise when all the asynchronous work is completed. So in your case, one possibility is to use Promise.all() because you don't know upfront how many documents are in the followers subcollection. Since Promise.all() returns a single Promise you can include it in the Promise chain that you need to return in the Cloud Function.
exports.writeToUserNewsFeed = functions.firestore
.document('posts/{userId}/user_posts/{postId}')
.onCreate((snap, context) => {
const postData = snap.data();
const postUid = postData['post_uid'];
const userUid = postData['user_uid'];
const postCreationDate = postData['post_creation_date'];
var followersColRef = db.collection('user_followers').doc(userUid).collection('followers');
return followersColRef.get().then((querySnapshot) => { // <= See return here
const promises = [];
querySnapshot.forEach((doc) => {
promises.push(
db.collection('user_news_feed')
.doc(doc.data['uid'])
.collection('feed')
.doc(postUid)
.set({
'post_uid': postUid,
'user_uid': userUid,
'post_uid': postCreationDate,
})
);
});
return Promise.all(promises); // <= See return here
})
.catch(error => {
console.log(error);
return null;
})
});
Note that instead of using Promise.all() you could also use a batched write but there is a limit of 500 operations for a batched write.
I am trying to update several Firestore documents, based on the result of a third-party service inside a transaction. Problem is, I am getting the following error:
Error: Cannot modify a WriteBatch that has been committed.
Here is my code:
export default async function debitDueTransactions(context: any) {
const now = new Date().getTime();
return db.runTransaction(async (transaction: FirebaseFirestore.Transaction) => {
const chargesToCaptureRef = db.collection(`charges_to_capture`)
.where('dateToCapture', '>=', now)
.where('dateToCapture', '<=', (now + 86400000))
.where('captureResult', '==', null);
return transaction.get(chargesToCaptureRef).then((chargeToCaptureQuerySnap: FirebaseFirestore.QuerySnapshot) => {
chargeToCaptureQuerySnap.forEach(async (doc: FirebaseFirestore.QueryDocumentSnapshot) => {
const chargeToCapture = <ChargeToCapture>doc.data();
chargeToCapture.id = doc.id;
let errorKey = null;
// Calling third party service here, waiting response
const captureResult = await captureCharge(chargeToCapture.chargeId).catch((error: any) => {
errorKey = error.code ? error.code : 'unknown_error';
});
transaction.update(doc.ref, { captureResult: captureResult, errorKey: errorKey });
});
return new Promise((resolve) => { resolve(); });
})
});
}
Can't get what I am doing wrong, any idea ?
As you can see from the API documentation, transaction.get() only accepts a DocumentReference type object. You're passing it a Query object. A Firestore transaction isn't capable of transacting on a Query. If you want to transact on all the documents returned from a Query, you should perform the query before the transaction, then use transaction.get() on each DocumentReference individually.