Firestore: slow query - firebase

I have a query that gets all the messages from my messages collection for a particular chatId.
It then sorts it in descending order on the createdAt timestamp and has a limit of 20 messages applied to that.. My queries take ~2500 ms... which I feel is slow.. I don't even have many messages in my collection so I feel like it could get slower once I have thousands of them..
I'm new to firestore, so perhaps I'm doing something wrong? I appreciate the help!
Here is the code for the query:
exports.getMessages = functions.https.onCall(async (data, context) => {
try {
const startAt = data.startAt;
const increment = 20;
const limit = startAt + increment;
const chatId = data.chatId;
const snapshot = await firestore
.collection('messages')
.where('chatId', '==', chatId)
.orderBy('createdAt', 'desc')
.limit(limit)
.get();
const promises = [];
snapshot.forEach(doc => {
const message = {id: doc.id, ...doc.data()};
const promise = combineUserAndMessage({
message,
userId: message.user._id,
});
promises.push(promise);
});
const messages = await Promise.all(promises);
return {data: messages, startAt: limit};
} catch (err) {
console.log('GET MESSAGES ERROR: ', err);
throw err;
}
});
const combineUserAndMessage = async ({userId, message}) => {
try {
const userSnapshot = await firestore.doc(`users/${userId}`).get();
const user = userSnapshot.data();
const createdAt = new admin.firestore.Timestamp(
message.createdAt._seconds,
message.createdAt._nanoseconds,
).toDate();
return {
...message,
user: {
_id: userId,
name: user.displayName,
avatar: user.profilePicture,
},
createdAt: createdAt.toString(),
};
} catch (err) {
console.log('COMBINE USER AND MESSAGE ERROR: ', err);
throw err;
}
};
EDIT: I took the benchmark from the function logs. I understand that there can be cold start costs included in that benchmark so I'll try to benchmark it using my own logs. That said, if anyone sees something wrong with the query, or knows of a way to make it faster/better, I'd still appreciate it!

Related

Firebase cloud function getting 404 timeout error

I'm trying to run a firebase cloud function. It reads 5k documents and it's taking more than one minute to run in insomnia. When it finally crashes showing a 404 timeout error, how can I fix it? I'm from Brazil and the function region is from the US by default, but I don't believe changing it will help.
I tried making the code cleaner, but didn't help.
This is one of the cloud functions:
exports.getTodosPedidos = functions.https.onRequest(
async (req: RequestType, res: ResponseType) => {
try {
const idProjeto = req.query.idProjeto;
if (!idProjeto) {
onFailureResponseStructure(res, "ID do Projeto não fornecido.", 404);
}
const projeto =
idProjeto &&
(await admin.firestore().collection("projetos").doc(idProjeto.toString()).get());
if (projeto != "" && !projeto?.createTime) {
onFailureResponseStructure(res, "O projeto fornecido não existe.", 405);
}
const snapshot = await admin.firestore().collection("pedidos").get();
const result = snapshot.docs.map(doc => ({ ...doc.data(), id: doc.id }));
let idsDosPedidos = result.map(elemento => elemento.id);
let somaDosArquivos = idsDosPedidos.map(async pedidoId => {
let bucketRef = await admin
.storage()
.bucket()
.getFiles({
prefix: `${idProjeto}/arquivos/${pedidoId}/`,
});
return bucketRef[0].map(elemento => parseInt(elemento.metadata.size));
});
const resultado = await Promise.all(somaDosArquivos);
onSuccessResponseStructure(res, { resultado, idsDosPedidos });
} catch (error: any) {
onFailureResponseStructure(res, error.message, 500);
}
}
);
const onSuccessResponseStructure = (res: ResponseType, data: any) => {
return res.status(200).json({
status: "OK",
data,
messages: [],
statusCode: "200",
});
};
const onFailureResponseStructure = (
res: ResponseType,
messages: string,
statusCode: number
) => {
return res.status(statusCode).json({ status: "ERRO", data: [], messages, statusCode });
};
I'm not entirely familiar with firebase. But other cloud services allow you to increase functions timeout lengths. It appears firebase does as well...https://firebase.google.com/docs/functions/manage-functions#set_timeout_and_memory_allocation
Though for long running tasks you may want to investigate other services.

Firebase listUsers fails to get All users after a certain page

I'm using a pubsub firebase function (cron), and inside this function Im calling firebase auth users, to fill some missing data in a profile collection
Im paginating with the pageToken, the first token passed is undefined then I save it in a config db and read the token to get the next page
The issue is that I have 170K records, and listusers returns an undefined token at the 6th page (6k users) which is frsutrating
here is the code:
functions.pubsub
.schedule('*/30 * * * *')
.onRun(async () => {
const page = firestore.collection('config').doc('pageToken');
const doc = (await page.get()).data();
// Check if last page don't run again
const last = doc?.last;
if (last) return;
// Load page
const pageToken = doc?.pageToken || undefined;
let pageNumber = doc?.pageNumber as number;
return firebaseAdmin
.auth()
.listUsers(1000, pageToken)
.then(async listUsersResult => {
for (const userRecord of listUsersResult.users) {
// Fetch Profile
try {
const profile = await firestore
.collection('profiles')
.doc(userRecord.uid);
// data foramtting here
// compared profile data & fixed data
const payload = JSON.parse(
JSON.stringify({
...profileData,
...{
lastName,
firstName,
language,
...(!userRecord.phoneNumber && {
phone,
}),
},
})
);
// Profile doesn't exist : Create
if (!profileData && payload) {
await profile.create({
...payload,
...{
Migrated: true,
},
});
} else if (profileData && payload) {
const data = compare(profileData, payload);
if (data) {
// Profile exists: Update
await profile.update(data);
if (userRecord.phoneNumber)
await profile.update({
phone: firebaseAdmin.firestore.FieldValue.delete(),
});
}
}
} catch (err) {
functions.logger.error('Some Error', err);
}
}
if (!listUsersResult.pageToken) {
return await firestore
.collection('config')
.doc('pageToken')
.update({
last: true,
});
}
// List next batch of users.
pageNumber++;
return await firestore
.collection('config')
.doc('pageToken')
.update({
pageToken: listUsersResult.pageToken,
pageNumber,
});
});
});
so after in page 6, I have a last:true property added to the firestore however there is 164k data are missing
any idea ?

FCM very slow and unreliable when sending to a group of recipients through Cloud Function

I have the following Function that:
Listens for document (text message) creation
Grab IDs of members of a group chat
Get the FCM Tokens for each member
With a for-loop, send messages to group members
exports.sendChatMessage = functions.firestore
.document("chats/{mealID}/messages/{messageID}")
.onCreate((snap, context) => {
const data = snap.data();
const mealID = context.params.mealID;
const senderID = data.senderID;
const senderName = data.senderName;
const messageContent = data.content;
var docRef = db.collection("chats").doc(mealID);
docRef
.get()
.then((doc) => {
if (doc.exists) {
const docData = doc.data();
const mealName = docData.name;
const userStatus = docData.userStatus;
var users = docData.to;
var eligibleUsers = users.filter(
(user) => userStatus[user] == "accepted"
);
eligibleUsers.push(docData.from);
// get fcmTokens from eligibleUsers and send the messagme
db.collection("users")
.where("uid", "in", eligibleUsers)
.get()
.then((snapshot) => {
var fcmTokens = [];
var thumbnailPicURL = "";
// get thumbnailpic of the sender and collect fcmTokens
snapshot.forEach((doc) => {
if (doc.data().uid == senderID) {
thumbnailPicURL =
doc.data().thumbnailPicURL == null
? "https://i.imgur.com/8wSudUk.png"
: doc.data().thumbnailPicURL;
} else {
fcmTokens.push(doc.data().fcmToken);
}
});
// send the message fcmTokens
fcmTokens.forEach((token) => {
if (token != "") {
const fcmMessage = {
message: {
token: token,
notification: {
title: mealName,
body: senderName + ": " + messageContent,
image: thumbnailPicURL,
},
apns: {
payload: {
aps: {
category: "MESSAGE_RECEIVED",
},
MEAL_ID: mealID,
},
},
},
};
tokenManger.sendFcmMessage(fcmMessage);
}
});
return true;
});
} else {
// doc.data() will be undefined in this case
console.log("No such document!");
return false;
}
})
.catch((error) => {
console.log("Error getting document:", error);
return false;
});
return true;
});
My send function comes from a helper file that uses the HTTP V1 protocol to build the send-request:
const { google } = require("googleapis");
const https = require("https");
const MESSAGING_SCOPE = "https://www.googleapis.com/auth/firebase.messaging";
const SCOPES = [MESSAGING_SCOPE];
const PROJECT_ID = MY_PROJECT_ID;
const HOST = "fcm.googleapis.com";
const PATH = "/v1/projects/" + PROJECT_ID + "/messages:send";
exports.getAccessToken = () => {
return new Promise(function (resolve, reject) {
const key = require("./service-account.json");
var jwtClient = new google.auth.JWT(
key.client_email,
null,
key.private_key,
SCOPES,
null
);
jwtClient.authorize(function (err, tokens) {
if (err) {
reject(err);
return;
}
resolve(tokens.access_token);
});
});
};
//send message
exports.sendFcmMessage = (fcmMessage) => {
this.getAccessToken().then(function (accessToken) {
var options = {
hostname: HOST,
path: PATH,
method: "POST",
headers: {
Authorization: "Bearer " + accessToken,
},
// … plus the body of your notification or data message
};
var request = https.request(options, function (resp) {
resp.setEncoding("utf8");
resp.on("data", function (data) {
console.log("Message sent to Firebase for delivery, response:");
console.log(data);
});
});
request.on("error", function (err) {
console.log("Unable to send message to Firebase");
console.log(err);
});
request.write(JSON.stringify(fcmMessage));
request.end();
});
};
It worked all fine in the emulator but once deployed, there're significant delays (~3 mins):
I also noticed that the console says the cloud function finishes execution BEFORE sendFcmMessage logs success messages.
I did some research online, it appears that it might have something to do with the usage of Promise but I wasn't sure if that's the sole reason or it has something to do with my for-loop.
The Problem
To summarize the issue, you are creating "floating promises" or starting other asynchronous tasks (like in sendFcmMessage) where you aren't returning a promise because they use callbacks instead.
In a deployed function, as soon as the function returns its result or the Promise chain resolves, all further actions should be treated as if they will never be executed as documented here. An "inactive" function might be terminated at any time, is severely throttled and any network calls you make (like setting data in database or calling out to FCM) may never be executed.
An indicator that you haven't properly chained the promises is when you see the function completion log message ("Function execution took...") before other messages you are logging. When you see this, you need to look at the code you are running and confirm whether you have any "floating promises" or are using callback-based APIs. Once you have changed the callback-based APIs to use promises and then made sure they are all chained together properly, you should see a significant boost in performance.
The fixes
Sending the message data to FCM
In your tokenManger file, getAccessToken() could be reworked slightly and sendFcmMessage should be converted to return a Promise:
exports.getAccessToken = () => {
return new Promise(function (resolve, reject) {
const key = require("./service-account.json");
const jwtClient = new google.auth.JWT(
key.client_email,
null,
key.private_key,
SCOPES,
null
);
jwtClient.authorize(
(err, tokens) => err ? reject(err) : resolve(tokens.access_token)
);
});
};
//send message
exports.sendFcmMessage = (fcmMessage) => {
// CHANGED: return the Promise
return this.getAccessToken().then(function (accessToken) {
const options = {
hostname: HOST,
path: PATH,
method: "POST",
headers: {
Authorization: "Bearer " + accessToken,
},
// … plus the body of your notification or data message
};
// CHANGED: convert to Promise:
return new Promise((resolve, reject) => {
const request = https.request(options, (resp) => {
resp.setEncoding("utf8");
resp.on("data", resolve);
resp.on("error", reject);
});
request.on("error", reject);
request.write(JSON.stringify(fcmMessage));
request.end();
});
});
};
However, the above code was built for googleapis ^52.1.0 and google-auth-library ^6.0.3. The modern versions of these modules are v92.0.0 and v7.11.0 respectively. This means you should really update the code to use these later versions like so:
// Import JWT module directly
const { JWT } = require('google-auth-library');
// FIREBASE_CONFIG is a JSON string available in Cloud Functions
const PROJECT_ID = JSON.parse(process.env.FIREBASE_CONFIG).projectId;
const FCM_ENDPOINT = `https://fcm.googleapis.com/v1/projects/${PROJECT_ID}/messages:send`;
const FCM_SCOPES = ["https://www.googleapis.com/auth/firebase.messaging"];
exports.sendFcmMessage = (fcmMessage) => {
const key = require("./service-account.json"); // consider moving outside of function (so it throws an error during deployment if its missing)
const client = new JWT({
email: key.client_email,
key: key.private_key,
scopes: FCM_SCOPES
});
return client.request({ // <-- this uses `gaxios`, Google's fork of `axios` built for Promise-based APIs
url: FCM_ENDPOINT,
method: "POST",
data: fcmMessage
});
}
Better yet, just use the messaging APIs provided by the Firebase Admin SDKs that handle the details for you. Just feed it the message and tokens as needed.
import { initializeApp } from "firebase-admin/app";
import { getMessaging } from "firebase-admin/messaging";
initializeApp(); // initializes using default credentials provided by Cloud Functions
const fcm = getMessaging();
fcm.send(message) // send to one (uses the given token)
fcm.sendAll(messagesArr) // send to many at once (each message uses the given token)
fcm.sendMulticast(message) // send to many at once (uses a `tokens` array instead of `token`)
The Cloud Function
Updating the main Cloud Function, you'd get:
exports.sendChatMessage = functions.firestore
.document("chats/{mealID}/messages/{messageID}")
.onCreate((snap, context) => {
const mealID = context.params.mealID;
const { senderID, senderName, content: messageContent } = snap.data();
const docRef = db.collection("chats").doc(mealID);
/* --> */ return docRef
.get()
.then((doc) => {
if (!doc.exists) { // CHANGED: Fail fast and avoid else statements
console.log(`Could not find "chat:${mealID}"!`);
return false;
}
const { userStatus, to: users, name: mealName, from: fromUser } = doc.data();
const eligibleUsers = users.filter(
(user) => userStatus[user] == "accepted"
);
eligibleUsers.push(fromUser);
// get fcmTokens from eligibleUsers and send the message
/* --> */ return db.collection("users")
.where("uid", "in", eligibleUsers) // WARNING: This will only work for up to 10 users! You'll need to break it up into chunks of 10 if there are more.
.get()
.then(async (snapshot) => {
const fcmTokens = [];
let thumbnailPicURL = "";
// get thumbnailpic of the sender and collect fcmTokens
snapshot.forEach((doc) => {
if (doc.get("uid") == senderID) {
thumbnailPicURL = doc.get("thumbnailPicURL"); // update with given thumbnail pic
} else {
fcmTokens.push(doc.get("fcmToken"));
}
});
const baseMessage = {
notification: {
title: mealName,
body: senderName + ": " + messageContent,
image: thumbnailPicURL || "https://i.imgur.com/8wSudUk.png", // CHANGED: specified fallback image here
},
apns: {
payload: {
aps: {
category: "MESSAGE_RECEIVED",
},
MEAL_ID: mealID,
},
}
}
// log error if fcmTokens empty?
// ----- OPTION 1 -----
// send the message to each fcmToken
const messagePromises = fcmTokens.map((token) => {
if (!token) // handle "" and undefined
return; // skip
/* --> */ return tokenManger
.sendFcmMessage({
message: { ...baseMessage, token }
})
.catch((err) => { // catch the error here, so as many notifications are sent out as possible
console.error(`Failed to send message to "fcm:${token}"`, err);
})
});
await Promise.all(messagePromises); // wait for all messages to be sent out
// --------------------
// ----- OPTION 2 -----
// send the message to each fcmToken
await getMessaging().sendAll(
fcmTokens.map((token) => ({ ...baseMessage, token }))
);
// --------------------
return true;
})
.catch((error) => {
console.log("Error sending messages:", error);
return false;
});
})
.catch((error) => {
console.log("Error getting document:", error);
return false;
});
});
I found out that the culprit is my queries to db. Like #samthecodingman commented, I was creating floating Promises.
Originally, I have codes like:
db.collection("users")
.where("uid", "in", eligibleUsers)
.get()
.then((snapshot) => {...}
All I needed to do is to return that call:
return db.collection("users")
.where("uid", "in", eligibleUsers)
.get()
.then((snapshot) => {...}
Although it's still not instant delivery, it's much faster now.

Function execution took 60002 ms, finished with status: 'timeout' for callable function

I am experiencing an issue with Firebase callable functions and Auth triggers. You can see the callable function below. When it works it usually takes less than 1 second to finish but it started give frequent timeout errors since yesterday. Same thing for the Auth trigger, I was simply returning a Promise that writes user email to the Firestore in that case.
exports.respondToInvite = functions.https.onCall(async (data, context) => {
if (!context.auth) {
throw new functions.https.HttpsError('failed-precondition', 'The function must be called ' +
'while authenticated.');
}
const uid = context.auth.token.uid;
const inviteId = data.inviteId;
const groupId = data.groupId;
const accepted: boolean = data.accepted;
try {
const batch = admin.firestore().batch();
const inviteRef = admin.firestore().collection("invites").doc(inviteId);
batch.update(inviteRef, {
userId: uid,
status: accepted ? "accepted" : "rejected",
})
if (accepted) {
const groupUsersRef = admin.firestore().collection("groups").doc(groupId).collection("users").doc(context.auth.uid);
batch.set(groupUsersRef, {
createdAt: admin.firestore.Timestamp.now()
})
const userRef = admin.firestore().collection("users").doc(uid);
batch.set(userRef, {
"groupId": groupId
});
}
await batch.commit();
return "invitation accepted";
} catch (error) {
console.error(error);
throw new functions.https.HttpsError('failed-precondition', 'invite response failed',error);
}
});
Edit:
Here is the Auth trigger function
exports.newUser = functions.auth.user().onCreate((user) => {
const userRef = admin.firestore().collection("users").doc(user.uid);
return userRef.create({
"email": user.email,
});
});

Delete Firestore documents based on where condtiion [duplicate]

var jobskill_ref = db.collection('job_skills').where('job_id','==',post.job_id);
jobskill_ref.delete();
Error thrown
jobskill_ref.delete is not a function
You can only delete a document once you have a DocumentReference to it. To get that you must first execute the query, then loop over the QuerySnapshot and finally delete each DocumentSnapshot based on its ref.
var jobskill_query = db.collection('job_skills').where('job_id','==',post.job_id);
jobskill_query.get().then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
doc.ref.delete();
});
});
I use batched writes for this. For example:
var jobskill_ref = db.collection('job_skills').where('job_id','==',post.job_id);
let batch = firestore.batch();
jobskill_ref
.get()
.then(snapshot => {
snapshot.docs.forEach(doc => {
batch.delete(doc.ref);
});
return batch.commit();
})
ES6 async/await:
const jobskills = await store
.collection('job_skills')
.where('job_id', '==', post.job_id)
.get();
const batch = store.batch();
jobskills.forEach(doc => {
batch.delete(doc.ref);
});
await batch.commit();
//The following code will find and delete the document from firestore
const doc = await this.noteRef.where('userId', '==', userId).get();
doc.forEach(element => {
element.ref.delete();
console.log(`deleted: ${element.id}`);
});
the key part of Frank's answer that fixed my issues was the .ref in doc.ref.delete()
I originally only had doc.delete() which gave a "not a function" error. now my code looks like this and works perfectly:
let fs = firebase.firestore();
let collectionRef = fs.collection(<your collection here>);
collectionRef.where("name", "==", name)
.get()
.then(querySnapshot => {
querySnapshot.forEach((doc) => {
doc.ref.delete().then(() => {
console.log("Document successfully deleted!");
}).catch(function(error) {
console.error("Error removing document: ", error);
});
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
or try this, but you must have the id beforehand
export const deleteDocument = (id) => {
return (dispatch) => {
firebase.firestore()
.collection("contracts")
.doc(id)
.delete()
}
}
You can now do this:
db.collection("cities").doc("DC").delete().then(function() {
console.log("Document successfully deleted!");
}).catch(function(error) {
console.error("Error removing document: ", error);
});
And of course, you can use await/async:
exports.delete = functions.https.onRequest(async (req, res) => {
try {
var jobskill_ref = db.collection('job_skills').where('job_id','==',post.job_id).get();
jobskill_ref.forEach((doc) => {
doc.ref.delete();
});
} catch (error) {
return res.json({
status: 'error', msg: 'Error while deleting', data: error,
});
}
});
I have no idea why you have to get() them and loop on them, then delete() them, while you can prepare one query with where to delete in one step like any SQL statement, but Google decided to do it like that. so, for now, this is the only option.
If you're using Cloud Firestore on the Client side, you can use a Unique key generator package/module like uuid to generate an ID. Then you set the ID of the document to the ID generated from uuid and store a reference to the ID on the object you're storing in Firestore.
For example:
If you wanted to save a person object to Firestore, first, you'll use uuid to generate an ID for the person, before saving like below.
const uuid = require('uuid')
const person = { name: "Adebola Adeniran", age: 19}
const id = uuid() //generates a unique random ID of type string
const personObjWithId = {person, id}
export const sendToFireStore = async (person) => {
await db.collection("people").doc(id).set(personObjWithId);
};
// To delete, get the ID you've stored with the object and call // the following firestore query
export const deleteFromFireStore = async (id) => {
await db.collection("people").doc(id).delete();
};
Hope this helps anyone using firestore on the Client side.
The way I resolved this is by giving each document a uniqueID, querying on that field, getting the documentID of the returned document, and using that in the delete. Like so:
(Swift)
func rejectFriendRequest(request: Request) {
DispatchQueue.global().async {
self.db.collection("requests")
.whereField("uniqueID", isEqualTo: request.uniqueID)
.getDocuments { querySnapshot, error in
if let e = error {
print("There was an error fetching that document: \(e)")
} else {
self.db.collection("requests")
.document(querySnapshot!.documents.first!.documentID)
.delete() { err in
if let e = err {
print("There was an error deleting that document: \(e)")
} else {
print("Document successfully deleted!")
}
}
}
}
}
}
The code could be cleaned up a bit, but this is the solution I came up with. Hope it can help someone in the future!
const firestoreCollection = db.collection('job_skills')
var docIds = (await firestoreCollection.where("folderId", "==", folderId).get()).docs.map((doc => doc.id))
// for single result
await firestoreCollection.doc(docIds[0]).delete()
// for multiple result
await Promise.all(
docIds.map(
async(docId) => await firestoreCollection.doc(docId).delete()
)
)
delete(seccion: string, subseccion: string)
{
const deletlist = this.db.collection('seccionesclass', ref => ref.where('seccion', '==', seccion).where('subseccion', '==' , subseccion))
deletlist.get().subscribe(delitems => delitems.forEach( doc=> doc.ref.delete()));
alert('record erased');
}
The code for Kotlin, including failure listeners (both for the query and for the delete of each document):
fun deleteJobs(jobId: String) {
db.collection("jobs").whereEqualTo("job_id", jobId).get()
.addOnSuccessListener { documentSnapshots ->
for (documentSnapshot in documentSnapshots)
documentSnapshot.reference.delete().addOnFailureListener { e ->
Log.e(TAG, "deleteJobs: failed to delete document ${documentSnapshot.reference.id}", e)
}
}.addOnFailureListener { e ->
Log.e(TAG, "deleteJobs: query failed", e)
}
}

Resources