How to optimize Firestore read/write operation - firebase

I try to make Stock monitoring app where users can make Alert when price change.
I have Stock document where I fetch from 3rd party API periodically.
Upon Stock update I fetch all Alert which meet condition and notify users with Firebase messaging.
Here is my models.
/stocks/{stockId}
id: String
price: Number
/alerts/{alertId}
stockId: String
price: Number
condition: [moreThan, lessThan]
uid: String // User who create alert
/users/{userId}
id: String
fcmToken: String
Every 5 minutes I fetch latest stock price and update stock document = number of stock write query.
function updateStockPrice(): Promise<FirebaseFirestore.WriteResult[]> {
const db = admin.firestore();
const options = {
// Options for stock api
};
return client(options)
.then(function(body) {
let batch = db.batch()
for(var stock in body){
let data = // create data from json response
let docRef = db.collection("stocks").doc(//id from 3rd party api)
batch.set(docRef, data)
}
return batch.commit()
});
}
On stock write update, loop through alerts and fetch fcm token from related user to send out push notifications = (alert query + number of alerts matched condition for user query) x number of stock update
export const onStocksUpdate = functions.firestore.document('stocks/{stockId}').onWrite(function(change, context) {
const db = admin.firestore()
const newValue = change.after.data();
const previousValue = change.before.data();
const stockId = Number(context.params.stockId)
let p1 = db.collection("alerts")
.where("stockId", "==", stockId)
.where("condition", "==", "moreThan")
.where("price", "<=", newValue.lastPrice)
.get()
.then(function(querySnapshot) {
const promises = []
querySnapshot.forEach(function(doc) {
let data = doc.data()
let user = db.collection("users").doc(data.uid).get()
user.then(function(snapshot) {
let userData = snapshot.data()
return userData.fcmToken
}).then(function(fcmToken) {
const payload = {
notification: {
title: // title,
body: // body
},
token: fcmToken
}
return admin.messaging().send(payload)
}).catch(function(error) {
console.log(error)
})
promises.push(user)
})
return Promise.all(promises)
}).then((response) => {
// Response is a message ID string.
console.log('Successfully sent message:', response);
})
.catch((error) => {
console.log('Error sending message:', error);
});
return p1
})
Total write = number of stock
Total read = (alert query + number of alerts matched condition for user query) x number of stock update
Say I have 20 stocks, 1 user with 3 alerts.
Each 5 minutes I will use 20 write query and (1 + 3) x 20 = 80 read
In 1 day = 5760 write and 23,040 read
This seem too much operations for such a small examples. Is this the right way I should lay out data structure? Or is there a better way to do this?

Related

How to properly use 'return' in a firebase cloud function when performing multiple actions against documents

I am trying to understand how to properly use a cloud function..
The cloud function below achieves the desired result, however, I am not using a 'return' in it.
is this an issue? if yes, how do I solve it?
exports.onJobChangeToClosedAndSubstateIsCancelled = functions.firestore
.document("job/{docId}")
.onUpdate(async (change, eventContext) => {
const afterSnapData = change.after.data();
const afterJobState = afterSnapData["Job state"];
const afterJobSubState = afterSnapData["Job substate"];
if (afterJobSubState == "Cancelled" && afterJobState == "Closed") {
// get all job related job applications
const relatedJobApplicationsList = await admin.firestore().collection("job application").where("Job id", "==", change.before.id).get();
// mark job applications closed so that other contractors cant waste their time
relatedJobApplicationsList.forEach(async doc => {
await admin.firestore().collection("job application").doc(doc.id).update({
"Job application state": "Closed",
"Job application substate": "Job cancelled by client",
})
})
}
});
My understanding from other questions such as the one below is that I should be returning the change on the document or the action taking place. ( is this correct? )
Can you help me?-firebase-function-onUpdate
Below is an example of a cloud function where I was able to place the 'return' statement:
// when a chat message is created
// send a notification to the user who is on the other side of the chat
// update the chat 'Chat last update' so that the Chats screen can put the chat at the top of the page
exports.onChatMessageCreate = functions.firestore
.document("chat messages/{docId}")
.onCreate(async (snapshot, context) => {
const snapData = snapshot.data();
// information about who sent the message
const senderUserName = snapData["Chat message creator first name"];
const senderUserId = snapData["User id"];
// chat id to which chat messages belong to
const chatId = snapData["Chat id"];
// information about who should receive the message
let receiverUserId;
let receiverToken = "";
// fetch user to send message to
const chatData = await (await admin.firestore().collection("chat").doc(chatId).get()).data();
const chatUsersData = chatData["User ids list"];
chatUsersData[0] == senderUserId ? receiverUserId = chatUsersData[1] : receiverUserId = chatUsersData[0];
receiverToken = await (await admin.firestore().collection("user token").doc(receiverUserId).get()).data()["token"];
console.log("admin.firestore.FieldValue.serverTimestamp():: " + admin.firestore.FieldValue.serverTimestamp());
// set chat last update
await admin.firestore().collection("chat").doc(chatId).update({
"Chat last update": admin.firestore.FieldValue.serverTimestamp(),
});
const payload = {
notification: {
title: senderUserName,
body: snapData["Chat message"],
sound: "default",
},
data: {
click_action: "FLUTTER_NOTIFICATION_CLICK",
message: "Sample Push Message",
},
};
return await admin.messaging().sendToDevice(receiverToken, payload);
});
Is this an issue?
Yes, by not returning a Promise (or a value since your callback function is async) you are incorrectly managing the life cycle of your Cloud Function.
In your case, since you have an if block, the easiest is to just do return null; (or return 0; or return;) at the end of the CF as follows.
In addition, you need to use Promise.all() since you want to execute a variable number of calls to an asynchronous method in parallel.
exports.onJobChangeToClosedAndSubstateIsCancelled = functions.firestore
.document("job/{docId}")
.onUpdate(async (change, eventContext) => {
const afterSnapData = change.after.data();
const afterJobState = afterSnapData["Job state"];
const afterJobSubState = afterSnapData["Job substate"];
if (afterJobSubState == "Cancelled" && afterJobState == "Closed") {
// get all job related job applications
const relatedJobApplicationsList = await admin.firestore().collection("job application").where("Job id", "==", change.before.id).get();
// mark job applications closed so that other contractors cant waste their time
const promises = [];
relatedJobApplicationsList.forEach(doc => {
promises.push(admin.firestore().collection("job application").doc(doc.id).update({
"Job application state": "Closed",
"Job application substate": "Job cancelled by client",
}));
})
await Promise.all(promises);
}
return null;
});
More info on how to properly manage CF life cycle here in the doc.
You may also be interested by watching the talk I gave at the I/O Extended UK & Ireland event in 2021. This talk was specifically dedicated to this subject.

why I receive null after push new notes to firebase (realtime database)

I push new info for user in category folder.
If I do it in version 8 like this:
const category = await firebase.database().ref(`/users/${uid}/categories`).push({title, limit})
I can receive feedback with information about notes (title, limit, and also special key, that added in firebase) in const category.
But when I use version 9, my const category is null..
However request work correctly, and in database I can see pushed info with special key.
async createCategory({ commit, dispatch }, { title, limit }) {
try {
const uid = await dispatch('getUid')
const db = getDatabase();
const postListRef = ref(db, `/users/${uid}/categories`);
const newPostRef = push(postListRef);
const category = await set(newPostRef, {title, limit});
return {title, limit, id: category.key}
} catch (e) {
commit('setError', e)
throw e
}
}
How to receive feedback from database after push new info? (in const category)
I want have {title, limit, id: category.key}

Firebase listUsers fails to get All users after a certain page

I'm using a pubsub firebase function (cron), and inside this function Im calling firebase auth users, to fill some missing data in a profile collection
Im paginating with the pageToken, the first token passed is undefined then I save it in a config db and read the token to get the next page
The issue is that I have 170K records, and listusers returns an undefined token at the 6th page (6k users) which is frsutrating
here is the code:
functions.pubsub
.schedule('*/30 * * * *')
.onRun(async () => {
const page = firestore.collection('config').doc('pageToken');
const doc = (await page.get()).data();
// Check if last page don't run again
const last = doc?.last;
if (last) return;
// Load page
const pageToken = doc?.pageToken || undefined;
let pageNumber = doc?.pageNumber as number;
return firebaseAdmin
.auth()
.listUsers(1000, pageToken)
.then(async listUsersResult => {
for (const userRecord of listUsersResult.users) {
// Fetch Profile
try {
const profile = await firestore
.collection('profiles')
.doc(userRecord.uid);
// data foramtting here
// compared profile data & fixed data
const payload = JSON.parse(
JSON.stringify({
...profileData,
...{
lastName,
firstName,
language,
...(!userRecord.phoneNumber && {
phone,
}),
},
})
);
// Profile doesn't exist : Create
if (!profileData && payload) {
await profile.create({
...payload,
...{
Migrated: true,
},
});
} else if (profileData && payload) {
const data = compare(profileData, payload);
if (data) {
// Profile exists: Update
await profile.update(data);
if (userRecord.phoneNumber)
await profile.update({
phone: firebaseAdmin.firestore.FieldValue.delete(),
});
}
}
} catch (err) {
functions.logger.error('Some Error', err);
}
}
if (!listUsersResult.pageToken) {
return await firestore
.collection('config')
.doc('pageToken')
.update({
last: true,
});
}
// List next batch of users.
pageNumber++;
return await firestore
.collection('config')
.doc('pageToken')
.update({
pageToken: listUsersResult.pageToken,
pageNumber,
});
});
});
so after in page 6, I have a last:true property added to the firestore however there is 164k data are missing
any idea ?

Firebase Cloud Function not executing

I'm working on the group functionality of my app where members of the group can add task to a group that they are currently working on. So, when a task is added I want to notify all members using FCM that a task had been added to the group.
EDIT:
The code to add the task to a group is run on the client and works successfully. The purpose of the cloud function is just to send cloud messages to all the members of the group that a task has been added by a particular member.
This is my logic for the cloud function :
1st. As a task can be added to multiple groups at a time I'm using a forEach().
2nd. I'm fetching the uids of the Members in the groups and pushing them into an array(uids) so that later I can retrieve their fcmTokens.
3rd. Running a forEach on the uids to retrieve the fcmTokens.
4th.Sending Cloud message to devices.
But the cloud function doesn't execute as expected.
This my cloud function:
exports.newTaskAdded = functions.https.onCall(async (data, context) => {
const groups = data.groups; //Can be multiple groups hence an array.
const uid = data.uid;
const author = data.author;
const taskId = data.taskId;
const taskTitle = data.taskTitle;
try {
groups.forEach(async group => {
const groupName = group.groupName;
console.log('groupName: ', groupName);
const groupId = groups.groupId;
const membersPromises = [];
membersPromises.push(
admin
.firestore()
.collection('Groups')
.doc(`${groupId}`)
.collection('Members') //Members collection has document for each user with their uid as the document name.
.get(),
);
console.log('memberPromises: ', membersPromises);//Function stops after this.
const membersSnapshot = await Promise.all(membersPromises);
console.log('membersSnapshots', membersSnapshot);
const uids = [];
membersSnapshot.forEach(doc => {
doc.forEach(snap => {
console.log(snap.id);
uids.push(snap.id);
});
});
console.log(uids);
const uidPromises = [];
uids.forEach(uid => {
uidPromises.push(
admin
.firestore()
.collection('Users')
.doc(`${uid}`)
.get(),
);
});
console.log('uidPromises: ', uidPromises);
const tokensSnapshots = await Promise.all(uidPromises);
const notifPromises = [];
tokensSnapshots.forEach(snap => {
console.log(snap.data());
const token = Object.keys(snap.data().fcmTokens);
const payload = {
notification: {
title: `${author} has added a new task to ${groupName}`,
body: `Task Added: ${taskTitle}`,
sound: 'default',
},
};
notifPromises.push(admin.messaging().sendToDevice(token, payload));
});
await Promise.all(notifPromises);
});
} catch (err) {
console.log(err);
}
return {result: 'OK'};
});
This is my log:
As you can see there is no error shown.
Help would be very much appreciated. Thank you

How to add a field to the document which is created before?

I need to store some details entered in Dialogflow console into Firestore database,
I have this following code:
index.js
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp(functions.config().firebase);
exports.dialogflowFirebaseFulfillment = functions.https.onRequest(
(request, response) => {
let action = request.body.result.action;
var Name = request.body.result.parameters.Name;
let params = request.body.result.parameters;
var Answer1 = request.body.result.parameters.Answer1;
let query = request.body.result.resolvedQuery;
const parameters = request.body.result.parameters;
const inputContexts = request.body.result.contexts;
console.log("Parameters is" + params);
console.log("Answer1 is " + Answer1);
console.log("Helllo");
console.log("action is " + action);
if (action == "save.name") {
admin
.firestore()
.collection("users")
.doc(Name)
.set({
name: Name,
answer: ""
})
.then(ref => {
var doc = admin
.firestore()
.collection("users")
.doc(Name)
.get();
console.log("Added new user");
console.log("Name is " + Name);
console.log("ref id is:" + ref.id);
});
}
//const ref = this.Name;
if (action == "save.answer1") {
// var Name = this.request.body.result.parameters.Name;
console.log("name in second if", Name);
admin
.firestore()
.collection("users")
.doc(doc)
.update({
answer: Answer1
})
.then(ref => {
console.log("Added new user");
console.log("Name is " + Name);
console.log("ref id is:" + ref.id);
});
}
}
);
This code successfully adds the name into the database, then quits out of code and again re-enters to save Answer1, So I don't know how to refer the document where the name is added, If I hard code the 'Name' previously entered by the user, then it adds the answer1 under the same document
example:
if (action == "save.answer1") {
admin
.firestore()
.collection("users")
.doc("ABC")
.update({
answer: Answer1
})
I have here hard-coded the name of the document as ABC so it adds answer1 under the document, How to avoid hard-coding the name?
I need to get the DocumentId of the document where Name is stored, So that I can add Answer1,
expected Output is:
First - don't try to find a query that finds the "last document added". I don't know if it exists, but it will do the wrong thing if you have two different users in different sessions add documents and then try to update them. This is a race condition, and it is very bad.
Instead, what you can do is save the document identifier (the name in your case) in a parameter in a Dialogflow Context. Then, the next time your webhook is called, you can retrieve the parameter from that context and use that in the .doc() reference.
You don't show how you're sending messages back to Dialogflow, but it looks like you might be using the Dialogflow V1 JSON. If so, you'd set the contexts back in the response under the contextOut attribute. This will be an array of contexts you want to set. So this portion of the JSON fragment might look something like:
"contextOut": [
{
"name": "doc_name",
"lifespan": 5,
"parameters": {
"name": Name
}
}
]
When handling the "save.answer1" action, you would go through the incoming contexts to find the correct context and use this name in your update. So that segment might look something like this:
if (action == "save.answer1") {
var Name;
for( var co=0; inputContexts && co<inputContexts.length && !Name; co++ ){
var context = inputContexts[co];
if( context.name == 'doc_name' ){
Name = context.parameters.name;
}
}
console.log("name in second if", Name);
admin
.firestore()
.collection("users")
.doc(Name)
.update({
answer: Answer1
})
.then(ref => {
console.log("Updated user");
console.log("Name is " + Name);
console.log("ref id is:" + ref.id);
});
}

Resources