I have two firebase triggers increaseCount and decreaseCount for onCreate and onDelete events respectively.
Scenario:
When a user likes something on the client a new path is created in the firebase database and the onCreate tigger is tiggered. The count is then incremented like I would expect.
When a user 'unlikes' something on the client a path is removed in the firebase database and the onDelete trigger is fired. The count is then decremented.
Issue:
When the user quickly likes something and then unlikes something, both the requests to create a new path and delete an existing path are sent to firebase at pretty much the same time.
Sometimes the onDelete trigger fires before the onCreate tigger.
I'm really looking for some insight into best practices here or anyones past experiences with a similar situation.
Here's a watered down version of my onCreate and onDelete functions
exports.increaseCount = functions.database.instance(dbInstance).ref('/path/to/existingPath')
.onCreate((snap, context) => {
ref.transaction(function(count) {
return (count || 0) + 1;
}
});
exports.decreaseCount = functions.database.instance(dbInstance).ref('/path/to/newPath').onDelete((change, context) => {
dbRef.transaction(function(count) {
if(count && (count > 0)){
var newCount = count - 1;
return newCount;
} else {
return 0;
}
}
});
This is not really a race condition. You should expect that Cloud Functions triggers may possibly fire out of order. This because different functions may be running on different server instances, and it wouldn't be scalable for all of them to somehow share a sense of event ordering. This means you may have to do some additional checking in your transactions to make sure that your database is in a state that's actionable for the current function invocation.
Related
I have an events as a parent collection that has Attendee subcollection to record all users that will attend the event like the image below. the Attendee subcollection contains users data
and then also have users as parent collection that has attendedEvents subcollection to record all events that will be visited by the user like the image below. attendedEvents` subcollection events data.
I use denormalization, so it seems the events data is duplicated in attendedEvents subcollection like this
and then I make a cron job using cloud function. this cron job task is to evaluate if an event has been passed (expired) or not. If the event has been passed, then this function should:
update the field of the event data from isActive == true to be isActive == false
read all its Attendee documents in all expired events, get all the attendeeIDs, and then delete all events data in attendedEvents subcollection of users collection.
As you can see, the second task of my cron job functions may need to read around 50.000 - 100.000 documents and then also need to delete around 50.000 - 100.000 documents as the worst case scenario (peak).
So my question is, Is it OK to perform thousand of read and delete operations in one function of Cloud Function like this ?
I am worried there is a limitation that I don't know. I am not sure, is there something that I have not been considered ? is there a better approach for this, maybe ?
Here is my cloud function code:
exports.cronDeactivatingExpiredEvents = functions.https.onRequest(async (request,response) => {
const now = new Date()
const oneMonthAgo = moment().subtract(1,"month").toDate()
try {
const expiredEventsSnapshot = await eventRef
.where("isActive","==",true)
.where("hasBeenApproved","==",true)
.where("dateTimeStart",">",oneMonthAgo)
.where("dateTimeStart","<",now)
.get()
const eventDocumentsFromFirestore = expiredEventsSnapshot.docs
const updateEventPromises = []
eventDocumentsFromFirestore.forEach(eventSnapshot => {
const event = eventSnapshot.data()
const p = admin.firestore()
.doc(`events/${event.eventID}`)
.update({isActive: false})
updateEventPromises.push(p)
})
// 1. update isActive to be false in firestore document
await Promise.all(updateEventPromises)
console.log(`Successfully deactivating ${expiredEventsSnapshot.size} expired events in Firestore`)
// getting all attendeeIDs.
// this may need to read around 50.000 documents
const eventAttendeeSnapshot = await db.collection("events").doc(eventID).collection("Attendee").get()
const attendeeDocuments = eventAttendeeSnapshot.docs
const attendeeIDs = []
attendeeDocuments.forEach( attendeeSnapshot => {
const attendee = attendeeSnapshot.data()
attendeeIDs.push(attendee.uid)
})
// 3. then delete expired event in users subcollection.
// this may need to delete 50.000 documents
const deletePromises = []
attendeeIDs.forEach( attendeeID => {
const p = db.collection("users").doc(attendeeID).collection("attendedEvents").doc(eventID).delete()
deletePromises.push(p)
})
await Promise.all(deletePromises)
console.log(`successfully delete all events data in user subcollection`)
response.status(200).send(`Successfully deactivating ${expiredEventsSnapshot.size} expired events and delete events data in attendee subcollection`)
} catch (error) {
response.status(500).send(error)
}
})
You have to pay attention to a few things here.
1) There are some limits on the side of the Cloud Function. A quota you might hit depending on how you use the data you're reading is Outbound Socket Data which is 10GB/100seconds excluding HTTP response data. In case you hit this quota you can request a quota increase by going to IAM & admin >> Quotas >> Edit Quotas and select Cloud Function API (Outgoing socket traffic for the Region you want).
However, there is also the Maximum function duration of 540 seconds. I believe what you have described should not take that long. In case it does, then if you are committing a batch delete the deletion will be done even if your function fails because of exceeding the duration.
2) On Firestore side, you have some limits too. Here you can read about some best practices when dealing with Read/Write operations and High read, write, and delete rates. Depending on the structure and the type of your data you might encounter some issues such as connection errors if you try to delete lexicographically close documents at a high rate.
Also keep in mind the more generic Firestore quotas on the number of Read/Write operations for each payment plan.
In any way, even with the best calculations there is always a room for error. So my advice would be to try a test scenario with the highest peak you are expecting. If you hit any quotas you can request a quota increase, or if you hit any hard limits you can contact Google Cloud Platform Support providing specific details on your project and use-case.
I've one firebase database instance and I would like to add a counter to a certain node.
Everytime the users run an specific action I would like to increment the node value. How to do that without getting synchronization problems? How to use google functions to do that?
Ex.:
database{
node {
counter : 0
}
}
At certain time 3 different users read the value on counter, and try to increment it. As they read at exact same time all of them read "0" and incremented to "1", but the desired value at end of execution should be "3" since it was read 3 times
==================update===================
#renaud pointed to use transactions to keep synchronization on of the saved data, but i have another scenario where i need the synchronization done on read side also:
ex.
the user read the actual value, acording to it does a different action and finishing by incrementing one...
in a sql like enviorement i would write a procedure for doing that, because doesn't matter what user will do with the info i will finish always by incrementing one
If i did understand #renaud answer right, in that scenario 4 different users reading the database at same time would get 0 as current value, then on transaction update the final stored value would be 4, but on client side each of them just read 0
You have to use a Transaction in this case, see https://firebase.google.com/docs/database/web/read-and-write#save_data_as_transactions and also https://firebase.google.com/docs/reference/js/firebase.database.Reference#transaction
A Transaction will "ensure there are no conflicts with other clients writing to the same location at the same time."
In a Cloud Function you could write your code along the following lines, for example:
....
const counterRef = admin
.database()
.ref('/node/counter');
return counterRef
.transaction(current_value => {
return (current_value || 0) + 1;
})
.then(counterValue => {
if (counterValue.committed) {
//For example update another node in the database
const updates = {};
updates['/nbrOfActionsExecuted'] = counterValue.snapshot.val();
return admin
.database()
.ref()
.update(updates);
}
})
or simply the following if you just want to update the counter (Since a transaction returns a Promise, as explained in the second link referred to above):
exports.testTransaction = functions.database.ref('/path').onWrite((change, context) => {
const counterRef = admin
.database()
.ref('/node/counter');
return counterRef
.transaction(current_value => {
return (current_value || 0) + 1;
});
});
Note that, in this second case, I have used a Realtime Database trigger as an example of trigger.
I build react native app with firebase & firestore.
what I'm looking to do is, when user open app, to insert/update his status to 'online' (kind of presence system), when user close app, his status 'offline'.
I did it with firebase.database.onDisconnect(), it works fine.
this is the function
async signupAnonymous() {
const user = await firebase.auth().signInAnonymouslyAndRetrieveData();
this.uid = firebase.auth().currentUser.uid
this.userStatusDatabaseRef = firebase.database().ref(`UserStatus/${this.uid}`);
this.userStatusFirestoreRef = firebase.firestore().doc(`UserStatus/${this.uid}`);
firebase.database().ref('.info/connected').on('value', async connected => {
if (connected.val() === false) {
// this.userStatusFirestoreRef.set({ state: 'offline', last_changed: firebase.firestore.FieldValue.serverTimestamp()},{merge:true});
return;
}
await firebase.database().ref(`UserStatus/${this.uid}`).onDisconnect().set({ state: 'offline', last_changed: firebase.firestore.FieldValue.serverTimestamp() },{merge:true});
this.userStatusDatabaseRef.set({ state: 'online', last_changed: firebase.firestore.FieldValue.serverTimestamp() },{merge:true});
// this.userStatusFirestoreRef.set({ state: 'online',last_changed: firebase.firestore.FieldValue.serverTimestamp() },{merge:true});
});
}
after that, I did trigger to insert data into firestore(because I want to work with firestore), this is the function(works fine, BUT it takes 3-4 sec)
module.exports.onUserStatusChanged = functions.database
.ref('/UserStatus/{uid}').onUpdate((change,context) => {
const eventStatus = change.after.val();
const userStatusFirestoreRef = firestore.doc(`UserStatus/${context.params.uid}`);
return change.after.ref.once("value").then((statusSnapshot) => {
return statusSnapshot.val();
}).then((status) => {
console.log(status, eventStatus);
if (status.last_changed > eventStatus.last_changed) return status;
eventStatus.last_changed = new Date(eventStatus.last_changed);
//return userStatusFirestoreRef.set(eventStatus);
return userStatusFirestoreRef.set(eventStatus,{merge:true});
});
});
then after that, I want to calculate the online users in app, so I did trigger when write new data to node of firestore so it calculate the size of online users by query.(it works fine but takes 4-7 sec)
module.exports.countOnlineUsers = functions.firestore.document('/UserStatus/{uid}').onWrite((change,context) => {
console.log('userStatus')
const userOnlineCounterRef = firestore.doc('Counters/onlineUsersCounter');
const docRef = firestore.collection('UserStatus').where('state','==','online').get().then(e=>{
let count = e.size;
console.log('count',count)
return userOnlineCounterRef.update({count})
})
return Promise.resolve({success:'added'})
})
then into my react native app
I get the count of online users
this.unsubscribe = firebase.firestore().doc(`Counters/onlineUsersCounter`).onSnapshot(doc=>{
console.log('count',doc.data().count)
})
All the operations takes about 12 sec. it's too much for me, it's online app
my firebase structure
what I'm doing wrong? maybe there is unnecessary function or something?
My final goals:
minimize time operation.
get online users count (with listener-each
change, it will update in app)
update user status.
if there are other way to do that, I would love to know.
Cloud Functions go into a 'cold start' mode, where they take some time to boot up. This is the only reason I can think of that it would take that long. Stack Overflow: Firebase Cloud Functions Is Very Slow
But your cloud function only needs to write to Firestore on log out to
catch the case where your user closes the app. You can write to it directly on log in from your client
with auth().onAuthStateChange().
You could also just always read who is logged in or out directly from the
realtime database and use Firestore for the rest of your data.
You can rearrange your data so that instead of a 'UserStatus' collection you have an 'OnlineUsers' collection containing only online users, kept in sync by deleting the documents on log out. Then it won't take a query operation to get them. The query's impact on your performance is likely minimal, but this would perform better with a large number of users.
The documentation also has a guide that may be useful: Firebase Docs: Build Presence in Cloud Firestore
So I have a cloud function that is triggered each time a transaction is liked/unliked. This function increments/decrements the likesCount. I've used firestore transactions to achieve the same. I think the problem is the Code inside the Transaction block is getting executed multiple times, which may be correct as per the documentation.
But my Likes count are being updated incorrectly at certain times.
return firestore.runTransaction(function (transaction) {
return transaction.get(transRef).then(function (transDoc) {
let currentLikesCount = transDoc.get("likesCount");
if (event.data && !event.data.previous) {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 1 : transDoc.get("likesCount") + 1;
} else {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 0 : transDoc.get("likesCount") - 1;
}
transaction.update(transRef, { likesCount: newLikesCount });
});
});
Anyone had similar experience
Guys finally found out the cause for this unexpected behaviour.
Firestore isn't suitable for maintaining counters if your application is going to be traffic intensive. They have mentioned it in their documentation. The solution they suggest is to use a Distributed counter.
Many realtime apps have documents that act as counters. For example,
you might count 'likes' on a post, or 'favorites' of a specific item.
In Cloud Firestore, you can only update a single document about once
per second, which might be too low for some high-traffic applications.
https://cloud.google.com/firestore/docs/solutions/counters
I wasn't convinced with that approach as it's too complex for a simple use case, which is when I stumbled across the following blog
https://medium.com/evenbit/on-collision-course-with-cloud-firestore-7af26242bc2d
These guys used a combination of Firestore + Firebase thereby eliminating their weaknesses.
Cloud Firestore is sitting conveniently close to the Firebase Realtime
Database, and the two are easily available to use, mix and match
within an application. You can freely choose to store data in both
places for your project, if that serves your needs.
So, why not use the Realtime database for one of its strengths: to
manage fast data streams from distributed clients. Which is the one
problem that arises when trying to aggregate and count data in the
Firestore.
Its not correct to say that Firestore is an upgrade to the Realtime database (as it is advertised) but a different database with different purposes and both can and should coexist in a large scale application. That's my thought.
It might have something to do with what you're returning from the function, as you have
return transaction.get(transRef).then(function (transDoc) { ... })
And then another return inside that callback, but no return inside the inner-most nested callback. So it might not be executing the transaction.update. Try removing the first two return keywords and add one before transaction.update:
firestore.runTransaction(function (transaction) {
transaction.get(transRef).then(function (transDoc) {
let currentLikesCount = transDoc.get("likesCount");
if (event.data && !event.data.previous) {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 1 : transDoc.get("likesCount") + 1;
} else {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 0 : transDoc.get("likesCount") - 1;
}
return transaction.update(transRef, { likesCount: newLikesCount });
});
});
Timeouts
First of all, check your Cloud Functions logs to see if you get any timeout messages.
Function execution took 60087 ms, finished with status: 'timeout'
If so, sort out your function so that it returns a Promise.resolve(). And shows
Function execution took 344 ms, finished with status: 'ok'
Idempotency
Secondly, write your data so that the function is idempotent. When your function runs, write a value to the document that you are reading. You can then check if that value exists before running the function again.
See this example for ensuring that functions are only run once.
I would like to make a chat using Firebase. I need to display for each users the list of group they belong to and also for each group all the members.
Because of how Firebase is designed, and in order to have acceptable performance, I am think making a list of all the groups containing the list of members, and for each users, an entry with all the group they belong too.
My first question is, is it the right approach?
If so, my second question is how can I atomically add (or removed) a user, i.e. making sure both the user is added in the group and the group added into the user or not added at all (i.e. never stored at 1 location only and make the DB inconsistent) ?
The data model you have proposed is consistent with recommendations for firebase. A more detailed explanation is outlined in best way to structure data in firebase
users/userid/groups
groups/groupid/users
Firebase provides .transaction for atomic operations. You may chain multiple transactions to ensure data consistency. You may also use onComplete callback function. For detailed explanation refer to firebase transaction. I am using this same approach to keep multiple message counts up to date for a dashboard display.
Sample code for nested transaction:
ref.child('users').child(userid).child(groupid).transaction(function (currentData) {
if (currentData === null) {
return groupid;
} else {
console.log('groupid already exists.');
return; // Abort the transaction.
}
}, function (error, committed, snapshot) {
if (committed) {
console.log('start a new transaction');
ref.child('groups').child(groupid).child(userid).tranaction(function (currentData) {
if (currentData === null) {
return userid;
} else {
console.log('Userid already exists.');
//add code here to roll back the groupid added to users
return; // Abort the transaction.
}
}, function (error, committed, snapshot) {
if (error) {
//rollback the groupid that was added to users in previous transaction
}
})
}
});