Is it OK to perform thousand of read and delete Firestore operations in one function of Cloud Function? - firebase

Question

I have an events as a parent collection that has Attendee subcollection to record all users that will attend the event like the image below. the Attendee subcollection contains users data
and then also have users as parent collection that has attendedEvents subcollection to record all events that will be visited by the user like the image below. attendedEvents` subcollection events data.
I use denormalization, so it seems the events data is duplicated in attendedEvents subcollection like this
and then I make a cron job using cloud function. this cron job task is to evaluate if an event has been passed (expired) or not. If the event has been passed, then this function should:
update the field of the event data from isActive == true to be isActive == false
read all its Attendee documents in all expired events, get all the attendeeIDs, and then delete all events data in attendedEvents subcollection of users collection.
As you can see, the second task of my cron job functions may need to read around 50.000 - 100.000 documents and then also need to delete around 50.000 - 100.000 documents as the worst case scenario (peak).
So my question is, Is it OK to perform thousand of read and delete operations in one function of Cloud Function like this ?
I am worried there is a limitation that I don't know. I am not sure, is there something that I have not been considered ? is there a better approach for this, maybe ?
Here is my cloud function code:
exports.cronDeactivatingExpiredEvents = functions.https.onRequest(async (request,response) => {
const now = new Date()
const oneMonthAgo = moment().subtract(1,"month").toDate()
try {
const expiredEventsSnapshot = await eventRef
.where("isActive","==",true)
.where("hasBeenApproved","==",true)
.where("dateTimeStart",">",oneMonthAgo)
.where("dateTimeStart","<",now)
.get()
const eventDocumentsFromFirestore = expiredEventsSnapshot.docs
const updateEventPromises = []
eventDocumentsFromFirestore.forEach(eventSnapshot => {
const event = eventSnapshot.data()
const p = admin.firestore()
.doc(`events/${event.eventID}`)
.update({isActive: false})
updateEventPromises.push(p)
})
// 1. update isActive to be false in firestore document
await Promise.all(updateEventPromises)
console.log(`Successfully deactivating ${expiredEventsSnapshot.size} expired events in Firestore`)
// getting all attendeeIDs.
// this may need to read around 50.000 documents
const eventAttendeeSnapshot = await db.collection("events").doc(eventID).collection("Attendee").get()
const attendeeDocuments = eventAttendeeSnapshot.docs
const attendeeIDs = []
attendeeDocuments.forEach( attendeeSnapshot => {
const attendee = attendeeSnapshot.data()
attendeeIDs.push(attendee.uid)
})
// 3. then delete expired event in users subcollection.
// this may need to delete 50.000 documents
const deletePromises = []
attendeeIDs.forEach( attendeeID => {
const p = db.collection("users").doc(attendeeID).collection("attendedEvents").doc(eventID).delete()
deletePromises.push(p)
})
await Promise.all(deletePromises)
console.log(`successfully delete all events data in user subcollection`)
response.status(200).send(`Successfully deactivating ${expiredEventsSnapshot.size} expired events and delete events data in attendee subcollection`)
} catch (error) {
response.status(500).send(error)
}
})

Answer 1

You have to pay attention to a few things here.
1) There are some limits on the side of the Cloud Function. A quota you might hit depending on how you use the data you're reading is Outbound Socket Data which is 10GB/100seconds excluding HTTP response data. In case you hit this quota you can request a quota increase by going to IAM & admin >> Quotas >> Edit Quotas and select Cloud Function API (Outgoing socket traffic for the Region you want).
However, there is also the Maximum function duration of 540 seconds. I believe what you have described should not take that long. In case it does, then if you are committing a batch delete the deletion will be done even if your function fails because of exceeding the duration.
2) On Firestore side, you have some limits too. Here you can read about some best practices when dealing with Read/Write operations and High read, write, and delete rates. Depending on the structure and the type of your data you might encounter some issues such as connection errors if you try to delete lexicographically close documents at a high rate.
Also keep in mind the more generic Firestore quotas on the number of Read/Write operations for each payment plan.
In any way, even with the best calculations there is always a room for error. So my advice would be to try a test scenario with the highest peak you are expecting. If you hit any quotas you can request a quota increase, or if you hit any hard limits you can contact Google Cloud Platform Support providing specific details on your project and use-case.