I am running a cronjob every evening to send notifications to a set of users in my database, call them 1-100. However, recently with an increase in my users, it's taking longer and longer for this job to run. Is there a way to have the pub/sub topic trigger multiple jobs, say triggering one job to handle users 1-50, and another to handle users 51-100?
Sure, just partition your users according to some bucketing scheme (maybe modulo their UID), create more pubsub topics and functions for those partitions, and send messages to those topics for each of the the partitions of users that you want to process. So, in your main scheduled function, just send messages to each pubsub topic partition, and let the delegate functions do their work.
Also, if your haven't already, consider increasing the timeout for the main function in case it just needs more time to finish processes. The max timeout for a Cloud Functions background trigger is 9 minutes.
Related
Use case :
I have a webhook which handles incoming slack messages. I have to respond to that webhook with 200 OK status within 300 ms otherwise it will retry hitting my webhook again.
The webhook is basically a http firebase cloud function. I have to process the incoming webhook immediately without any delays. the processing will be more than 300 ms most of the times which is leading to multiple hits on my webhook although i returned a 200OK status.
what i was doing till now was immediately saving the incoming webhook's payload to a firestore collection and handling the processing in onCreate of the document. But onCreate takes a few seconds to be triggered which is unacceptable in our case. I am not sure if this is because of cold-start.
I want to see if I can use pub/sub triggers for this. The following are my doubts
How much time does it take to publish to the topic
How much delay is there after I publish to the topic and the subscriber function starts executing. If this has the same delay as of onCreate then this won't work.
If this doesn't work what is the best way to proceed?
Cloud tasks are pretty quick to respond. I've used them to call the server again to process time consuming tasks such as uploading content submitted by user to the user to something like Cloud Storage or any CDN.
You can actually set the duration in how long the Google Task should trigger. Using it with Cloud Compute the trigger is almost instantaneous. In Cloud functions, the worst case scenario would be a cold start but that should be rare if you keep invoking them frequently. Just set the timeout for cloud tasks to 0 and it's call your function immediately. But again, sometimes Cloud Compute may be a better choice than Cloud Functions.
If I understand correctly, you’d like to respond to incoming Slack webhook requests with HTTP 200 OK in under 300ms while durably triggering downstream computation. Your proposed use of Cloud Pub/Sub is to publish the webhook contents to a topic, respond to the webhook request immediately and trigger a Cloud Function for the computation using the Pub/Sub topic.
This is a reasonable solution. While Cloud Pub/Sub does not offer an SLA on latency, the service is optimized for low publish latency and so most requests will be successfully handled in the 300ms envelope provided the Publisher object is already initialized and reused. The actual delivery of messages to the function can take slightly longer, but if there is a consistent flow of messages it should be within 1-2 seconds. Please note that Pub/Sub guarantees at-least-once deliveries which means the function may be triggered multiple times for the same message so your processing logic should be idempotent.
I have an app, where I have an admin and a customer.
Admins can invite users that are in a line.
Admins can configure maximum waiting time for which a customer should show up after he gets invited.
If he does not show up in specified period of time I need to update the database to set the status of a customer's ticket to "no-appearance" and send notification to a customer.
How can I do that using firebase cloud functions?
It looks like I cannot setTimeout because functions are limited to 540 seconds and it is not applicable in my case.
Looks like Pub-sub functions can only be created with a specific period which you have to determine statically
You can use Google Cloud Task to send a scheduled job to trigger cloud functions. Here an article about that: https://medium.com/swlh/event-driven-scheduled-jobs-using-gcp-cloud-tasks-e712c760ab95
The difference of a Pub/Sub is that you can send a task with a dynamic schedule and that task runs only once.
Actually what I am working on a to-do task manager app. Currently I am adding tasks to cloud Firestore after as a new document. I want to send notification to user at the time user scheduled to do that work. I am so confused. It can be lots of tasks from a single user. And the the notification tone has to be selected by user itself.
Please help me in this..
Thanks you!
So you'll need:
A database where you keep the notification payload, user FCM token, and the time they are to be sent.
A schedule Cloud Function that runs every day or every hour based on the granularity of the task.
Now, when the Cloud Function gets triggered, it:
Runs a query against the database for notifications that are scheduled to be sent before the current time
Loops over the results send each notification in turn.
You can now delete the database entry if you want to.
Furthermore, if you want to reduce the number of cloud function triggers, you have a combination of the FCM and an alarm manager to show a notification after a delay (feasible in flutter through local notifications too). this way you can send the payload at a particular time of the day and deploy it at any time locally.
Not sure how optimized this is, looking for better ways!
I am trying to schedule a function to run based on DateTime object field in my Firestore database.
Example, I have a collection of events in the Database and each event has a DateTime field (Start time) for the start of the event. The event also has a list of users that are subscribed to the event.
Is there a way to use Cloud Functions to notify the subscribed users that the event is about to start at the start time of the event or 5 or 10 mins before the event starts?
I am currently using Cloud Functions for notification and its working fine but that is based writes to specific portions of the database, not time event.
Thank you in advance.
The only way to do this with a scheduled Cloud Function for this sort of work is to schedule it run periodically (every 1 minute is the most often), and query for documents that meet the criteria for when you want to send the notification at the moment that the function is currently running. This means you should be looking for documents where the time is in the 5-10 minute near future from the current time.
You can't use scheduled Cloud Functions to delay some work to happen later. But you can integrate with Cloud Tasks for that.
According to firestore documentation:
a transaction is a set of read and write operations on one or more documents.
Also:
Transactions will fail when the client is offline.
Now the limitation in firestore is that:
In Cloud Firestore, you can only update a single document about once per second, which might be too low for some high-traffic applications.
So using cloud functions and running transactions to increment/decrement counters when the traffic is high will fail.
So they have discussed to use the approach of distributed counters.
According to the algorithm of distrbuted counter:
create shards
choose a shard randomly
run a transaction to increment/decrement the counter
get all the shards and aggregate the
result to show the value of a counter
Scenerio:
consider you have a counter which is to be updated when a document is added and that counter is being displayed in the UI. Now for good UX, I cannot block the UI when network is offline. So I must allow creation/updation of documents even when client is offline and sync these changes once client is online so that everyone else listening to these changes receive the correct value of the counter.
Now transactions fail when the client is offline.
So my question for best user experience (even when offline) is:
Do you really require a transaction to increment a counter? I know
transactions ensure that writes are atomic and are either
successful/unsuccessful and prevent partial writes. But what's the
point when they fail offline? I was thinking maybe write them to local cache and sync it once the network is back online.
Should this be done via client sdks of via cloud functions?
Do you really require a transaction to increment a counter?
Definitely yes! Because we are creating apps that can be used in a multi user environment, transactions are mandatory, so we can provide consistent data.
But what's the point when they fail offline?
When there is a loss of network connectivity (there is no network connection on user device), transactions are not supported for offline use. This is because a transaction absolutely requires round trip communications with server in order to ensure that the code inside the transaction completes successfully. So, transactions can only execute when you are online.
Should this be done via client sdks of via cloud functions?
Please note, that the Firestore SDK for Android has a local cache that's enabled by default. According to the official documentation regarding Firestore offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
So all read operations will come from cache if there are no updates on the server. So Firestore provides this feature for handle offline data.
You can also write a function in Cloud Function that will increment the counter while a new document is added or to decrement the conter while a document is deleted.
I also recommend you to take a look:
How to count the number of documents under a collection in Firestore?
So you may also consider using Firebase realtime database for that. Cloud Firestore and Firebase realtime database work very well together.
Edit:
It allows one to upvote the answer even when the device is offline. After the network is online, it syncs to the server and the counter is updated. Is there a way i can do this in firestore when the device is offline.
This is also happening by default. So if the user tries to add/delete documents while offline, every operation is added to a queue. Once the user regains the connection, every change that is made while offline, will be updated on Firebase servers. With other words, all queries will be commited on the server.
Cloud fnctions are triggered only when the change is received and that can only happen when the device is online.
Yes, that correct. Once the device regains the network connection, the document is added/deleted from the database, moment in which the function fires and increases/decreases the counter.
Edit2:
Suppose I have made around 100 operations offline, will that not put a load on the cloud functions when the device comes online? What's your thought on this?
When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations. So I suggest use this database for its online capabilities.
Regarding Cloud Functions for those 100 offline operations, there will be no issues. Firebase servers work very well with concurent operations.