Let's say I have a Cloud Firebase Function - called by a cron job - that produces 30+ tasks every time it's invoked.
These tasks are quite slow (5 - 6 second each in average) and I can't process them directly in the original because it would time out.
So, the solution would be invoking another "worker" function, once per task, to complete the tasks independently and write the results in a database. So far I can think of three strategies:
Pubsub messages. That would be amazing, but it seems that you can only listen on pubsub messages from within a Cloud Function, not create one. Resorting to external solutions, like having a GAE instance, is not an option for me.
Call the worker http-triggered Firebase Cloud Function from the first one. That won't work, I think, because I would need to wait for a response from the all the invoked worker functions, after they finish and send, and my original Function would time out.
Append tasks to a real time database list, then have a worker function triggered by each database change. The worker has to delete the task from the queue afterwards. That would probably work, but it feels there are a lot of moving parts for a simple problem. For example, what if the worker throws? Another cron to "clean" the db would be needed etc.
Another solution that comes to mind is firebase-queue, but its README explicitly states:
"There may continue to be specific use-cases for firebase-queue,
however if you're looking for a general purpose, scalable queueing
system for Firebase then it is likely that building on top of Google
Cloud Functions for Firebase is the ideal route"
It's not officially supported and they're practically saying that we should use Functions instead (which is what I'm trying to do). I'm a bit nervous on using in prod a library that might be abandoned tomorrow (if it's not already) and would like to avoid going down that route.
Sending Pub/Sub messages from Cloud Functions
Cloud Functions are run in a fairly standard Node.js environment. Given the breadth of the Node/NPM ecosystem, the amount of things you can do in Cloud Functions is quite broad.
it seems that you can only listen on pubsub messages from within a Cloud Function, not create one
You can publish new messages to Pub/Sub topics from within Cloud Functions using the regular Node.js module for Pub/Sub. See the Cloud Pub/Sub documentation for an example.
Triggering new actions from Cloud Functions through Database writes
This is also a fairly common pattern. I usually have my subprocesses/workers clean up after themselves at the same moment they write their result back to the database. This works fine in my simple scenarios, but your mileage may of course vary.
If you're having a concrete cleanup problem, post the code that reproduces the problem and we can have a look at ways to make it more robust.
Related
I need a bit of expert advice. I'm using the firebase cloud function to automate few things ( using this brilliant nodejs package "https://github.com/jdgamble555/adv-firestore-functions".
What happens is, it runs on - onWrite trigger, as I understand when onWrite triggers, it executes the function on each new document or childnode within a document being updated or created or deleted. That package has taken care many of things, but my concern is, executing the functions multiple times does any harm? I'm already making sure than if not required do not hit firestore by using condition checks. So (all) the function executes as I can see it in the log not writing/updating firestore db if not required.
I'm worried if all functions execute all the time will I finish my limits quickly. ( right now I'm testing on Firebase emulator), specially when userbase with increase.
Anything can be done to reduce these calls or its normal?
As per the firebase documentation, the first 2 million invocations are free per month, after that 0.40$ for every million invocations. Also, there is a resource limit for each function call. https://cloud.google.com/functions/pricing#free_tier
As of my knowledge and experience, this is normal. Just make sure that your code does not make any infinite function calls & database reads/writes.
I'm also using cloud functions for my social media platform which also uses triggers to execute & write to the database based on conditions. It never went beyond the free quota.
So, I'm making multiplayer mobile game using Xamarin and Firebase. In game there are many moment when I'm letting players decide what to do and send their decision to the server (by putting decision enum in player-specific Firebase database node). Decision is time limited (short time, no longer than 20s).
I set listener to that specific node in my Firebase functions to check if all player decided or player decision comes after time deadline, but I need to deal with case when: some players send their decision in time - sersnmart were will not execute next action, and that one player just will not send his decision (eave game or something) - server won't be poke again to check deadline and invoke functions.
That why I'm looking for something else, I found method for schedule functions using crontab, but the minimal time interval there seems to be minutes, which is way more to long for me.
Second idea includes wait that specific time interval in previous Firebase thread, but it seems too bad way to deal with this.
Which way is best for dynamic invoking short-interval scheduled Firebase functions?
The best way to schedule Cloud Functions to run at a specific time it through the Google Cloud Tasks schedules. See Doug's blog post for a full description of this: How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL)
That said, I regularly use setTimeOut in my own Cloud Functions too when I need to delay an operation for a short period of time. Just keep in mind that you pay for the seconds that the function is sleeping, so cost-wise you'll want to trade that time off against what another invocation would cost.
So for now I decided to use setTimeout, free firebase plan seems to limit only functions invoke number, not working plan so this shouldn't be a problem. Depsite this, I'm still waiting for advice from you
I need to run a function in firebase every 10 seconds (I call an external api and process data).
With a normal cron I cant because it is limited to maximum once a minute. Using a setTimeOut is also not convenient since Functions charges per second of use.
My only idea so far is to use Cloud Tasks. But I don't know if it is the most convenient to use this tool for my purpose.
On Google Cloud Platform, Cloud Tasks is almost certainly the best/only way to get this job done without using other products in a way that they weren't intended. Cloud Functions is not good for the reasons you mentioned. With Cloud Tasks, you will need to have the prior task schedule the next task upon completion, as they will not repeat themselves automatically like cron.
I'm building an app using Firebase. There's an admin setting I want to build, that essentially populates one of my nodes with data. It would do this every 10 minutes, and there are 50-80 data points that I'd want to add.
So this would take roughly 13 hours total, I'd probably want to expand this though in the future. I would only call this function, maybe once a week.
I would simply do this using a setTimeout but I've heard this can be expensive? Does someone know roughly how expensive this would be?
I'm not that experienced with CRON jobs, is there a better way of doing this with Firebase? It's not something I want to have running constantly, and it's not at a specific time, but just whenever I need it. I'd also potentially want to have the job running multiple times, at the same time. Which seems to be super easy using Firebase Functions.
Any ideas? Thank you!
You would have to trigger the functions from an outside source.
One way you could do that is by creating and subscribing to a pub-sub system:
exports.someSubscription = functions.pubsub.topic('some-subscription').onPublish((event) => {
// Your code here
});
Another way you could trigger the functions is by exposing them as http requests and hitting those endpoints - See this amazing article on the firebase documentation for some resources on creating http endpoints.
Both of these ways require you to have some sort of scheduling service to either publish a new event to your subscription or to hit the http endpoint. You will need to either implement these services yourself, however I am sure there are services out there that can do that for you.
I'm interested in evaluating the performance of using Cloud Functions for Firebase, so I've set up a simple function with basically the same code as this example from Firebase. What the function does is that is simply listens for new messages at a certain node in Firebase, and then sends a push notification to the receiver of that message.
Next, I wrote a script that bombards the messages node by adding x new children there every second. I found out that the function seems to be able to handle up to about 40 new messages per second, anything more than that and the notifications start to lag more and more behind. This seems like pretty poor performance – too poor for us to use in production – and I wonder what the bottleneck might be here. Of course, Functions are still in beta, but according to the documentation, each function should be able to run 400 concurrent invocations.