i'm deploying firebase cloud functions to listen on changes in our firestore via continuous deployment.
I can only find a way to activate retries manually. Which goes contrary to our cd approach.
Looking at the normal gcp cloud functions the retry flag can be given when deploying. But i cannot find a similar option in the firebase-cli or the firebase-functions interface (2.1.0)
Any hints on how to solve this?
Carsten
You can enable retries in Firebase Functions using GCloud Console, manually by hand. Programmatically retrying trigger-based functions was added in firebase-functions 3.10.0 (see changelog and associated pull request).
Since it's not entirely obvious from the PR or the docs, here's a quick example of the syntax:
export const myFirebaseFunc = functions
.runWith({
failurePolicy: {
retry: {},
},
memory: '512MB',
timeoutSeconds: 60,
})
.firestore.document('/path/to/some/doc')
.onCreate(async (snap, context) => {
/* do stuff */
})
At the time of writing this, it looks like the failure policy is simply on or off. Thus, this is equivalent
export const myFirebaseFunc = functions
.runWith({
failurePolicy: true,
memory: '512MB',
timeoutSeconds: 60,
})
.firestore.document('/path/to/some/doc')
.onCreate(async (snap, context) => {
/* do stuff */
})
Some caveats:
You'll also have to deploy with --force
You can enable retries only on triggered functions, not http-called functions.
It would be idiotic not to build in some safeguards. Retry policy maxes out at 7 days, and bills just like any other function invocation, so if you have some unhandled error, it could repeatedly run for a full week. You can use context.eventTimestamp to know when the first attempt roughly started.
Read this: https://firebase.google.com/docs/functions/retries and make sure your function in idempotent.
It was also difficult to discover what to return to force a retry or avoid a retry.
Triggered Firebase functions must return a Promise. (See this vid)
A retry-enabled Firebase function will retry if:
it returns a rejected promise
throws an exception
or the function times out
That means that if you run into an error that you know won't eventually resolve itself with a retry (i.e. you want to stop the function execution and not retry), you can return Promise.resolve({message: 'some message'});
There is currently no similar option for deployment using the Firebase CLI.
This is something being worked on by the Firebase team, so stay tuned for updates.
Related
Using Firebase Functions, I have code that runs every hour via a Google Cloud Scheduler Job.
It looks like this:
exports.hourly_tick =
functions.pubsub.topic("hourly-tick").onPublish((message, context) => {
return getData()
.then((data) => {
sendEmail(data["message"]);
})
.catch((error) => {
return console.log("🚩 Caught error: ", error);
});
});
I need to be able to test this locally, and am able to start my Firebase Emulator via firebase emulators:start from my terminal. However I do not know how to trigger this function in my local test environment to see logs in the local emulator.
How can I test this scheduled job / firebase function with the local emulator?
This is an ongoing feature request in Firebase tools (see GitHub issue).
As mentioned in the thread:
I think we maybe misled with how we represented #2011. It lets those functions be loaded into the emulator but doesn't actually trigger them on a schedule. Instead you'd have to manually trigger them using a Pub/Sub message.
You can check a workaround on this answer where you'd have to manually trigger a scheduled function using a Pub/Sub message.
I am working on cloud functions especially schedule functions. I need to trigger a function periodically each 5 minutes, but in only test step. I need to run it on pubsub emulator without deploying it.
How to do it?
I tried to use firebase shell, but it triggered only once
exports.scheduledFunctionPlainEnglish =functions.pubsub.schedule('every 2 minutes')
.onRun((context) => {
functions.logger.log("this runs every 2 minutes")
return null;
})
Scheduled functions are loaded to the Cloud Functions emulator runtime and are bound to the PubSub emulator topic.
But as #samstern said (https://github.com/firebase/firebase-tools/issues/2034):
you'd have to manually trigger them using a Pub/Sub message.
You can do it like this:
import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
import { PubSub } from '#google-cloud/pubsub';
if (!admin.apps.length) {
admin.initializeApp();
}
const pubsub = new PubSub({
apiEndpoint: 'localhost:8085' // Change it to your PubSub emulator address and port
});
setInterval(() => {
const SCHEDULED_FUNCTION_TOPIC = 'firebase-schedule-yourFunctionName';
console.log(`Trigger sheduled function via PubSub topic: ${SCHEDULED_FUNCTION_TOPIC}`);
const msg = await pubsub.topic(SCHEDULED_FUNCTION_TOPIC).publishJSON({
foo: 'bar',
}, { attr1: 'value1' });
}, 5 * 60 * 1000); // every 5 minutes
Additional info about this concept (thanks to #kthaas):
https://github.com/firebase/firebase-tools/pull/2011/files#diff-6b2a373d8dc24c4074ee623d433662831cadc7c178373fb957c06bc12c44ba7b
https://github.com/firebase/firebase-tools/pull/2011/files#diff-73f0f0ab73ffbf988f109e0a4c8b3b8a793f30ef33929928a892d605f0f0cc1f
As you said, you can use firebase shell to run your function once.
And in firebase shell, you can use NodeJS commands.
Use setInterval
Inside firebase functions:shell, use setInterval to run your function every 2 minutes.
user#laptop:~$ firebase functions:shell
✔ functions: functions emulator started at http://localhost:5000
i functions: Loaded functions: myScheduledFunction
firebase > setInterval(() => myScheduledFunction(), 120000)
> this runs every 2 minutes
Single line script
Since version 8.4.3 of firebase-tools, and especially this PR, the pipe solution does not work anymore.
In Bash, you can even pipe the setInterval command to firebase shell
user#laptop:~$ echo "setInterval(() => myScheduledFunction(), 120000)" | firebase functions:shell
For those of you seeing this in 2023, it's still not supported.
My solution was to abstract the code that does the "work" out of functions.pubsub.schedule and into their own functions. Then create a separate file (i added it at the top of the functions folder) with a setInterval inside it that fires the aforementioned abstracted function.
For example, somewhere in your code:
exports.myScheduledFunctionCode = () => {
console.log('why, hello there interval');
}
And in the timers.js (for example) file at the top of the /functions directory:
setInterval(() => {
myScheduledFunctionCode();
}, 60000);
Then, you can fire up your Firebase Emulator suite. In another Terminal session, just run a vanilla $ node functions/timers.js. Now your scheduled function code is running, and your whole emulator suite too.
Hope this helps someone!
This is currently not supported for scheduled functions. The documentation states:
Using the shell, you mock data and perform function calls to simulate interaction with products that the Emulator Suite does not currently support: Storage, PubSub, Analytics, Remote Config, Storage, Auth, and Crashlytics.
Scheduled functions are an unsupported extension of pubsub triggers.
Feel free to file a feature request with Firebase support.
I am trying to set up a scheduled function in Firebase Cloud Functions. As a simple test, I have tried to recreate the sample shown on the documentation page:
const functions = require('firebase-functions')
exports.scheduledFunction = functions.pubsub
.schedule('every 5 minutes')
.onRun(context => {
console.log('This will be run every 5 minutes!')
return null
})
However, when I run firebase serve --only functions, I get the following error:
function ignored because the pubsub emulator does not exist or is not running.
Any idea why I get this message and how I can fix it?
From the documentation on Firebase's local emulator:
The Firebase CLI includes a Cloud Functions emulator which can emulate the following function types:
HTTPS functions
Callable functions
Cloud Firestore functions
So the local Firebase emulators don't currently support pubsub, and the error message seems to confirm that. So for the moment, you can't run pubsub triggered Cloud Functions locally.
A feature request for adding PubSub support to the emulator was filed. You might want to read up (and possibly comment) there, as the direction taken may or may not match with your needs.
The local shell does support invoking pubsub functions. That is of course quite different, but might be useful as a workaround for the moment.
For what it is worth, you need to enable the pubsub emulator in firebase. Add this to your emulators block:
{
"emulators": {
"pubsub": {
"port": 8085
},
}
}
Even then, it only creates the definition. The emulator doesn't support running the function on a schedule.
To simulate that behavior, I define a HTTP trigger, in which I manually send a message to the topic. For a schedule topic, it is firebase-schedule-<functionName>. In your case it will be firebase-schedule-scheduledFunction.
Sample code looks like:
const pubsub = new PubSub()
export const triggerWork = functions.https.onRequest(async (request, response) => {
await pubsub.topic('firebase-schedule-scheduledFunction').publishJSON({})
response.send('Ok')
})
Then on the command line, I trigger the HTTP function on a schedule.
while [ 1 ];
do wget -o /dev/null -O /dev/null http://localhost:5001/path/to/function/triggerWork;
sleep 300;
done
Is there any way to manually trigger a scheduled function and/or a Firestore trigger function? I have two scenarios I need to solve:
A cloud function that is listening to a Firestore document (onCreate) didn't fire - it failed on 3 of about 1,000 invocations, so I need to manually trigger it for these 3 documents. Is this possible (to manually trigger this function)?
I have a scheduled function that runs hourly, but threw an error b/c of a map in the Firestore document when the code expected an array. Any way I can manually run the scheduled function once rather than waiting an hour before it runs again?
-- firebase console
-- functions
-- "..." at right side of cron job
-- "view in cloud scheduler"
-- "run now" at right side of function
You can run a firestore scheduled function via the FirebaseTools and running it locally. Starting the shell command eg npm run build && firebase functions:shell will allow you to invoke a Scheduled Function eg:
export const parseGarminHealthAPIActivityQueue = functions.region('europe-west2').runWith({
timeoutSeconds: TIMEOUT_IN_SECONDS,
memory: MEMORY
}).pubsub.schedule('every 10 minutes').onRun(async (context) => {
await parseQueueItems(ServiceNames.GarminHealthAPI);
});
It's not possible to manually trigger a function from the Firebase console. Your best bet is to use the methods shown in the Cloud documentation, which involve using gcloud's call command or the Cloud console's Testing tab. Neither of these are very easy, as you will have to construct the JSON payload to the function manually.
If I may make a suggestion - if your functions are failing due to errors, you should consider enabling retry on your functions, and making sure that your functions only generate errors for situations that should be retried. Depending on manual invocation in the event of a failure will not scale very well - errors should be handled by code as much as possible.
I have a firebase HTTP function which in turns calls some firestore operations. If I call the HTTP function several times, letting each call finish before calling the next, I get the following error in the firebase functions log:
(node:2) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
The firebase function is an import task which takes the data to import, check for duplicates by calling a firestore query, and if there is none, it adds the data to the firestore DB by another DB operation.
Here is the firebase function, with parts removed for brevity:
module.exports = functions.https.onCall(async (obj, context) => {
// To isolate where the problem is
const wait = (ms: number) => new Promise(resolve => setTimeout(resolve, ms))
try {
const photo = getPhoto(obj)
// Query to look for duplicates
const query = db
.collection(`/Users/${context.auth.uid}/Photos`)
.where('date', '==', photo.date)
.where('order', '==', photo.order)
.limit(1)
await wait(300)
log.info('Before query')
const querySnap = await query.get()
log.info('After Query')
await wait(300)
// And then the rest of the code, removed for brevity
} catch (error) {
throw new functions.https.HttpsError('internal', error.message)
}
})
I inserted a pause before and after the const querySnap = await query.get() to show that it really is this invocation that causes the error message.
I also set the firestore logger to output its internal logging to help debug the issue, by doing this:
import * as admin from 'firebase-admin'
admin.initializeApp()
admin.firestore.setLogFunction(log => {
console.log(log)
})
So the more complete log output I get is this: (read it bottom to top)
12:50:10.087 pm: After Query
12:50:10.087 pm: Firestore (2.3.0) 2019-09-13T19:50:10.087Z RTQ7I [Firestore._initializeStream]: Received stream end
12:50:10.084 pm: Firestore (2.3.0) 2019-09-13T19:50:10.084Z RTQ7I [Firestore._initializeStream]: Releasing stream
12:50:10.084 pm: Firestore (2.3.0) 2019-09-13T19:50:10.084Z RTQ7I [Firestore.readStream]: Received response: {"document":null,"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1568404210","nanos":76771000},"skippedResults":0}
12:50:10.026 pm: (node:2) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
12:50:10.020 pm: Firestore (2.3.0) 2019-09-13T19:50:10.020Z RTQ7I [Firestore.readStream]: Sending request: {"parent":"[redacted]/documents/Users/SpQ3wTsFzofj6wcsF7efRrSMrtV2","structuredQuery":{"from":[{"collectionId":"Photos"}],"where":{"compositeFilter":{"op":"AND","filters":[{"fieldFilter":{"field":{"fieldPath":"date"},"op":"EQUAL","value":{"stringValue":"2019-06-26"}}},{"fieldFilter":{"field":{"fieldPath":"order"},"op":"EQUAL","value":{"integerValue":0}}}]}},"limit":{"value":1}}}
12:50:10.019 pm: Firestore (2.3.0) 2019-09-13T19:50:10.019Z RTQ7I [ClientPool.acquire]: Re-using existing client with 100 remaining operations
12:50:10.012 pm: Before query
The interesting thing is that I usually run these imports in batches of 10. I seem to only get the error during the first batch of 10. If I then quickly run more batches, I don't seem to get the error again. But if I wait some time, it returns. Also, it is not consistent in which invocation within a batch the error occurs. It may be the 9th or 2nd or invocation, or any other.
Finally, the error doesn't stop execution. In fact, the imports seem to never fail. But, I don't like have unaccounted for errors in my logs! I won't be able to sleep at night with them there. :-)
I'm grateful for any help you can offer.
I got a useful response from the Firebase support team. They told me to try to install the latest version of firebase-admin (which upgraded it from 8.5.0 to 8.6.0) and that resolved the issue, even without the workaround of installing grpc. So, I think this should be the correct answer now.
Looks like this bug MaxListenersExceededWarning: Possible EventEmitter memory leak detected #694 might the problem here.
Workaround is to use npm install #grpc/grpc-js#0.5.2 --save-exact until bug is fixed and the Firestore library starts using it.