I have many v1 cloud functions. Written in typescript and running on NodeJS 18. Deployed with "firebase deploy functions". Completely server-less.
If I open Cloud Trace ( https://console.cloud.google.com/traces ), I find all my functions, but there are no traces, just one single long bar (yes of course, I am not collecting them...)
Now, my question is, can I add tracing mechanism to my functions so to see detailed traces? How would I go about that?
I found this guide, https://cloud.google.com/trace/docs/setup/nodejs-ot but it is either for Compute Engine or Google Kubernetes Engine. And Firebase Functions are neither of these...
Also, I know that sometimes the function instance is disposed, some other time no, like google try to reuse resources (indeed the second call to one of my function, within a 1 minute, is waaaaaaaay faster than the initial call). In that case, wouldn't this "recycling" be an issue for the sake of traces?
Edit: what I did (thanks to Doug for your comment).
I haven't set up anything at all for tracing, my question is to know if it is possible and how to do it.
The only documentation I found, as said before is ( https://cloud.google.com/trace/docs/setup/nodejs-ot ) and to me it seems that said doc does not apply to my case (I am not on Compute Engine nor on Google Kubernetes Engine) I have searched also for the documentation mentioned by Doug https://cloud.google.com/trace but I could not find a practical step by step implementation guide.
So If you want to reproduce my behaviour
firebase init a project
then write a function definition
deploy it with firebase deploy --only functions
go to trace on google cloud platform: https://console.cloud.google.com/traces
observe that the function has much as it is complex, calls different subfunctions, and makes multiple other http requests etc. has no detailed trace, and just one single span as depicted in the picture above
Related
We are using Firebase Functions with a few different HTTP functions .
One of the functions runs via a manual trigger from our website. It then pulls in a lot of data from an external resource and saves it into our Firestore database. Our function resources are Node.js 10, 1 GB of Memory and 540s before it times out.
However, when we have large datasets that we need to pull in, e.g. 5 000 - 10 000 records to write to the database, we start running into issues. We receive an error on large data sets of:
8 RESOURCE_EXHAUSTED: Bandwidth exhausted
The full error on Firebase Functions Health Dashboard logs looks like this:
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
at Object.callErrorFromStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client.js:176:52)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:342:141)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:305:181)
at Http2CallStream.outputStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:117:74)
at Http2CallStream.maybeOutputStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:156:22)
at Http2CallStream.endCall (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:142:18)
at ClientHttp2Stream.stream.on (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:420:22)
at ClientHttp2Stream.emit (events.js:198:13)
at ClientHttp2Stream.EventEmitter.emit (domain.js:466:23)
Our Firebase project is on the blaze plan and also, on GCP connected to an active billing account.
Upon inspection on GCP, it seems like we are NOT exceeding our WRITES per minute quote, as previously thought, however, we are exceeding our Cloud Build limit. We are also using batched writes when we save data to firestore from within the function, which seems to also make the amount of db writes less. e.g.
We don't use Cloud Build, so I assume that Firebase Functions uses Cloud Build in the back end to run the functions or something, but I can't find any documentation on the matter. We also have a few firestore database functions that run when documents are created. Not sure if that uses Cloud build in the back end or not.
Any idea why this would happen ? Whenever this happens, our function gets terminated with that error which causes us to only import half of our data. The data import works flawlessly with smaller amounts of data.
See our usage here for this particular project:
Cloud Build is used during the deployment of Cloud Functions. If you check this documentation you can see that:
Deployments work by uploading an archive containing your function's source code to a Google Cloud Storage bucket. Once the source code has been uploaded, Cloud Build automatically builds your code into a container image and pushes that image to Container Registry. Cloud Functions uses that image to create the container that executes your function.
This by itself is not enough to justify the charges you are seeing, but if you check the container image documentation it says:
Because the entire build process takes place within the context of your project, the project is subject to the pricing of the included resources:
For Cloud Build pricing, see the Pricing page. This process uses the default instance size of Cloud Build, as these instances are pre-warmed and are available more quickly. Cloud Build does provide a free tier: please review the pricing document for further details.
So with that information in mind, I would make an educated guess that your website is triggering the HTTP function enough times to make Cloud Functions scale up this particular function with new intances of it, which triggers a build process for the container that hosts the function and charges you as a Cloud Build charge. So to keep doing what you doing you are going to have to increase your Cloud Build Quota to meet this demand of your website.
There was a Firestore trigger that was triggering on new records of the same type I was importing.
So in short, I was creating thousands of records in a collection, and for every one of those, the firestore rule (function) triggered, but what I did not know at the time, is that it created a new build process in the background for each firestore trigger that ran, which is not documented anywhere.
Does anyone know if there is an easy way to trigger a function everytime i re-deploy some funciont to firebase?
I have an specific firabase functions which i define inside GCP (this way when i do "firebase deploy" it doesnt re-deploy, unnisntal or touch in any form my current function)
but sometimes i might update this function manually on GCP and i would like to trigger a inner function of its code everytime it happens... is it possible?
ex:
exports.decrementAction = (req, res) => {/*do stuff*/res.status(200).send("ok")};
function auxiliary(){
//to be called on re-deploy
}
Unfortunately, there isn't an easy way for you to trigger a function within a code that is being redeployed. Since this code is only being deployed at the moment, this wouldn't be possible to be done automatically.
The alternative would be to have this function separately from the "root" function in the moment of deploying and use triggers to run this other Cloud Function, when the first is redeployed. This way, it would be possible to run it based in the deployment of the other.
You can get more information on the triggers available for Cloud Functions here: Calling Cloud Functions. With them, you should be able to configure the timing for the execution.
Besides that, it might be worth it to raise a Feature Request for Google's to verify the possibility of adding this in future releases.
Let me know if the information clarified!
I think there exists a manner.
With Pub/Sub you can catch logs from Stackdriver (docs). Those services allow you to store only the logs related to the deployment of a Cloud Function.
The store could be, for instance, Cloud Firestore. As you should know, there is available a trigger for Cloud Firestore events.
Finally, every time an event log related to a function's deployment is generated, it will be stored and triggers a function attached to that event. In the function, you can parse or filter the logs.
I am currently working on a Firebase function. As I am still in the development stage, I have a number of logs in my code to see what's going on.
async function getAddressByIdAsync(address) {
let addr = addressesRef.child(address);
console.log(addr);
return admin.database().ref('Addresses/' + address);
}
The only way I have found to be able to see these logs in real time is though the Firebase console, which is very slow and generally a bad UI experience IMO.
I'm looking for a command line solution that will let me see in real time the logs coming from the Firebase cloud function.
I have tried this command
firebase functions:log
Which appears to return the last twenty or so log entries into my Firebase Functions.
I know Google Cloud has a tail option on the end of that CLI but it doesn't work here.
firebase functions:log tail
Error: Too many arguments. Run firebase help functions:log for usage instructions
Is there a way to get a live running output of the logs from firebase cloud functions?
You can try the local emulator, that way you can first run your functions locally (and log in your terminal), before deploying.
https://firebase.google.com/docs/functions/local-emulator
I need to connect monitoring and tracing tools for our application. Our main code is on Express 4 running on Google Cloud Functions. All requests incoming from front nginx proxy server that handle domain and pretty routes names. Unfortunately, trace agent traces this requests, that coming on nginx front proxy without any additional information, and this is not enough to collect useful information about app. I found the Stack Driver custom API, which, as I understand might help to collect appropriate data on runtime, but I don't understand how I can connect it to Google Cloud Functions app. All other examples saying, that we must extend our startup script, but Google Cloud Functions fully automated thing, there is no such possibility here.
Found solution. I included require("#google-cloud/trace-agent"); not at the top of the index.js. It should be included before all other modules. After that it started to work.
Placing require("#google-cloud/trace-agent") as the very first import didn't work for me. I still kept getting:
ERROR:#google-cloud/trace-agent: express tracing might not work as /var/tmp/worker/node_modules/express/index.js was loaded before the trace agent was initialized.
However I managed to work around it by manually patching express:
var traceApi = require('#google-cloud/trace-agent').get();
require("#google-cloud/trace-agent/src/plugins/plugin-express")[0].patch(
require(Object.keys(require('module')._cache).find( _ => _.indexOf("express") !== -1)),
traceApi
);
I'm using Google Cloud Functions to:
Watch for a new Firebase entry
Download a file that's referenced in the Firebase entry
Generate a thumbnail based on that file.
Upload the thumbnail to the cloud bucket.
Unfortunately I'm getting ECONNRESET errors repeatedly on step 4, and the only way to fix it seems to be to redeploy the function. Any ideas how to further debug this?
Edit:
It seems like many times when this happens, when I try to deploy the function again, it errors, and I have to run the deploy twice. Is something hanging or something?
Update May 9 2017
According to this thread, the google cloud nodejs API developers have made some changes to the defaults that are used when initializing that should solve these ECONNRESET socket issues.
From #stephen++ on github GoogleCloudPlatform/google-cloud-node issue 2254:
We have disabled the forever agent by default in Cloud Function
environments. If you un- and re-install #google-cloud/storage, you
will pick up the new behavior automatically. Thanks for all of the
helpful debugging, everyone!
Older Post Follows:
The solution for me to similar ECONNRESET issues using storage on the cloud functions platform was to use npm:promise-retry, but set up your own retry strategy because the default of 10 retries is too many.
I reported an ECONNRESET issue with cloud functions to Google Support (which you might star if you are also getting ECONNRESET in this context but not in other contexts) and they replied with a "won't fix" that the behavior is expected. Google support said the socket that the API client library uses to connect times out after a few minutes, and then when your cloud function tries to use it again you get ECONNRESET. They recommended adding autoRetry:true when initializing the storage API, but that did not help.
The ECONNRESETs happen on the read side too. In both read and write cases promise-retry helps, and most of the time with only 1 retry needed to reset the bad socket.
So I wrote npm:pipe-to-storage to return a promise to do the retries, check md5, etc., but I haven't tested it with binary data, only text, so I don't know if you could use it with image files. The calls would look like this:
const fs = require('fs');
const storage = require('#google-cloud/storage')();
const pipeToStorage = require('pipe-to-storage')(storage);
const source = ()=>(fs.createReadStream("/path/to/your/file/to/upload"));
pipeToStorage(source, bucketName, fileNameInBucket).then(//do next step);
See also How do I read the contents of a new cloud storage file of type .json from within a cloud function?
You can directly report a bug to the Firebase Support team, or open a support ticket with Firebase to troubleshoot a specific issue.
You may also report a Cloud Functions specific issue in the Google Issue Tracker, which is similar to Stack Overflow in that it is accessible by the public (but specifically used for filing issue reports).