I have a Google Cloud Function and within that I called two external APIs (or URLs), using the Python requests library, one requests.get and another requests.post. Please note that these APIs are tested and working using Postman.
The requests.get is downloading an MP3 file and it is working in Cloud Functions as I was able to see the downloaded file in Cloud Storage:
download_url = "https://some.url.com/music.mp3"
resp = requests.get(download_url)
[get the resp.content, put to storage bucket]
Aside from storing in Cloud Storage, I also send this mp3 file over to Google Cloud Speech-to-Text API and get the transcribed text, which I want to post to the another URL.
[transcribe the audio using Speech-to-Text API, get the transcibed text]
data = { "download_url": download_url, "transcription": transcribed_text }
upload_api = https://another.url.com/api"
resp = requests.post(upload_api, data = data)
Likewise, the transcription is working because I print/log the text and it shows in the View Logs console of Cloud Functions. In the requests.post however, I am getting a time out, also according to the logs.
requests.post(upload_api, data = data, timeout=120)
I even lengthen the timeout setting but that still happens. What could be the explanation for this? Is there a Google Cloud configuration that I miss to set somewhere that might be causing this?
This is because GCP cloud functions has a 1 minute timeout, to fix it is necessary to declare the timeout limit when you create your function
if you are using Gcloud CLI commands to deploy your function, you need to add the flag --timeout
for example
gcloud functions deploy FUNCTION_NAME --timeout=TIMEOUT FLAGS
if you are using the GCP console (Web UI) you need to follow the steps on this link
Related
We are using Firebase Functions with a few different HTTP functions .
One of the functions runs via a manual trigger from our website. It then pulls in a lot of data from an external resource and saves it into our Firestore database. Our function resources are Node.js 10, 1 GB of Memory and 540s before it times out.
However, when we have large datasets that we need to pull in, e.g. 5 000 - 10 000 records to write to the database, we start running into issues. We receive an error on large data sets of:
8 RESOURCE_EXHAUSTED: Bandwidth exhausted
The full error on Firebase Functions Health Dashboard logs looks like this:
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
at Object.callErrorFromStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client.js:176:52)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:342:141)
at Object.onReceiveStatus (/workspace/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:305:181)
at Http2CallStream.outputStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:117:74)
at Http2CallStream.maybeOutputStatus (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:156:22)
at Http2CallStream.endCall (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:142:18)
at ClientHttp2Stream.stream.on (/workspace/node_modules/#grpc/grpc-js/build/src/call-stream.js:420:22)
at ClientHttp2Stream.emit (events.js:198:13)
at ClientHttp2Stream.EventEmitter.emit (domain.js:466:23)
Our Firebase project is on the blaze plan and also, on GCP connected to an active billing account.
Upon inspection on GCP, it seems like we are NOT exceeding our WRITES per minute quote, as previously thought, however, we are exceeding our Cloud Build limit. We are also using batched writes when we save data to firestore from within the function, which seems to also make the amount of db writes less. e.g.
We don't use Cloud Build, so I assume that Firebase Functions uses Cloud Build in the back end to run the functions or something, but I can't find any documentation on the matter. We also have a few firestore database functions that run when documents are created. Not sure if that uses Cloud build in the back end or not.
Any idea why this would happen ? Whenever this happens, our function gets terminated with that error which causes us to only import half of our data. The data import works flawlessly with smaller amounts of data.
See our usage here for this particular project:
Cloud Build is used during the deployment of Cloud Functions. If you check this documentation you can see that:
Deployments work by uploading an archive containing your function's source code to a Google Cloud Storage bucket. Once the source code has been uploaded, Cloud Build automatically builds your code into a container image and pushes that image to Container Registry. Cloud Functions uses that image to create the container that executes your function.
This by itself is not enough to justify the charges you are seeing, but if you check the container image documentation it says:
Because the entire build process takes place within the context of your project, the project is subject to the pricing of the included resources:
For Cloud Build pricing, see the Pricing page. This process uses the default instance size of Cloud Build, as these instances are pre-warmed and are available more quickly. Cloud Build does provide a free tier: please review the pricing document for further details.
So with that information in mind, I would make an educated guess that your website is triggering the HTTP function enough times to make Cloud Functions scale up this particular function with new intances of it, which triggers a build process for the container that hosts the function and charges you as a Cloud Build charge. So to keep doing what you doing you are going to have to increase your Cloud Build Quota to meet this demand of your website.
There was a Firestore trigger that was triggering on new records of the same type I was importing.
So in short, I was creating thousands of records in a collection, and for every one of those, the firestore rule (function) triggered, but what I did not know at the time, is that it created a new build process in the background for each firestore trigger that ran, which is not documented anywhere.
I need to use the Twitter Stream API to stream tweet data to my firebase cloud function like this:
client.stream('statuses/filter', params, stream => {
stream.on('data', tweet => {
console.log(tweet);
})
stream.on('error', error => {
console.log(error)
})
})
The stream is continuous but the firebase cloud function shuts down after a certain period of time. What solution could I make use of to be able to continuously receive the stream data?
Cloud Functions have a max running time of 540 seconds as documented. You would have to look at probably using a Compute Engine Instance from Google Cloud where you can have code running without limits. Or you could look at using the Google Cloud Scheduler to run your function every x time to get new tweets.
The accepted response suggests running GCE and while it's certainly correct, I'd like to point out that anyone who was interested in Cloud Functions - a serverless solution - might find GAE (App Engine) much more viable for streaming data.
Our application utilises App Engine Standard as an ingestion service and it works like a charm - removing overhead required by GCE. If advanced networking features are required by your app App Engine Flexible or GKE (Kubernetes Engine) might also be something to look at!
I was hoping to trigger a Pub/Sub function (using functions.pubsub / onPublish) whenever a new Pub/Sub message is sent to a topic/subscription in a third-party project i.e. cross projects.
After some research and experimentation I found that TopicBuilder throws an error if the topic name contains a / and it defaults to "projects/" + process.env.GCLOUD_PROJECT + "/topics/" + topic (https://github.com/firebase/firebase-functions/blob/master/src/providers/pubsub.ts).
I also found a post in Stack Overflow that says that "Firebase provides a (relatively thin) wrapper around Google Cloud Functions"
(What is the difference between Cloud Function and Firebase Functions?)
This led me to look into Google Cloud Functions. Whilst I was able to create a subscription in a project I own to a topic in a third-party project - after changing permissions in IAM - I could not find a way associate a function with the topic. Nor was I successful in associating a function with a topic and subscription in a third-party project. In the console I only see the topics in my project and I had no success using gcloud.
Has anyone had any success in using a function across projects and, if so, how did you achieve this and is there a documentation URL you could provide? If a function can't be triggered by a message to a topic and subscription in a third-party project can you think of a way that I could ingest third-party Pub/Sub data?
As Pub/Sub fees are billed to the project that contains the subscription I would prefer that the subscription resides in the third-party project with the topic.
Thank you
Google Cloud Functions currently does not not allow a function to listen to a resource in another project. For Cloud Pub/Sub triggers specifically you could get around this by deploying an HTTP-function and add a Pub/Sub push subscription to the topic that you want to fire that cross-project function.
A Google Cloud Function can't be triggered by a subsription to a topic of another project (since you can't subscribe to another project's topic).
But a Google Cloud Function can publish to a topic of another project (and then subscribers of this topic will be triggered).
I solved it by establishing a Google Cloud Function in the original project which listens to the original topic and reacts with publishing to a new topic in the other project. Therefore, the service account (...#appspot.gserviceaccount.com) of this function "in the middle" needs to be authorized by the new topic (console.cloud.google.com/cloudpubsub/topic/detail/...?project=...), i.e. add principal with role: "Pub/Sub Publisher"
import base64
import json
import os
from google.cloud import pubsub_v1
#https://cloud.google.com/functions/docs/calling/pubsub?hl=en#publishing_a_message_from_within_a_function
# Instantiates a Pub/Sub client
publisher = pubsub_v1.PublisherClient()
def notify(event, context):
project_id = os.environ.get('project_id')
topic_name = os.environ.get('topic_name')
# References an existing topic
topic_path = publisher.topic_path(project_id, topic_name)
message_json = json.dumps({
'data': {'message': 'here would be the message'}, # or you can pass the message of event/context
})
message_bytes = message_json.encode('utf-8')
# Publishes a message
try:
publish_future = publisher.publish(topic_path, data=message_bytes)
publish_future.result() # Verify the publish succeeded
return 'Message published.'
except Exception as e:
print(e)
return (e, 500)
google endpoints can be a easier solution to add auth to the function http.
I have the following code in index.js file located into functions folder in my google firebase proyect:
net=require('express')()
net.get('/',function(req,res){res.sendFile(__dirname+'/slave.htm')})
exports.run=require('firebase-functions').https.onRequest(net)
require('socket.io').listen(net).on("connection",function(socket){})
But when I execute \gfp1>firebase deploy in command prompt, this give me that errors:
You are trying to attach socket.io to an express request handler function. Please, pass a http.Server instance.
Yes, and I pass and http server instance in the following code:
net=require('firebase-functions').https.onRequest((req,res)=>{res.send("socket.io working!")})
exports.run=net;require('socket.io').listen(net).on("connection",function(socket){})
It gives me again the following error:
You are trying to attach socket.io to an express request handler function. Please, pass a http.Server instance.
And I try attaching socket.io to firebase functions with that code:
net=https.onRequest((req,res)=>{res.send("socket.io working!")})
exports.run=require('firebase-functions').net
require('socket.io').listen(require('firebase-functions').net).on("connection",function(socket){})
And that gives this error:
https is not defined
When I run this code in localhost:
app=require('express')()
app.get('/',function(req,res){res.sendFile(__dirname+'/slave.htm')})
net=require('http').createServer(app);net.listen(8888,function(){console.log("Server listening.")})
require('socket.io').listen(net).on("connection",function(socket){})
The console emmit \gfp>Server listening., and when I go to url http://127.0.0.1:8888, it works sending an html file to navigator, as I expected:
<script>
document.write("File system working!")
document.body.style.backgroundColor="black"
document.body.style.color="white"
</script>
But the problem happens when I try to convert net=require('http').createServer(app);net.listen(8888,function(){console.log("Server listening.")}) to net=exports.run=require('firebase-functions').https.onRequest((req,res)=>{res.send("Firebase working!")}), it seems to be impossible.
You can't run code to listen on some port with Cloud Functions. This is because you aren't guaranteed to have a single machine or instance running your code. It could be distributed among many instances all running concurrently. You shouldn't know or care if that happens - Cloud Functions will just scale to meet the needs placed on your functions.
If you deploy an HTTP type function, it will automatically listen on the https port for the dedicated host for your project, and you can send web requests to that.
If you want to perform transactions over a persistently held socket, use the realtime database client write values in the database, then respond to those writes with a database trigger function that you write. That function can send data back to the client by writing something back to the database in a location that's being listened to by the client.
I'm using Google Cloud Functions to:
Watch for a new Firebase entry
Download a file that's referenced in the Firebase entry
Generate a thumbnail based on that file.
Upload the thumbnail to the cloud bucket.
Unfortunately I'm getting ECONNRESET errors repeatedly on step 4, and the only way to fix it seems to be to redeploy the function. Any ideas how to further debug this?
Edit:
It seems like many times when this happens, when I try to deploy the function again, it errors, and I have to run the deploy twice. Is something hanging or something?
Update May 9 2017
According to this thread, the google cloud nodejs API developers have made some changes to the defaults that are used when initializing that should solve these ECONNRESET socket issues.
From #stephen++ on github GoogleCloudPlatform/google-cloud-node issue 2254:
We have disabled the forever agent by default in Cloud Function
environments. If you un- and re-install #google-cloud/storage, you
will pick up the new behavior automatically. Thanks for all of the
helpful debugging, everyone!
Older Post Follows:
The solution for me to similar ECONNRESET issues using storage on the cloud functions platform was to use npm:promise-retry, but set up your own retry strategy because the default of 10 retries is too many.
I reported an ECONNRESET issue with cloud functions to Google Support (which you might star if you are also getting ECONNRESET in this context but not in other contexts) and they replied with a "won't fix" that the behavior is expected. Google support said the socket that the API client library uses to connect times out after a few minutes, and then when your cloud function tries to use it again you get ECONNRESET. They recommended adding autoRetry:true when initializing the storage API, but that did not help.
The ECONNRESETs happen on the read side too. In both read and write cases promise-retry helps, and most of the time with only 1 retry needed to reset the bad socket.
So I wrote npm:pipe-to-storage to return a promise to do the retries, check md5, etc., but I haven't tested it with binary data, only text, so I don't know if you could use it with image files. The calls would look like this:
const fs = require('fs');
const storage = require('#google-cloud/storage')();
const pipeToStorage = require('pipe-to-storage')(storage);
const source = ()=>(fs.createReadStream("/path/to/your/file/to/upload"));
pipeToStorage(source, bucketName, fileNameInBucket).then(//do next step);
See also How do I read the contents of a new cloud storage file of type .json from within a cloud function?
You can directly report a bug to the Firebase Support team, or open a support ticket with Firebase to troubleshoot a specific issue.
You may also report a Cloud Functions specific issue in the Google Issue Tracker, which is similar to Stack Overflow in that it is accessible by the public (but specifically used for filing issue reports).