I'm puzzled at what we see when running this setup:
FuncA: Google Cloud Function trigger-http
FuncB: Google Cloud Function trigger-topic
FuncA is called by a HTTP client. Upon being called, FuncA does some light work setting up a JSON object describing a task to perform, and stores this JSON into Google Cloud Storage. When the file has been written FuncA published a topic with a pointer to the gs file. At this point FuncA responds to the client and exits. Duration typically 1-2 seconds.
FuncB is informed that a topic has been published and is invoked. The task JSON is picked up and work beings. After processing FuncB stores the result info Firebase Realtime Database. FuncB exits at this point. Duration typically 10-20 seconds.
As FuncA and FuncB are in no way associated, live their individual process lifecycles on different function names (and triggers) and only share communication through pub/sub topic message passing (one-direction from A to B) we would expect that FuncA can run again and again, publishing topics at any rate. FuncB should be triggered and fan-out to scale with what ever pace FuncA is called with.
This is however not what happens.
In the logs we see results following this pattern:
10:00:00.000 FuncA: Function execution started
10:00:02.000 FuncA: Function execution took 2000 ms, finished with status: 'ok'
10:00:02.500 FuncB: Function execution started
10:00:17.500 FuncB: Function execution took 15000 ms, finished with status: 'ok'
10:00:18.000 FuncA: Function execution started
10:00:20.000 FuncA: Function execution took 2000 ms, finished with status: 'ok'
10:00:20.500 FuncB: Function execution started
10:00:35.500 FuncB: Function execution took 15000 ms, finished with status: 'ok'
...
The client calling FuncA clearly gets to wait for both FuncA and FuncB to finish, before being let through with the next request. It is expected that FuncA would finish, and allow a new call in immediately at what ever pace the calling client can "throw at it".
Beefing the client up with more threads only repeats this pattern, such that "paired" calls to FuncA->FuncB always waits for each other.
Dicsuss, clarify, ... stackoverflow, do your magic! :-)
Thanks in advance.
Related
I want to know what happens if connection becomes nothing while executing some tasks for a Cloud Function.
cron-job.org says this:
You should design your scripts in a way that they send as little data as possible, ideally just a short status message at the end of the execution, e.g. "OK" — or simply nothing. In case your script is written in PHP and needs more than 30 seconds of run-time, you can use the following trick to let it continue to execute in the background: Use the PHP function ignore_user_abort(true) to tell PHP to continue the script execution after disconnection.
Let's say doing task like query through database with certain condition and delete matched data.
If there are too many data and not finish execution of the task within 30 seconds, what will happen?
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
We have a cloud function set up with pub/sub triggers.
The function is invoked topic(NAME).onPublish()
If the function is invoked when it is cold, it always runs twice.
Function execution took 284 ms, finished with status: 'ok' METHOD_NAME METHOD_ID
Received message from pub sub METHOD_NAME METHOD_ID
Function execution started METHOD_NAME METHOD_ID
Function execution took 24271 ms, finished with status: 'ok' METHOD_NAME METHOD_ID
Received message from pub sub METHOD_NAME METHOD_ID
Function execution started METHOD_NAME METHOD_ID
After that all future messages only run once, until the function goes cold again.
Is this because it takes a long time for the first invocation to complete and the timeout causes it to be run again? Any way to prevent this?
Startup time is almost for sure the issue. To verify this, try these:
comment out portion of the function until fast, to see if the problem goes away (time it in your local terminal, if you can, using timeit module)
increasing Acknowledgement Deadline seconds (upon subscription); defaults to 10 so could easily be the problem; try 20, 40 etc
ensure that the first time run, the function takes less time than the Timeout value of the function (defaults to 60 seconds - not likely to be the problem)
I read here endpoint spin-up is supposed to be transparent, which I assume means cold start times should not differ from regular execution times. Is this still the case? We are getting extremely slow and unusable cold start times - around 16 seconds - across all endpoints.
Cold start:
Function execution took 16172 ms, finished with status code: 200
After:Function execution took 1002 ms, finished with status code: 304
Is this expected behaviour and what could be causing it?
UPDATE: The cold start times seem to no longer be an issue with node 8, at least for me. I'll leave my answer below for any individuals curious about keeping their functions warm with a cron task via App Engine. However, there is also a new cron method available that may keep them warm more easily. See the firebase blog for more details about cron and Firebase.
My cold start times have been ridiculous, to the point where the browser will timeout waiting for a request. (like if it's waiting for a Firestore API to complete).
Example
A function that creates a new user account (auth.user().onCreate trigger), then sets up a user profile in firestore.
First Start After Deploy: consistently between 30 and 60 seconds, frequently gives me a "connection error" on the first try when cold (this is after waiting several seconds once Firebase CLI says "Deploy Complete!"
Cold Start: 10 - 20 seconds
When Warm: All of this completes in approximately 400ms.
As you can imagine, not many users will sit around waiting more than a few seconds for an account to be setup. I can't just let this happen in the background either, because it's part of an application process that needs a profile setup to store input data.
My solution was to add "ping" function to all of my API's, and create a cron-like scheduler task to ping each of my functions every minute, using app engine.
Ensure the ping function does something, like access a firestore document, or setup a new user account, and not just respond to the http request.
See this tutorial for app engine scheduling: https://cloud.google.com/appengine/docs/flexible/nodejs/scheduling-jobs-with-cron-yaml
Well it is about resource usage of Cloud Functions I guess, I was there too. While your functions are idle, Cloud Functions also releases its resources, at first call it reassignes those resources and at second call you are fine. I cannot say it is good or not, but that is the case.
if you try to return a value from an async function there won't be any variables in the main function definition (functions.https.onCall) and GCP will think that the function has finished and try to remove resources from it.
Previous breaking code (taking 16 + seconds):
functions.https.onCall((data, context) => {
return asyncFunction()
});
After returning a promise in the function definition the function times are a lot faster (milliseconds) as the function waits for the promise to resolve before removing resources.
functions.https.onCall((data, context) => {
return new Promise((resolve) => {
asyncFunction()
.then((message) => {
resolve(message);
}).catch((error) => {
resolve(error);
});
});
});
I am working with:
let callTheAPI = async {
printfn "\t\t\tMAKING REQUEST at %s..." (System.DateTime.Now.ToString("yyyy-MM-ddTHH:mm:ss"))
let! response = Http.AsyncRequestStream(url,query,headers,httpMethod,requestBody)
printfn "\t\t\t\tREQUEST MADE."
}
And
let cts = new System.Threading.CancellationTokenSource()
let timeout = 1000*60*4//4 minutes (4 mins no grace)
cts.CancelAfter(timeout)
Async.RunSynchronously(callTheAPI,timeout,cts.Token)
use respStrm = response.ResponseStream
respStrm.Flush()
writeLinesTo output (responseLines respStrm)
To call a web API (REST) and the let! response = Http.AsyncRequestStream(url,query,headers,httpMethod,requestBody) just hangs on certain queries. Ones that take a long time (>4 minutes) particularly. This is why I have made it Async and put a 4 minute timeout. (I collect the calls that timeout and make them with smaller time range parameters).
I started Http.RequestStream from FSharp.Data first, but I couldn't add a timeout to this so the script would just 'hang'.
I have looked at the API's IIS server and the application pool Worker Process active requests in IIS manager and I can see the requests come in and go again. They then 'vanish' and the F# script hangs. I can't find an error message anywhere on the script side or server side.
I included the Flush() and removed the timeout and it still hung. (Removing the Async in the process)
Additional:
Successful calls are made. Failed calls can be followed by successful calls. However, it seems to get to a point where all the calls time out and the do so without even reaching the server any more. (Worker Process Active Requests doesn't show the query)
Update:
I made the Fsx script output the queries and ran them through IRM with now issues (I have timeout and it never locks up). I have a suspicion that there is an issue with FSharp.Data.Http.
Async.RunSynchronously blocks. Read the remarks section in the docs: RunSynchronously. Instead, use Async.AwaitTask.