I have a Firebase Cloud Function that handles some user registration information and I'm finding it to be very slow given what the function is actually doing. This function is taking approximately 6-7s to complete, routinely. I've narrowed it down to this line that's taking the most time (approximately 5s):
let snapshot = await admin.firestore().collection('users').doc(context.auth.uid).get();
if (!snapshot.exists) {
throw new functions.https.HttpsError('not-found', 'User profile could not be found');
}
90% of the time, the document being fetched should exist. I would have expected this to return incredibly fast; instead it's routinely taking about 5s to return. Any thoughts on what I could possibly be doing wrong would be appreciated.
NOTE: Everything is deployed to us-central-1.
Here is a snippet of the logs for one of the profiling runs I made:
5:36:51.248 AM addMember Function execution started
5:36:52.256 AM addMember Checkpoint #1
5:36:52.256 AM addMember Checkpoint #2
5:36:57.253 AM addMember Checkpoint #3
5:36:57.254 AM addMember Checkpoint #4
5:36:57.597 AM addMember Checkpoint #5
5:36:57.982 AM addMember Checkpoint #6
5:36:58.051 AM addMember Function execution took 6804 ms, finished with status code: 200
The code snippet above is what is executed between Checkpoint #2 and Checkpoint #3; it is the only lines of code between those two statements. I've executed this function repeatedly (10 times) and the average is about 5s for that block to complete.
EDIT:
I've been able to narrow down the code to try and identify any bottlenecks. Here is what I've got (which still exhibits the slow behavior)
functions/index.js:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const addMember = require('./add-member');
admin.initializeApp();
exports.addMember = functions.https.onCall(addMember(admin));
functions/add-member.js:
const functions = require('firebase-functions');
const { generateCode} = require('../common');
module.exports = (admin) => async ({ email }, context) => {
console.log('Checkpoint #1')
const payload = {
code: generateCode(),
invited: new Date(),
joined: null
};
console.log('Checkpoint #2');
let snapshot = await admin.firestore().collection('users').doc(context.auth.uid).get();
if (!snapshot.exists) {
throw new functions.https.HttpsError('not-found', 'User profile could not be found');
}
console.log('Checkpoint #3');
return Promise.resolve(payload);
};
For what it's worth, I came across these which has been helpful. The speed of the functions isn't quite what I expect/hoped for and I'm starting to lean towards it being the CPU speed of the instance (128MB). Regardless, hopefully someone else finds these posts useful:
https://stackoverflow.com/a/47985480/541277
https://github.com/firebase/functions-samples/issues/170#issuecomment-323375462
Related
I have a marketplace web app running on Firebase Hosting, with backend on Firestore, and images stored on Cloud Storage. The load time for this is a really poor approx. 9 secs. The website is pinged within a couple of seconds and key data starts loading from Realtime based on which data from Firestore and then Cloud storage is loaded. But firestore and Cloud storage data takes about 6 to 8 seconds to load??!! even when there is no cloud storage involved it gets worse
I have read about making the entire Google Cloud bucket public makes the images load in 1/3 the time, but that seems like a potential security issue for all data? I am sure I am doing something incorrectly as the platform is used by so many developers and if you cant load a website in less than 3 secs, then users generally leave.... (Mobile is worse at about 17 secs fully loaded). Performance is not much different even if I am in the same city as the hosting of the cloud
https://app.latom.in/co/markusconsulting - with images
below is the link to the gtmetrix performance to see the waterfall
[https://gtmetrix.com/reports/app.latom.in/54P1tHJE/][2]
Any guidance on what can be done better?
CODE:
Initializing:
const app = initializeApp(firebaseConfig);
const db = getFirestore (app)
const realTimeDB = getDatabase()
Query:
export const getMyCompanyServiceData = (companyId, firebaseUserID, pageLimit = 10)=>new Promise( async function(successCallback, errorCallback) {
const myDocRef = collection(db, "UserServices/"+companyId+"/Services/")
const q = query(myDocRef,
where("companyID", "==", parseInt(companyId)),
where("serviceProviderId."+firebaseUserID, "==", true),
where("status", "==", SERVICE_STATUS[0]),
limit(pageLimit)
// orderBy("serviceType")
)
try {
const docSnap = await getDocs(q)
const _data = []
docSnap.forEach((doc) => {
_data.push(doc.data())
});
successCallback(_data)
} catch (err) {
errorCallback(err)
}
});
Once the object is obtained it is rendered to get images and display data
In short: Is there some kind of cold start when connecting to Firestore directly from Client SDK
Hey. I'm using Firestore client sdk in Andoid and IOS application through #react-native-firebase.
Everything works perfectly but I have noticed weird behavior I haven't found explanation.
I have made logging to see how long it takes from user login to retrieve uid corresponding data from Firestore and this time has been ~0.4-0.6s. This is basically the whole onAuthStateChanged workflow.
let userLoggedIn: Date;
let userDataReceived: Date;
auth().onAuthStateChanged(async (user) => {
userLoggedIn = new Date();
const eventsRetrieved = async (data: UserInformation) => {
userDataReceived = new Date();
getDataDuration = `Get data duration: ${(
(userDataReceived.getTime() - userLoggedIn.getTime()) /
1000
).toString()}s`;
console.log(getDataDuration)
// function to check user role and to advance timing logs
onUserDataReceived(data);
};
const errorRetrieved = () => {
signOut();
authStateChanged(false);
};
let unSub: (() => void) | undefined;
if (user && user.uid) {
const userListener = () => {
return firestore()
.collection('Users')
.doc(user.uid)
.onSnapshot((querySnapshot) => {
if (querySnapshot && querySnapshot.exists) {
const data = querySnapshot.data() as UserInformation;
data.id = querySnapshot.id;
eventsRetrieved(data);
} else errorRetrieved();
});
};
unSub = userListener();
} else {
if (typeof unSub === 'function') unSub();
authStateChanged(false);
}
});
Now the problem. When I open the application ~30-50 minutes after last open the time to retrieve uid corresponding data from Firestore will be ~3-9s. What is this time and why does it happen? And after I open the application right after this time will be low again ~0.4-0-6s.
I have been experiencing this behavior for weeks. It is hard to debug as it happens only on build application (not in local environments) and only between +30min interval.
Points to notice
The listener query (which I'm using in this case, I have used also simple getDoc function) is really simple and focused on single document and all project configuration works well. Only in this time interval, which seems just like cold start, the long data retrieval duration occurs.
Firestore Rules should not be slowing the query as subsequent request are fast. Rules for 'Users' collection are as follows in pseudo code:
function checkCustomer(){
let data =
get(/databases/$(database)/documents/Users/$(request.auth.uid)).data;
return (resource.data.customerID == data.customerID);
}
match /Users/{id}{
allow read:if
checkUserRole() // Checks user is logged in and has certain customClaim
&& idComparison(request.auth.uid, id) // Checks user uid is same as document id
&& checkCustomer() // User can read user data only if data is under same customer
}
Device cache doesn't seem to affect the issue as application's cache can be cleaned and the "cold start" still occurs
Firestore can be called from another environment or just another mobile device and this "cold start" will occur to devices individually (meaning that it doesn't help if another device opened the application just before). Unlike if using Cloud Run with min instances, and if fired from any environment the next calls right after will be fast regardless the environment (web or mobile).
EDIT
I have tested this also by changing listener to simple getDoc call. Same behavior still happens on a build application. Replacing listener with:
await firestore()
.collection('Users')
.doc(user.uid)
.get()
.then(async document => {
if (document.exists) {
const data = document.data() as UserInformation;
if (data) data.id = document.id;
eventsRetrieved(data);
}
});
EDIT2
Testing further there has been now 3-15s "cold start" on first Firestore getDoc. Also in some cases the timing between app open has been only 10 minutes so the minimum 30 min benchmark does not apply anymore. I'm going to send dm to Firebase bug report team to see things further.
Since you're using React Native, I assume that the documents in the snapshot are being stored in the local cache by the Firestore SDK (as the local cache is enabled by default on native clients). And since you use an onSnapshot listener it will actually re-retrieve the results from the server if the same listener is still active after 30 minutes. From the documentation on :
If offline persistence is enabled and the listener is disconnected for more than 30 minutes (for example, if the user goes offline), you will be charged for reads as if you had issued a brand-new query.
The wording here is slightly different, but given the 30m mark you mention, I do expect that this is what you're affected by.
In the end I didn't find straight answer why this cold start appeared. I ended up changing native Client SDK to web Client SDK which works correctly first data fetch time being ~0.6s (always 0.5-1s). Package change fixed the issue for me while functions to fetch data are almost completely identical.
I have a (very slightly) modified version of the generateThumbnail Firebase Cloud Function found in the Firebase Github repo. The function was working correctly at one point, and now it will time out every time it is called. I haven't made any changes to the function or to the rules for my storage bucket. The rules are the default ones that check for authentication. After adding some logging I can see that it never makes it past this line:
await file.download({destination: tempLocalFile});
The image file I am testing with is a 15.21KB PNG. The timeout of the function happens after ~60000 ms (default). There is no error in the logs, only the timeout.
Any suggestions as to why it started timing out all of the sudden? Or how to debug this single call further?
Node: 14
Firebase Admin: 9.8.0
Firebase Functions: 3.14.1
EDITS
I have deployed a minimum reproducable function and am seeing the same results.
exports.newGenerateThumbnail = functions.storage.object().onFinalize(async (object) => {
const filePath = object.name;
const tempLocalFile = path.join(os.tmpdir(), filePath);
const tempLocalDir = path.dirname(tempLocalFile);
// Cloud Storage files.
const bucket = admin.storage().bucket(object.bucket);
const file = bucket.file(filePath);
functions.logger.log('Creating Temp Directory');
await mkdirp(tempLocalDir);
functions.logger.log('Temp Directory Created');
functions.logger.log('Downloading File');
await file.download({ destination: tempLocalFile });
functions.logger.log('File Downloaded');
functions.logger.log('Removing File');
fs.unlinkSync(tempLocalFile);
functions.logger.log('File Deleted');
return true;
});
The logs show this
I tried to reproduce the issue using your code, but it is working fine for me without any timeout as shown in the screenshot.
I cross checked my versions with yours, I am using Node: 16.6, Firebase: 10.2.1, Firebase Functions: 3.16
It may be because of the version as it is seen in the past as well, as shown in the stackoverflow thread. So I suggest you upgrade your version and try again, it might help you resolve the issue.
I've got a Google Cloud app with several cloud functions that call an API, then save the response in Firebase. I have this scheduled function set up to retry the API on error and it works great. But I want to retry the call if the data I need isn't there yet. Calling again right away could work if I throw an error, but it's highly unlikely that the missing data will be available seconds later, so I'd rather check once an hour after that until I have valid data.
Below is a shortened version of my function. I can imagine adding a setTimeout and having the function call itself again, but I'm not sure I would do that or if it's a good idea since it would keep the function alive for a long time. Is there a way to automatically retry this scheduled function on an arbitrary time interval?
exports.fetchData= functions.pubsub
.schedule('every Tuesday 6:00')
.timeZone('America/New_York')
.onRun(async context => {
const response = fetch(...)
.then(res => {
if (res.status < 400) {
return res;
} else {
throw new Error(`Network response was not ok. ${res}`);
}
})
.then(res => res.json());
const resObj = await response;
resObj.map(x => {
// check response for valid data
})
if (// data is valid) {
// save to Firebase
} else {
// retry in 1 hour
}
});
});
Scheduled functions only run on the schedule you specify. There is no "arbitrary" scheduling. If you think that the function might frequently fail, consider just increasing the frequency of the schedule, and bail out of function invocations that don't need to run because of recent success.
If you enable retries, and the function generates an error by throwing an exception, returning a rejected promise, or timing out, then Cloud Functions will automatically retry the function on a schedule that you can't control.
setTimeout is not a feasible option to keep a function alive for longer than its configured timeout. Cloud Functions will terminate the function and all of its ongoing work after the timeout expires (and you would be paying for the time the function sits idle, which is kind of a waste).
The code below shows the minimum example where we see the bug. As you can see, the fanout test/channels/sameKey/chats/${key} while the transaction updates test/user_phone_numbers/${key}.
If I'm understanding transaction and update correctly, these two don't overlap so it should be safe to run concurrently. However, as soon as two concurrent requests come in, Firebug errors out with [Error: set].
'use strict';
const express = require('express');
const server = express();
const firebaseRootRef = new Firebase(process.env.FIREBASE_URL)
const random = max => Math.floor(Math.random() * max)
let key = 0
const nextKey = () => ++key % 2
const fanout = key => {
const fanout = {
[`test/channels/sameKey/chats/${key}`]: random(1000)
}
return firebaseComponent.update(firebaseRootRef, fanout)
}
const transaction1 = key => firebaseRootRef.child('test/user_phone_numbers/' + key)
.transaction(_userId => !_userId ? random(100000) : undefined)
server.get('/', (req, res) =>
transaction1(nextKey())
.then(() => fanout(key))
.then(() => res.send(200))
.catch(e => {console.log(e); res.send(501)})
)
server.listen(3001, function () {
console.log('incoming.controller listening on port 3001!');
});
The apache benchmark command to replicate:
ab -n 1000 -c 2 -r http://localhost:3001/
This isn't a bug, it's working as intended. I ran into this several months back and here's the official response they gave me (emphasis mine).
The issue here are transactions in combination with update calls.
We'll abort any transactions at, below or above the path in any set or
update call. So while the transaction is technically unaffected by
your update call at /venues/1, we still go ahead and cancel the
transaction. We know this is not optimal and we're looking into
improving this with a future release. One workaround is to defer the
update calls until the transactions have completed, or keep the
data in an entirely different subtree. The simplest workaround
might be to move all the writes in the update call into separate set
calls, which will not abort the transaction.