In short: Is there some kind of cold start when connecting to Firestore directly from Client SDK
Hey. I'm using Firestore client sdk in Andoid and IOS application through #react-native-firebase.
Everything works perfectly but I have noticed weird behavior I haven't found explanation.
I have made logging to see how long it takes from user login to retrieve uid corresponding data from Firestore and this time has been ~0.4-0.6s. This is basically the whole onAuthStateChanged workflow.
let userLoggedIn: Date;
let userDataReceived: Date;
auth().onAuthStateChanged(async (user) => {
userLoggedIn = new Date();
const eventsRetrieved = async (data: UserInformation) => {
userDataReceived = new Date();
getDataDuration = `Get data duration: ${(
(userDataReceived.getTime() - userLoggedIn.getTime()) /
1000
).toString()}s`;
console.log(getDataDuration)
// function to check user role and to advance timing logs
onUserDataReceived(data);
};
const errorRetrieved = () => {
signOut();
authStateChanged(false);
};
let unSub: (() => void) | undefined;
if (user && user.uid) {
const userListener = () => {
return firestore()
.collection('Users')
.doc(user.uid)
.onSnapshot((querySnapshot) => {
if (querySnapshot && querySnapshot.exists) {
const data = querySnapshot.data() as UserInformation;
data.id = querySnapshot.id;
eventsRetrieved(data);
} else errorRetrieved();
});
};
unSub = userListener();
} else {
if (typeof unSub === 'function') unSub();
authStateChanged(false);
}
});
Now the problem. When I open the application ~30-50 minutes after last open the time to retrieve uid corresponding data from Firestore will be ~3-9s. What is this time and why does it happen? And after I open the application right after this time will be low again ~0.4-0-6s.
I have been experiencing this behavior for weeks. It is hard to debug as it happens only on build application (not in local environments) and only between +30min interval.
Points to notice
The listener query (which I'm using in this case, I have used also simple getDoc function) is really simple and focused on single document and all project configuration works well. Only in this time interval, which seems just like cold start, the long data retrieval duration occurs.
Firestore Rules should not be slowing the query as subsequent request are fast. Rules for 'Users' collection are as follows in pseudo code:
function checkCustomer(){
let data =
get(/databases/$(database)/documents/Users/$(request.auth.uid)).data;
return (resource.data.customerID == data.customerID);
}
match /Users/{id}{
allow read:if
checkUserRole() // Checks user is logged in and has certain customClaim
&& idComparison(request.auth.uid, id) // Checks user uid is same as document id
&& checkCustomer() // User can read user data only if data is under same customer
}
Device cache doesn't seem to affect the issue as application's cache can be cleaned and the "cold start" still occurs
Firestore can be called from another environment or just another mobile device and this "cold start" will occur to devices individually (meaning that it doesn't help if another device opened the application just before). Unlike if using Cloud Run with min instances, and if fired from any environment the next calls right after will be fast regardless the environment (web or mobile).
EDIT
I have tested this also by changing listener to simple getDoc call. Same behavior still happens on a build application. Replacing listener with:
await firestore()
.collection('Users')
.doc(user.uid)
.get()
.then(async document => {
if (document.exists) {
const data = document.data() as UserInformation;
if (data) data.id = document.id;
eventsRetrieved(data);
}
});
EDIT2
Testing further there has been now 3-15s "cold start" on first Firestore getDoc. Also in some cases the timing between app open has been only 10 minutes so the minimum 30 min benchmark does not apply anymore. I'm going to send dm to Firebase bug report team to see things further.
Since you're using React Native, I assume that the documents in the snapshot are being stored in the local cache by the Firestore SDK (as the local cache is enabled by default on native clients). And since you use an onSnapshot listener it will actually re-retrieve the results from the server if the same listener is still active after 30 minutes. From the documentation on :
If offline persistence is enabled and the listener is disconnected for more than 30 minutes (for example, if the user goes offline), you will be charged for reads as if you had issued a brand-new query.
The wording here is slightly different, but given the 30m mark you mention, I do expect that this is what you're affected by.
In the end I didn't find straight answer why this cold start appeared. I ended up changing native Client SDK to web Client SDK which works correctly first data fetch time being ~0.6s (always 0.5-1s). Package change fixed the issue for me while functions to fetch data are almost completely identical.
Related
I have a marketplace web app running on Firebase Hosting, with backend on Firestore, and images stored on Cloud Storage. The load time for this is a really poor approx. 9 secs. The website is pinged within a couple of seconds and key data starts loading from Realtime based on which data from Firestore and then Cloud storage is loaded. But firestore and Cloud storage data takes about 6 to 8 seconds to load??!! even when there is no cloud storage involved it gets worse
I have read about making the entire Google Cloud bucket public makes the images load in 1/3 the time, but that seems like a potential security issue for all data? I am sure I am doing something incorrectly as the platform is used by so many developers and if you cant load a website in less than 3 secs, then users generally leave.... (Mobile is worse at about 17 secs fully loaded). Performance is not much different even if I am in the same city as the hosting of the cloud
https://app.latom.in/co/markusconsulting - with images
below is the link to the gtmetrix performance to see the waterfall
[https://gtmetrix.com/reports/app.latom.in/54P1tHJE/][2]
Any guidance on what can be done better?
CODE:
Initializing:
const app = initializeApp(firebaseConfig);
const db = getFirestore (app)
const realTimeDB = getDatabase()
Query:
export const getMyCompanyServiceData = (companyId, firebaseUserID, pageLimit = 10)=>new Promise( async function(successCallback, errorCallback) {
const myDocRef = collection(db, "UserServices/"+companyId+"/Services/")
const q = query(myDocRef,
where("companyID", "==", parseInt(companyId)),
where("serviceProviderId."+firebaseUserID, "==", true),
where("status", "==", SERVICE_STATUS[0]),
limit(pageLimit)
// orderBy("serviceType")
)
try {
const docSnap = await getDocs(q)
const _data = []
docSnap.forEach((doc) => {
_data.push(doc.data())
});
successCallback(_data)
} catch (err) {
errorCallback(err)
}
});
Once the object is obtained it is rendered to get images and display data
I am trying to send user's location to Firestore even when the app is killed. I'm successfully able to send data to Firestore when the app is in the foreground or background but unable to send data when the app is killed. Can anyone explain what could be the problem?
I'm using
background_locator: ^1.6.4
Flutter
Flutter 2.2.3 • channel stable • https://github.com/flutter/flutter.git
Framework • revision f4abaa0735 (2 weeks ago) • 2021-07-01 12:46:11 -0700
Engine • revision 241c87ad80
Tools • Dart 2.13.4
Dart
Dart SDK version: 2.13.4 (stable) (Wed Jun 23 13:08:41 2021 +0200) on "linux_x64"
This code that I'm trying to execute once data is received
receiverPort.listen(
(dynamic data) async {
if (data != null) {
WidgetsFlutterBinding.ensureInitialized();
await Firebase.initializeApp();
final LocationDto position = data as LocationDto;
print('data: $position');
final Member member = await FirebaseService().getCurrentUserProfile();
member.currentPosition = '${position.latitude},${position.longitude}';
await FirebaseService()
.updateData(
FirebaseService.memberRef,
FirebaseService.memberChildId,
FirebaseService.getCurrentUserId(),
member.toJson())
.then((value) {});
} else {
print('data is null');
}
},
);
Please let me know if I need to share anything else. Thanks
I found the problem. I need to add Firestore code in the callback instead of in the ReceivePort.
Here is the working code:
Future<void> _startLocator() async {
final Map<String, dynamic> data = {'countInit': 1};
return BackgroundLocator.registerLocationUpdate(
LocationCallbackHandler.callback, // <-- You need to add your callback code here
initCallback: LocationCallbackHandler.initCallback,
initDataCallback: data,
disposeCallback: LocationCallbackHandler.disposeCallback,
iosSettings: const IOSSettings(
accuracy: location_settings.LocationAccuracy.BALANCED,),
androidSettings: AndroidSettings(
accuracy: location_settings.LocationAccuracy.BALANCED,
interval: 5,
androidNotificationSettings: AndroidNotificationSettings(
notificationChannelName: 'Location tracking',
notificationTitle:
location.isNotEmpty ? location : 'Start Location Tracking',
notificationMsg: 'Track location in background',
notificationBigMsg:
'Background location is on to keep the app up-to-date with your location. This is required for main features to work properly when the app is not running.',
notificationIconColor: Colors.grey,
notificationTapCallback:
LocationCallbackHandler.notificationCallback)));
}
Future<void> callback(LocationDto locationDto) async {
// Your call back code goes here
}
When the app's process is killed, all of its code stops executing. This includes any Firestore listeners. The OS does this in order to prevent an app from unexpectedly burning battery and data when the user isn't using it. This is a feature intended for the benefit of the end user, and your app should respect that.
See also:
Flutter: cross-platform way to keep application running in the background
How to keep my flutter app running in the background when close?
How do I run code in the background, even with the screen off?
Inside the Firebase Console, under the Cloud Messaging view, users are able to create test notifications. This functionality also allows you to schedule the time at which the notification will send to a device or set of devices.
Is it possible to create and send scheduled FCM notifications to specific devices by using firebase cloud functions and the Firebase Admin SDK? Is there an alternative way to solving this?
The current way that I send scheduled messages to users is like so:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const schedule = require('node-schedule');
admin.initializeApp();
exports.setScheduledNotification = functions.https.onRequest(async (req, res) => {
const key = req.query.notification_key;
const message = {
notification: {
title: 'Test Notification',
body: 'Test Notification body.'
}
};
var currentDate = new Date();
var laterDate = new Date(currentDate.getTime() + (1 * 60000));
var job = schedule.scheduleJob(key, laterDate, () => {
const snapshot = admin.messaging().sendToDevice(key, message);
});
return res.status(200).send(`Message has been scheduled.`);
});
First of all, I am unsure how node-schedule interacts with firebase cloud functions. The logs appear that the function terminates very quickly which I would think would be correct. The longer the operation runs the more costly it is in our firebase bills. The notification does still run on a scheduled time though. I'm confused on how that all is working behind the scenes.
Secondly, I am having issues canceling these scheduled notifications. The notifications will most likely be on a 2hr timed schedule from the point it gets created. Before the 2hrs is up, I'd like the have the ability to cancel/overwrite the notification with an updated scheduled time.
I tried this to cancel the notification and it failed to find the previously created notification. Here is the code for that:
exports.cancelScheduledNotification = functions.https.onRequest(async (req, res) => {
const key = req.query.notification_key;
var job = schedule.scheduledJobs[key];
job.cancel();
return res.status(200).send(`Message has been canceled.`);
});
Is it possible to tap into the scheduling functionality of firebase cloud messaging outside of the firebase console? Or am I stuck with hacking my way around this issue?
A Cloud Function can run for a maximum of 9 minutes. So unless you're using node-schedule for periods shorter than that, your current approach won't work. Even if it would work, or if you are scheduling for less than 9 minutes in advance, using this approach is very uneconomic as you'll be paying for the Cloud Functions for all this time while it's waiting.
A more common approach is to store information about what message you want to be delivered to whom at what time in a database, and then use regular scheduled functions to periodically check what messages to send. For more on this, see these previous questions:
Firebase scheduled notification in android
How to schedule push notifcations for react native expo?
Schedule jobs in Firebase
Ionic: Is it possible to delay incoming push FCM push notification from showing to my device until a specific time
Cloud Functions for Firebase trigger on time?
How to create cron jobs dynamically in firebase
A recent improvement on this is to use the Cloud Tasks API to programmatically schedule Cloud Functions to be called at a specific time with a specific payload, and then use that to send the message through FCM. Doug Stevenson wrote a great blog post about this here: How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL). While the post is about deleting documents at a certain time, you can combine it with the previous approach to schedule FCM messages too.
Scheduling of tasks is now also described in the documentation on enqueueing functions with Cloud Tasks
A final option, and one I'd actually recommend nowadays, is to separate the delivery of the message from the display of the notification.
Display of data messages (unlike notification messages) is never handled by the system, and always left to your application. So you can deliver the FCM data message straight away that then contains the time to display the message, and then wake the device up to display the message (often called a local notification) at that time.
To make Frank's answer more tangible, I am including some sample code below for scheduled cloud functions, that can help you achieve the 'scheduled FCM notifications'.
You should store the information required to send your notification(s) in Firestore (e.g. the when-to-notify parameter and the FCM token(s) of the users you want to send the notification to) and run a cloud function every minute to evaluate if there is any notification that needs to be delivered.
The function checks what Firestore documents have a WhenToNofity parameter that is due, and send the notifications to the receiver tokens immediately. Once sent, the function sets the boolean 'notificationSent' to true, to avoid that the users receive the same notification again on the next iteration.
The code below achieves just that:
const admin = require('firebase-admin');
admin.initializeApp();
const database = admin.firestore();
exports.sendNotification = functions.pubsub.schedule('* * * * *').onRun(async (context) => {
//check whether notification should be sent
//send it if yes
const query = await database.collection("experiences")
.where("whenToNotify", '<=', admin.firestore.Timestamp.now())
.where("notificationSent", "==", false).get();
query.forEach(async snapshot => {
sendNotification(snapshot.data().tokens);
await database.doc('experiences/' + snapshot.id).update({
"notificationSent": true,
});
});
function sendNotification(tokens) {
let title = "INSERT YOUR TITLE";
let body = "INSERT YOUR BODY";
const message = {
notification: { title: title, body: body},
tokens: tokens,
android: {
notification: {
sound: "default"
}
},
apns: {
payload: {
aps: {
sound: "default"
}
}
}
};
admin.messaging().sendMulticast(message).then(response => {
return console.log("Successful Message Sent");
}).catch(error => {
console.log(error);
return console.log("Error Sending Message");
});
}
return console.log('End Of Function');
});
If you're unfamiliar with setting up cloud functions, you can check how to set them up here. Cloud functions require a billing account, but you get 1M cloud invocations per month for free, which is more than enough to cover the costs of this approach.
Once done, you can insert your function in the index.js file.
'use strict';
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const {WebhookClient} = require('dialogflow-fulfillment');
process.env.DEBUG = 'dialogflow:*'; // enables lib debugging statements
admin.initializeApp({
credential: admin.credential.applicationDefault(),
databaseURL: "https://my_db.firebaseio.com/",
});
var database = admin.database();
var transition = database.ref('/stage');
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
console.log('Inside :) yessssssss !');
const agent = new WebhookClient({ request, response });
function moveToStage (agent) {
transition.set('2');
agent.add('Welcome to xx console. Please accept the notification on your watch');
}
transition.on('value', (snapshot) => {
console.log("Reading value succesfully from firebase");
console.log(snapshot.val());
if(snapshot.val() == '3'){
agent.add('Thank you for granting me the access.');
// OR
// response.setHeader('Content-Type','applicaiton/json');
// response.send(JSON.stringify({"fulfillmentText": 'Thank you for granting me the access.'}));
}
});
let intentMap = new Map();
intentMap.set('welcome_and_ask_to_sync', moveToStage);
agent.handleRequest(intentMap);
});
I have an intent welcome_and_ask_to_sync, which has webhook activated.
When that Intent is fired by a successful voice input, it reponds with a text/voice from the agent and updates a field stage in the respective firebase DB.
Now another external application, under some circumstences, updates that stage field in the firebase DB.
No this this part in the fulfillment code, wtahces that change
transition.on('value', (snapshot) => {
console.log("Reading value succesfully from firebase");
console.log(snapshot.val());
if(snapshot.val() == '3'){
agent.add('Thank you for granting me the access.');
// OR
// response.setHeader('Content-Type','applicaiton/json');
// response.send(JSON.stringify({"fulfillmentText": 'Thank you for granting me the access.'}));
}
});
The intention here is to then make google home speak something, like in thsi case Thank you for granting me the access. .
NOTE: I do not need an intent to be fired (sorry for the confusion earlier). I just need google home voice agent to acknowledge this change/trigger.
Now when I watch the logs, I see it breaks here agent.add('Thank you for granting me the access.');
And the err log si somewhat like:
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:356:11)
at transition.on (/user_code/index.js:36:22)
at /user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:4465:22
at exceptionGuard (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:691:9)
at EventList.raise (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:9727:17)
at EventQueue.raiseQueuedEventsMatchingPredicate_ (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:9681:41)
at EventQueue.raiseEventsForChangedPath (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:9665:14)
at Repo.onDataUpdate_ (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:12770:26)
at PersistentConnection.onDataPush_ (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:12070:18)
at PersistentConnection.onDataMessage_ (/user_code/node_modules/firebase-admin/node_modules/#firebase/database/dist/index.node.cjs.js:12064:18)
So the basic question remains: How can I make the agent speak/update text response and acknowledge on that DB's field change.
The short answer is that you can't - certainly not the way you're doing it.
Once a response is sent from your webhook back to Dialogflow, the HTTPS connection is closed, and any further replies will generate the error that you see.
Furthermore, the conversational model used by AoG and Dialogflow is that the user must always initiate each round of the conversation. It is not possible for AoG to "announce" something at this point. That would be considered somewhat invasive.
You can send a notification through Actions on Google, which would re-start the conversation when the user acknowledged the notification. However, notifications are only sent to smartphones - not to speakers. So this may not meet your needs.
If you're expecting just a few updates that take place fairly quickly after the initial send, you may want to see if you can use a Media Response to keep the conversation going with "hold music" while you wait for more updates. When the "hold music" ends, it will send an event to your Action, and you can either re-start the hold music or reply. In this case, you wouldn't use the .on() method for updates to come in, but would have to check each time the Media finishes playing to see if there have been updates that are unsent.
I'm able to authorize the Firebase app from my existing Electron app using firebase.auth().signInWithCustomToken. The promise for this method resolves and I'm able to obtain the current authorized user with firebase.auth().currentUser.uid.
At this point I must technically be able to write to /users/<currentUser>. However calling the userRef.set() and userRef.update() methods does not update the database reference and fails silently (both the callback and the promise from these methods do not resolve and there is no error thrown).
What is strange is that the exact same code works in a different, newly created Electron app. My code looks like below:
const writeToFirebase = (customToken) => {
syncApp.auth().signInWithCustomToken(customToken).then(user => {
const userId = firebase.auth().currentUser.uid; // this is successfull
const userRef = firebase.database().ref("/users/" + userId);
userRef.set({data: data}, () => { //callback does not trigger });
userRef.update({data: data})
.then(() => {//promise does not resolve})
.catch(err) => {// promise is not rejected either! }
});
}
Any pointers on how to go about debugging this would be helpful.
I discovered the problem. It's unlikely anybody else would have the same issue, but if you do, take a look at the userAgent value in your browserWindow.loadURL in Electron.
Mine was set to an Android mobile device & Firebase was not setting/updating due to this reason. I presume the Firebase server reacts differently when it sees a mobile userAgent and I was using the Firebase JS SDK and not the Android SDK which caused the issue.