The code below shows the minimum example where we see the bug. As you can see, the fanout test/channels/sameKey/chats/${key} while the transaction updates test/user_phone_numbers/${key}.
If I'm understanding transaction and update correctly, these two don't overlap so it should be safe to run concurrently. However, as soon as two concurrent requests come in, Firebug errors out with [Error: set].
'use strict';
const express = require('express');
const server = express();
const firebaseRootRef = new Firebase(process.env.FIREBASE_URL)
const random = max => Math.floor(Math.random() * max)
let key = 0
const nextKey = () => ++key % 2
const fanout = key => {
const fanout = {
[`test/channels/sameKey/chats/${key}`]: random(1000)
}
return firebaseComponent.update(firebaseRootRef, fanout)
}
const transaction1 = key => firebaseRootRef.child('test/user_phone_numbers/' + key)
.transaction(_userId => !_userId ? random(100000) : undefined)
server.get('/', (req, res) =>
transaction1(nextKey())
.then(() => fanout(key))
.then(() => res.send(200))
.catch(e => {console.log(e); res.send(501)})
)
server.listen(3001, function () {
console.log('incoming.controller listening on port 3001!');
});
The apache benchmark command to replicate:
ab -n 1000 -c 2 -r http://localhost:3001/
This isn't a bug, it's working as intended. I ran into this several months back and here's the official response they gave me (emphasis mine).
The issue here are transactions in combination with update calls.
We'll abort any transactions at, below or above the path in any set or
update call. So while the transaction is technically unaffected by
your update call at /venues/1, we still go ahead and cancel the
transaction. We know this is not optimal and we're looking into
improving this with a future release. One workaround is to defer the
update calls until the transactions have completed, or keep the
data in an entirely different subtree. The simplest workaround
might be to move all the writes in the update call into separate set
calls, which will not abort the transaction.
Related
In short: Is there some kind of cold start when connecting to Firestore directly from Client SDK
Hey. I'm using Firestore client sdk in Andoid and IOS application through #react-native-firebase.
Everything works perfectly but I have noticed weird behavior I haven't found explanation.
I have made logging to see how long it takes from user login to retrieve uid corresponding data from Firestore and this time has been ~0.4-0.6s. This is basically the whole onAuthStateChanged workflow.
let userLoggedIn: Date;
let userDataReceived: Date;
auth().onAuthStateChanged(async (user) => {
userLoggedIn = new Date();
const eventsRetrieved = async (data: UserInformation) => {
userDataReceived = new Date();
getDataDuration = `Get data duration: ${(
(userDataReceived.getTime() - userLoggedIn.getTime()) /
1000
).toString()}s`;
console.log(getDataDuration)
// function to check user role and to advance timing logs
onUserDataReceived(data);
};
const errorRetrieved = () => {
signOut();
authStateChanged(false);
};
let unSub: (() => void) | undefined;
if (user && user.uid) {
const userListener = () => {
return firestore()
.collection('Users')
.doc(user.uid)
.onSnapshot((querySnapshot) => {
if (querySnapshot && querySnapshot.exists) {
const data = querySnapshot.data() as UserInformation;
data.id = querySnapshot.id;
eventsRetrieved(data);
} else errorRetrieved();
});
};
unSub = userListener();
} else {
if (typeof unSub === 'function') unSub();
authStateChanged(false);
}
});
Now the problem. When I open the application ~30-50 minutes after last open the time to retrieve uid corresponding data from Firestore will be ~3-9s. What is this time and why does it happen? And after I open the application right after this time will be low again ~0.4-0-6s.
I have been experiencing this behavior for weeks. It is hard to debug as it happens only on build application (not in local environments) and only between +30min interval.
Points to notice
The listener query (which I'm using in this case, I have used also simple getDoc function) is really simple and focused on single document and all project configuration works well. Only in this time interval, which seems just like cold start, the long data retrieval duration occurs.
Firestore Rules should not be slowing the query as subsequent request are fast. Rules for 'Users' collection are as follows in pseudo code:
function checkCustomer(){
let data =
get(/databases/$(database)/documents/Users/$(request.auth.uid)).data;
return (resource.data.customerID == data.customerID);
}
match /Users/{id}{
allow read:if
checkUserRole() // Checks user is logged in and has certain customClaim
&& idComparison(request.auth.uid, id) // Checks user uid is same as document id
&& checkCustomer() // User can read user data only if data is under same customer
}
Device cache doesn't seem to affect the issue as application's cache can be cleaned and the "cold start" still occurs
Firestore can be called from another environment or just another mobile device and this "cold start" will occur to devices individually (meaning that it doesn't help if another device opened the application just before). Unlike if using Cloud Run with min instances, and if fired from any environment the next calls right after will be fast regardless the environment (web or mobile).
EDIT
I have tested this also by changing listener to simple getDoc call. Same behavior still happens on a build application. Replacing listener with:
await firestore()
.collection('Users')
.doc(user.uid)
.get()
.then(async document => {
if (document.exists) {
const data = document.data() as UserInformation;
if (data) data.id = document.id;
eventsRetrieved(data);
}
});
EDIT2
Testing further there has been now 3-15s "cold start" on first Firestore getDoc. Also in some cases the timing between app open has been only 10 minutes so the minimum 30 min benchmark does not apply anymore. I'm going to send dm to Firebase bug report team to see things further.
Since you're using React Native, I assume that the documents in the snapshot are being stored in the local cache by the Firestore SDK (as the local cache is enabled by default on native clients). And since you use an onSnapshot listener it will actually re-retrieve the results from the server if the same listener is still active after 30 minutes. From the documentation on :
If offline persistence is enabled and the listener is disconnected for more than 30 minutes (for example, if the user goes offline), you will be charged for reads as if you had issued a brand-new query.
The wording here is slightly different, but given the 30m mark you mention, I do expect that this is what you're affected by.
In the end I didn't find straight answer why this cold start appeared. I ended up changing native Client SDK to web Client SDK which works correctly first data fetch time being ~0.6s (always 0.5-1s). Package change fixed the issue for me while functions to fetch data are almost completely identical.
I'm trying to upload something to my firestore database, but if the user doesn't have an internet connection it just tries to upload it foreever withour giving me an error.
Is there a way to cancel it when I don't have a connection?
I see two options:
use react-native-netinfo to detect if there's a connection before uploading, something like
NetInfo.fetch().then(({isConnected}) => {
if (isConnected) {
doSomethingWithFirebase();
}
});
Add a timeout to make it fail, like so:
// make it a promise, if it isn't already
const firebaseResult = new Promise(resolve => {
doSomethingWithFirebase()
.then(() => resolve(true))
.catch(() => resolve(false))
})
// resolve after 30s
const timeout = new Promise(resolve => setTimeout(() => resolve(false), 30 * 1000));
const didUpload = await Promise.race([firebasePromise, timeOut]);
I'd personally go with #2, because you can show the user an error (something like "failed to upload. does your connection work?") but it depends what it's for, like if it's analytics data that you don't want them to know about #1 is good for that.
Edit: as OP pointed out, with #2 the action would take place when the connection came back online again, without notice, which may not be desired behavior. You could indicate that there's an open connection somehow, like with an "uploading" icon, and clear it when it finally resolves (with failure or success).
I have a Firebase Cloud Function that handles some user registration information and I'm finding it to be very slow given what the function is actually doing. This function is taking approximately 6-7s to complete, routinely. I've narrowed it down to this line that's taking the most time (approximately 5s):
let snapshot = await admin.firestore().collection('users').doc(context.auth.uid).get();
if (!snapshot.exists) {
throw new functions.https.HttpsError('not-found', 'User profile could not be found');
}
90% of the time, the document being fetched should exist. I would have expected this to return incredibly fast; instead it's routinely taking about 5s to return. Any thoughts on what I could possibly be doing wrong would be appreciated.
NOTE: Everything is deployed to us-central-1.
Here is a snippet of the logs for one of the profiling runs I made:
5:36:51.248 AM addMember Function execution started
5:36:52.256 AM addMember Checkpoint #1
5:36:52.256 AM addMember Checkpoint #2
5:36:57.253 AM addMember Checkpoint #3
5:36:57.254 AM addMember Checkpoint #4
5:36:57.597 AM addMember Checkpoint #5
5:36:57.982 AM addMember Checkpoint #6
5:36:58.051 AM addMember Function execution took 6804 ms, finished with status code: 200
The code snippet above is what is executed between Checkpoint #2 and Checkpoint #3; it is the only lines of code between those two statements. I've executed this function repeatedly (10 times) and the average is about 5s for that block to complete.
EDIT:
I've been able to narrow down the code to try and identify any bottlenecks. Here is what I've got (which still exhibits the slow behavior)
functions/index.js:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const addMember = require('./add-member');
admin.initializeApp();
exports.addMember = functions.https.onCall(addMember(admin));
functions/add-member.js:
const functions = require('firebase-functions');
const { generateCode} = require('../common');
module.exports = (admin) => async ({ email }, context) => {
console.log('Checkpoint #1')
const payload = {
code: generateCode(),
invited: new Date(),
joined: null
};
console.log('Checkpoint #2');
let snapshot = await admin.firestore().collection('users').doc(context.auth.uid).get();
if (!snapshot.exists) {
throw new functions.https.HttpsError('not-found', 'User profile could not be found');
}
console.log('Checkpoint #3');
return Promise.resolve(payload);
};
For what it's worth, I came across these which has been helpful. The speed of the functions isn't quite what I expect/hoped for and I'm starting to lean towards it being the CPU speed of the instance (128MB). Regardless, hopefully someone else finds these posts useful:
https://stackoverflow.com/a/47985480/541277
https://github.com/firebase/functions-samples/issues/170#issuecomment-323375462
I'm trying use Telegraf library with Firebase Functions but it's not working as I expected.
I follow these this article and instructions as appear in webhooks (as appears for express example) and webhookcallback as appear in telegraf docs.
const Telegraf = require('telegraf')
// The Cloud Functions for Firebase SDK to create Cloud Functions and setup triggers.
const functions = require('firebase-functions')
// The Firebase Admin SDK to access the Firebase Realtime or Firestore Database.
const admin = require('firebase-admin')
// set telegraf and responses.
const BOT_TOKEN = 'my-telegram-bot-token'
const bot = new Telegraf(BOT_TOKEN)
bot.start((ctx) => ctx.reply("Start instructions"))
bot.help((ctx) => ctx.reply("This is help"))
bot.hears('hi', (ctx) => ctx.reply('Hola'))
bot.on('text', (ctx) => ctx.reply('Response to any text'))
bot.catch((err, ctx) => {
console.log(`Ooops, ecountered an error for ${ctx.updateType}`, err)
})
// initialize bot
bot.launch() // <-- (2)
//appends middleware
exports.ideas2coolBot = functions.https.onRequest(bot.webhookCallback(`/my-path`));
In firebase server I need add bot.launch() (2) to get worked, but it works just for max timeout set in Firebase Function. I need to recall Telegram "setWebhook" API to get work again and it works for the same time. It's like it's generate one function instance and shut down when time is over.
I noted the telegraf.launch() have options to start in poll or webhook mode but its not pretty clear for me how to use this options.
How should I use telegram.launch() to get worked in webhook mode in Firebase?
Edit:
When I used getWebhookInfo I get this result:
{
"ok": true,
"result": {
"url": "https://0dbee201.ngrok.io/test-app-project/us-central1/testAppFunction/bot",
"has_custom_certificate": false,
"pending_update_count": 7,
"last_error_date": 1573053003,
"last_error_message": "Read timeout expired",
"max_connections": 40
}
}
and console shows incoming conection but do nothing...
i functions: Beginning execution of "ideas2coolBot"
i functions: Finished "ideas2coolBot" in ~1s
Edit2:
I've been trying adding Express too...
app.use(bot.webhookCallback('/bot'))
app.get('/', (req, res) => {
res.send('Hello World from Firebase!')
})
exports.ideas2coolBot = functions.https.onRequest(app);
it's works '/' path but got nothing with '/bot'. POST to '/bot' not response.
By the way, I tried a express standalone version and works prefect, but using it with firebase doesn't respond ("Read timeout expired").
delete
bot.launch()
try add this
exports.YOURFUNCTIONNAME = functions.https.onRequest(
(req, res) => bot.handleUpdate(req.body, res)
)
then set ur webhook manually
https://api.telegram.org/bot{BOTTOKEN}/setWebhook?url={FIREBASE FUNCTION URL}'
For example, this code this jsbin example:
const pingEpic = action$ =>
action$.ofType(PING)
.delay(1000) // Asynchronously wait 1000ms then continue
.mapTo({ type: PONG })
.takeUntil(action$.ofType(CANCEL));
When I use takeUntil as above, after dispatching the CANCEL action and a delay of 1 second, the action never fires again. Why?
The problem is a subtle but critical misunderstanding of how RxJS works--but fear not, this is very common.
So given your example:
const pingEpic = action$ =>
action$.ofType(PING)
.delay(1000)
.mapTo({ type: PONG })
.takeUntil(action$.ofType(CANCEL));
This epic's behavior can be described as filtering out all actions that don't match type PING. When an action matches, wait 1000ms and then map that action to a different action { type: PONG }, which will be emitted and then dispatched by redux-observable. If at any time while the app is running someone dispatches an action of type CANCEL, then unsubscribe from the source, which means this entire chain will unsubscribe, terminating the epic.
It might be helpful to see how this looks if you did it imperatively:
const pingEpic = action$ => {
return new Rx.Observable(observer => {
console.log('[pingEpic] subscribe');
let timer;
const subscription = action$.subscribe(action => {
console.log('[pingEpic] received action: ' + action.type);
// When anyone dispatches CANCEL, we stop listening entirely!
if (action.type === CANCEL) {
observer.complete();
return;
}
if (action.type === PING) {
timer = setTimeout(() => {
const output = { type: PONG };
observer.next(output);
}, 1000);
}
});
return {
unsubscribe() {
console.log('[pingEpic] unsubscribe');
clearTimeout(timer);
subscription.unsubscribe();
}
};
});
};
You can run this code with a fake store here: http://jsbin.com/zeqasih/edit?js,console
Instead, what you usually want to do is insulate the subscriber chain you want to be cancellable from the top-level chain that is suppose to listen indefinitely. Although your example (amended from the docs) is contrived, let's run through it first.
Here we use the mergeMap operator to let us take the matched action and map to another, separate observable chain.
Demo: http://jsbin.com/nofato/edit?js,output
const pingEpic = action$ =>
action$.ofType(PING)
.mergeMap(() =>
Observable.timer(1000)
.takeUntil(action$.ofType(CANCEL))
.mapTo({ type: PONG })
);
We use Observable.timer to wait 1000ms, then map the value it emits (which happens be to be the number zero, but that's not important here) to our PONG action. We also say we want to "take" from the timer source until either it completes normally or we receive an action of type CANCEL.
This isolates the chains because mergeMap will continue to subscribe to the observable you return until it errors or completes. But when that happens, it doesn't itself stop subscribing to the source you applied it to; the action$.ofType(PING) in this example.
A more real-world example is in the redux-observable docs in the Cancellation section
Here we placed the .takeUntil() after inside our .mergeMap(), but after our AJAX call; this is important because we want to cancel only the AJAX request, not stop the Epic from listening for any future actions.
const fetchUserEpic = action$ =>
action$.ofType(FETCH_USER)
.mergeMap(action =>
ajax.getJSON(`/api/users/${action.payload}`)
.map(fetchUserFulfilled)
.takeUntil(action$.ofType(FETCH_USER_CANCELLED))
);
This all may sound confusing, but like most powerful things, once you get it, it'll become intuitive. Ben Lesh does an excellent job of explaining how Observables work in his recent talk, including discussing how operators are a chain of Observables and even about isolating subscriber chains. Even though the talk is at AngularConnect, it's not Angular specific.
As an aside, it's important to note that your epics do not swallow or otherwise prevent actions from reaching the reducers, e.g. when you map an incoming action to another, different action. In fact, when your epic receives an action, it has already been through your reducers. Think of your epics as sidecar processes that listens to a stream of your apps actions, but can't prevent normal redux things from happening, it can only emit new actions.