I am working on a project with Firestore + Cloud Function where a user's "credits" balance will be updated on many occasions.
For each update to the main balance we are maintaining a separate ledger in a separate collection with the details of all the movements (date, amounts etc.).
There are several areas of the codebase that interact with the balance.
I am contemplating using a Firestore background function with retries on to take care of updating my ledger on every update to the balance mainly to avoid duplication of logic throughout the codebase.
The tradeoff however is that what could be done in one transaction now needs two:
A) Transaction to update the balance (from wherever in the codebase)
B) Another transaction from the CF to log the details to the ledger
Is this a bad idea (if the CF is configured to retry)?
Thanks,
If I understood correctly your question, you don't need to use two transactions. You can very well execute the two writes (one update and one creation) within the same Transaction, as an atomic operation.
Something along the following lines with the JS SDK:
const balanceDocRef1 = doc(db, 'balances', '....');
const historyDocRef = doc(db, 'history', '....');
try {
await runTransaction(db, async (transaction) => {
const balanceDoc = await transaction.get(balanceDocRef1);
const newBalance = balanceDoc.data().balance + ....;
transaction
.update(balanceDocRef1, { balance: newBalance })
.set(historyDocRef, { amount: '...', date: '...' });
});
console.log('Transaction successfully committed!');
} catch (e) {
console.log('Transaction failed: ', e);
}
Related
I have a document MAIN which holds an array of objects. On every change of this array i want to update related documents. So in the example below If I add something to the targets of MAIN i want to grab all documents where the field "parent" holds a reference to MAIN and then update their targets accordingly. I wanted to do this with cloud functions so the client does not have to care about updating all related documents himself. But as stated in the docs cloud-funcion triggers do not guarantee order. So if f.e. a user adds a new object to the targets of MAIN and then removes it, cloud trigger would maybe receive the remove event before the add event and thus RELATED documents would be left with inconsistent data. As stated by Doug Stevenson in this stackoverflow post this could happen, even when using transactions. Am I right so far?
const MAIN = {
ID:"MAIN"
targets:[{name: "House"}, {name:"Car"}]
}
const RELATED_1 = {
parent: "MAIN",
targets:[{name: "House"}, {name:"Car"}]
}
const RELATED_2 = {
parent: "MAIN",
targets:[{name: "House"}, {name:"Car"}]
}
If yes, I was thinking about adding a servertimestamp to object MAIN whenever I modify the Document. I would use this timestamp to only update RELATED Documents if their timestamp is smaller then the one of the parent. If yes I update the array and set the timestamp of the parent.
const MAIN = {
ID:"MAIN",
targets:[{name: "House"}, {name:"Car"}],
modifiedAt: 11.04.2022 10:25:33:233
}
const RELATED_1 = {
parent: "MAIN",
lastSync: 11.04.2022 10:25:33:233,
targets:[{name: "House"}, {name:"Car"}]
}
const RELATED_1 = {
parent: "MAIN",
lastSync: 11.04.2022 10:25:33:233,
targets:[{name: "House"}, {name:"Car"}]
}
Would that work? Or how could one sync denormalized data with cloud functions and keep data consistent? Is this even possible?
Using the Cloud Firestore client libraries, you can group multiple operations into a single transaction. Transactions are useful when you want to update a field's value based on its current value, or the value of some other field.
A transaction consists of any number of get() operations followed by any number of write operations such as set(), update(), or delete(). In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data.
Transactions are a way to always ensure a write occurs with the latest information available on the server. Transactions never partially apply writes & all writes execute at the end of a successful transaction.
For a transaction to succeed, the documents retrieved by its read operations must remain unmodified by operations outside the transaction. If another operation attempts to change one of those documents, that operations enters a state of data contention with the transaction.
A transaction works differently than a regular update. It goes like this:
Firestore runs the transaction.
You get the document ready to update whatever property you want to update.
Firestore checks if the document has changed. If not, you’re good, and your update goes through.
If the document has changed, let’s say a new update happened before yours did, then Firestore gets you the new version of the document and repeats the process until it finally updates the document.
The following example from firebase documentation shows how to create and run a transaction:
// Initialize document
const cityRef = db.collection('cities').doc('SF');
await cityRef.set({
name: 'San Francisco',
state: 'CA',
country: 'USA',
capital: false,
population: 860000
});
try {
await db.runTransaction(async (t) => {
const doc = await t.get(cityRef);
// Add one person to the city population.
// Note: this could be done without a transaction
// by updating the population using FieldValue.increment()
const newPopulation = doc.data().population + 1;
t.update(cityRef, {population: newPopulation});
});
console.log('Transaction success!');
} catch (e) {
console.log('Transaction failure:', e);
}
For more information on the above can refer to how do transactions work ,updating data and fastest way to perform bulk data creation
I want to check user's online status, in realtime database I used to check this with the help of onDisconnect(), but now I've shifted to firestore and can't find any similar method in that.
According to this onDisconnect:
The onDisconnect class is most commonly used to manage presence in applications where it is useful to detect how many clients are connected and when other clients disconnect.
To be able to use presence in firestore, you need to connect firestore with realtime firebase(no other way).
Please check this for more info:
https://firebase.google.com/docs/firestore/solutions/presence
NOTE: This solution is not especially efficient
Off the top of my head (read: I haven't thought through the caveats), you could do something like this:
const fiveMinutes = 300000 // five minutes, or whatever makes sense for your app
// "maintain connection"
setInterval(() => {
userPresenceDoc.set({ online: new Date().getTime() })
}, fiveMinutes)
then, on each client...
userPresenceDoc.onSnapshot(doc => {
const fiveMinutesAgo = new Date().getTime() - fiveMinutes
const isOnline = doc.data().online > fiveMinutesAgo
setUserPresence(isOnline)
})
You'd probably want the code checking for presence to use an interval a little more than the interval used by the code maintaining the connection to account for network lag, etc.
A NOTE ABOUT COST
So, obviously, there could be significant lag between when someone disconnects and when that disconnection is recognized by other clients. You can decrease that lag time by increasing the frequency of writes to Firestore, and thus increasing your costs. Running the numbers, I came out with the following costs for different intervals, assuming a single client connection running continuously for a month:
Interval Cost/User/Month
----------------------------
10m $0.007776
5m $0.015552
1m $0.07776
10s $0.46656
5s $0.93312
1s $4.6656
Going with an interval of one second is pretty pricy, at $46,656 a month for a system with 10,000 users who leave the app open all month long. An interval of 10 minutes with the same number of users would only cost $77.76 a month. A more reasonable interval of one minute, 10,000 users, and only four hours of app usage per day per user, rings in at $129.60 / month.
There is no equivalent. The Firestore SDK currently doesn't have presence management like the Realtime Database SDK.
Instead, you might want to use the Realtime Database onDisconnect() in conjunction with Cloud Functions to kick off some work when the client disconnects from RTDB. You would be assuming that that your app probably also lost its connection to Firestore at the same time.
Try this, but this method kind of hackish because we cannot use onDisconnected in firestore. As far as I know, realtime database use secure WebSocket technology, so thats why onDisconnected have on it
But you can use realtime database that can be implemented in cloud functions
to update the firestore data,
functions.database.ref('users/{userId}').onUpdate()
Somewhere in clientside:
firebase.database()
.ref('.info/connected')
.on('value', async (snap) => {
if (snap.val() === true) {
// Update the online status in RTDB
await firebase.database()
.ref(`users/${credentials.user.uid}/`)
.set({
online_status: true,
start_online: firebase.database.ServerValue.TIMESTAMP
});
// OnDisconnect
firebase.database()
.ref(`users/${credentials.user.uid}/`)
.onDisconnect()
.set({
online_status: false,
last_online: firebase.database.ServerValue.TIMESTAMP
});
}
});
in cloudfunctions (this will be trigger on update)
export const onUserOnlineStatusChanged = functions.database.ref('users/{userId}').onUpdate((event: functions.Change<functions.database.DataSnapshot>, context: functions.EventContext) => {
return event.after.ref.once('value')
.then((dataSnapshot) => dataSnapshot.val()) // Get the latest value from the Firebase Realtime database
.then((value: any) => {
// Update the value from RTDB to Firestore
console.log('value.online_status', value.online_status);
if (value.online_status == true) {
// Set the value to the firestore
admin.firestore()
.collection('users_info')
.doc(context.params.userId) // Get document by the userId / Or use .where
.set({
online_status: value.online_status,
updated_at: new Date
}, {
mergeFields: [
'online_status',
'updated_at'
]
});
// Add code if necessary (when the online_status is true)
} else if (value.online_status == false) {
// Set the value to the firestore
admin.firestore()
.collection('users_info')
.doc(context.params.userId) // Get document by the userId / Or use .where
.set({
online_status: value.online_status,
updated_at: new Date
}, {
mergeFields: [
'online_status',
'updated_at'
]
});
// Add code if necessary (when the online_status is false)
}
});
});
It would take about 1 or 2 seconds to update from cloud function to firestore
There is not a direct way to do this thing but this trick helped me achieve this disconnect listener.
window.addEventListener("beforeunload", async function (e) { e.preventDefault(); await firestoreRef.doc("doc-ref").update({ online: false }); });
For example : WorkingTask App. I use Redux - React
Structure
JobOrders (collections) / eachJob (docu)
- the app is allow only insert new JOB.
- the whole day has 500-1000 Jobs.
- the all jobs must be used to recompute the report realtime.
- clear data once every midnight.
I know how to fetch and listen-realtime callback for the whole.
but the problem is the callback function must recompute the whole documents (maybe 300-1000 Jobs every-time when a new JOB arrived.)
This is I want:
- every new JOB arrived only new data in callback then I can recompute in REDUX.
- and prevent for some data is delay due to internet network connection.
I come across with : to listen limit to 1 and sortBy timestamp 'desc'.
But the problem is if some JOB are delay due to network connection, it will not call in callback.
I'm looking for this problem also but now i find out
onAddSnapshotListener
// as real time update.
only get the new or changed documents
https://youtu.be/3aoxOtMM2rc?t=195
Try to watch the whole series
I think Firestore querySnaptshot should have Information for documents changed
for example Node.js
let observer = db
.collection('cities')
.where('state', '==', 'CA')
.onSnapshot(querySnapshot => {
querySnapshot.docChanges().forEach(change => {
if (change.type === 'added') {
console.log('New city: ', change.doc.data());
}
if (change.type === 'modified') {
console.log('Modified city: ', change.doc.data());
}
if (change.type === 'removed') {
console.log('Removed city: ', change.doc.data());
}
});
});
I build react native app with firebase & firestore.
what I'm looking to do is, when user open app, to insert/update his status to 'online' (kind of presence system), when user close app, his status 'offline'.
I did it with firebase.database.onDisconnect(), it works fine.
this is the function
async signupAnonymous() {
const user = await firebase.auth().signInAnonymouslyAndRetrieveData();
this.uid = firebase.auth().currentUser.uid
this.userStatusDatabaseRef = firebase.database().ref(`UserStatus/${this.uid}`);
this.userStatusFirestoreRef = firebase.firestore().doc(`UserStatus/${this.uid}`);
firebase.database().ref('.info/connected').on('value', async connected => {
if (connected.val() === false) {
// this.userStatusFirestoreRef.set({ state: 'offline', last_changed: firebase.firestore.FieldValue.serverTimestamp()},{merge:true});
return;
}
await firebase.database().ref(`UserStatus/${this.uid}`).onDisconnect().set({ state: 'offline', last_changed: firebase.firestore.FieldValue.serverTimestamp() },{merge:true});
this.userStatusDatabaseRef.set({ state: 'online', last_changed: firebase.firestore.FieldValue.serverTimestamp() },{merge:true});
// this.userStatusFirestoreRef.set({ state: 'online',last_changed: firebase.firestore.FieldValue.serverTimestamp() },{merge:true});
});
}
after that, I did trigger to insert data into firestore(because I want to work with firestore), this is the function(works fine, BUT it takes 3-4 sec)
module.exports.onUserStatusChanged = functions.database
.ref('/UserStatus/{uid}').onUpdate((change,context) => {
const eventStatus = change.after.val();
const userStatusFirestoreRef = firestore.doc(`UserStatus/${context.params.uid}`);
return change.after.ref.once("value").then((statusSnapshot) => {
return statusSnapshot.val();
}).then((status) => {
console.log(status, eventStatus);
if (status.last_changed > eventStatus.last_changed) return status;
eventStatus.last_changed = new Date(eventStatus.last_changed);
//return userStatusFirestoreRef.set(eventStatus);
return userStatusFirestoreRef.set(eventStatus,{merge:true});
});
});
then after that, I want to calculate the online users in app, so I did trigger when write new data to node of firestore so it calculate the size of online users by query.(it works fine but takes 4-7 sec)
module.exports.countOnlineUsers = functions.firestore.document('/UserStatus/{uid}').onWrite((change,context) => {
console.log('userStatus')
const userOnlineCounterRef = firestore.doc('Counters/onlineUsersCounter');
const docRef = firestore.collection('UserStatus').where('state','==','online').get().then(e=>{
let count = e.size;
console.log('count',count)
return userOnlineCounterRef.update({count})
})
return Promise.resolve({success:'added'})
})
then into my react native app
I get the count of online users
this.unsubscribe = firebase.firestore().doc(`Counters/onlineUsersCounter`).onSnapshot(doc=>{
console.log('count',doc.data().count)
})
All the operations takes about 12 sec. it's too much for me, it's online app
my firebase structure
what I'm doing wrong? maybe there is unnecessary function or something?
My final goals:
minimize time operation.
get online users count (with listener-each
change, it will update in app)
update user status.
if there are other way to do that, I would love to know.
Cloud Functions go into a 'cold start' mode, where they take some time to boot up. This is the only reason I can think of that it would take that long. Stack Overflow: Firebase Cloud Functions Is Very Slow
But your cloud function only needs to write to Firestore on log out to
catch the case where your user closes the app. You can write to it directly on log in from your client
with auth().onAuthStateChange().
You could also just always read who is logged in or out directly from the
realtime database and use Firestore for the rest of your data.
You can rearrange your data so that instead of a 'UserStatus' collection you have an 'OnlineUsers' collection containing only online users, kept in sync by deleting the documents on log out. Then it won't take a query operation to get them. The query's impact on your performance is likely minimal, but this would perform better with a large number of users.
The documentation also has a guide that may be useful: Firebase Docs: Build Presence in Cloud Firestore
I want to check user's online status, in realtime database I used to check this with the help of onDisconnect(), but now I've shifted to firestore and can't find any similar method in that.
According to this onDisconnect:
The onDisconnect class is most commonly used to manage presence in applications where it is useful to detect how many clients are connected and when other clients disconnect.
To be able to use presence in firestore, you need to connect firestore with realtime firebase(no other way).
Please check this for more info:
https://firebase.google.com/docs/firestore/solutions/presence
NOTE: This solution is not especially efficient
Off the top of my head (read: I haven't thought through the caveats), you could do something like this:
const fiveMinutes = 300000 // five minutes, or whatever makes sense for your app
// "maintain connection"
setInterval(() => {
userPresenceDoc.set({ online: new Date().getTime() })
}, fiveMinutes)
then, on each client...
userPresenceDoc.onSnapshot(doc => {
const fiveMinutesAgo = new Date().getTime() - fiveMinutes
const isOnline = doc.data().online > fiveMinutesAgo
setUserPresence(isOnline)
})
You'd probably want the code checking for presence to use an interval a little more than the interval used by the code maintaining the connection to account for network lag, etc.
A NOTE ABOUT COST
So, obviously, there could be significant lag between when someone disconnects and when that disconnection is recognized by other clients. You can decrease that lag time by increasing the frequency of writes to Firestore, and thus increasing your costs. Running the numbers, I came out with the following costs for different intervals, assuming a single client connection running continuously for a month:
Interval Cost/User/Month
----------------------------
10m $0.007776
5m $0.015552
1m $0.07776
10s $0.46656
5s $0.93312
1s $4.6656
Going with an interval of one second is pretty pricy, at $46,656 a month for a system with 10,000 users who leave the app open all month long. An interval of 10 minutes with the same number of users would only cost $77.76 a month. A more reasonable interval of one minute, 10,000 users, and only four hours of app usage per day per user, rings in at $129.60 / month.
There is no equivalent. The Firestore SDK currently doesn't have presence management like the Realtime Database SDK.
Instead, you might want to use the Realtime Database onDisconnect() in conjunction with Cloud Functions to kick off some work when the client disconnects from RTDB. You would be assuming that that your app probably also lost its connection to Firestore at the same time.
Try this, but this method kind of hackish because we cannot use onDisconnected in firestore. As far as I know, realtime database use secure WebSocket technology, so thats why onDisconnected have on it
But you can use realtime database that can be implemented in cloud functions
to update the firestore data,
functions.database.ref('users/{userId}').onUpdate()
Somewhere in clientside:
firebase.database()
.ref('.info/connected')
.on('value', async (snap) => {
if (snap.val() === true) {
// Update the online status in RTDB
await firebase.database()
.ref(`users/${credentials.user.uid}/`)
.set({
online_status: true,
start_online: firebase.database.ServerValue.TIMESTAMP
});
// OnDisconnect
firebase.database()
.ref(`users/${credentials.user.uid}/`)
.onDisconnect()
.set({
online_status: false,
last_online: firebase.database.ServerValue.TIMESTAMP
});
}
});
in cloudfunctions (this will be trigger on update)
export const onUserOnlineStatusChanged = functions.database.ref('users/{userId}').onUpdate((event: functions.Change<functions.database.DataSnapshot>, context: functions.EventContext) => {
return event.after.ref.once('value')
.then((dataSnapshot) => dataSnapshot.val()) // Get the latest value from the Firebase Realtime database
.then((value: any) => {
// Update the value from RTDB to Firestore
console.log('value.online_status', value.online_status);
if (value.online_status == true) {
// Set the value to the firestore
admin.firestore()
.collection('users_info')
.doc(context.params.userId) // Get document by the userId / Or use .where
.set({
online_status: value.online_status,
updated_at: new Date
}, {
mergeFields: [
'online_status',
'updated_at'
]
});
// Add code if necessary (when the online_status is true)
} else if (value.online_status == false) {
// Set the value to the firestore
admin.firestore()
.collection('users_info')
.doc(context.params.userId) // Get document by the userId / Or use .where
.set({
online_status: value.online_status,
updated_at: new Date
}, {
mergeFields: [
'online_status',
'updated_at'
]
});
// Add code if necessary (when the online_status is false)
}
});
});
It would take about 1 or 2 seconds to update from cloud function to firestore
There is not a direct way to do this thing but this trick helped me achieve this disconnect listener.
window.addEventListener("beforeunload", async function (e) { e.preventDefault(); await firestoreRef.doc("doc-ref").update({ online: false }); });