I've one firebase database instance and I would like to add a counter to a certain node.
Everytime the users run an specific action I would like to increment the node value. How to do that without getting synchronization problems? How to use google functions to do that?
Ex.:
database{
node {
counter : 0
}
}
At certain time 3 different users read the value on counter, and try to increment it. As they read at exact same time all of them read "0" and incremented to "1", but the desired value at end of execution should be "3" since it was read 3 times
==================update===================
#renaud pointed to use transactions to keep synchronization on of the saved data, but i have another scenario where i need the synchronization done on read side also:
ex.
the user read the actual value, acording to it does a different action and finishing by incrementing one...
in a sql like enviorement i would write a procedure for doing that, because doesn't matter what user will do with the info i will finish always by incrementing one
If i did understand #renaud answer right, in that scenario 4 different users reading the database at same time would get 0 as current value, then on transaction update the final stored value would be 4, but on client side each of them just read 0
You have to use a Transaction in this case, see https://firebase.google.com/docs/database/web/read-and-write#save_data_as_transactions and also https://firebase.google.com/docs/reference/js/firebase.database.Reference#transaction
A Transaction will "ensure there are no conflicts with other clients writing to the same location at the same time."
In a Cloud Function you could write your code along the following lines, for example:
....
const counterRef = admin
.database()
.ref('/node/counter');
return counterRef
.transaction(current_value => {
return (current_value || 0) + 1;
})
.then(counterValue => {
if (counterValue.committed) {
//For example update another node in the database
const updates = {};
updates['/nbrOfActionsExecuted'] = counterValue.snapshot.val();
return admin
.database()
.ref()
.update(updates);
}
})
or simply the following if you just want to update the counter (Since a transaction returns a Promise, as explained in the second link referred to above):
exports.testTransaction = functions.database.ref('/path').onWrite((change, context) => {
const counterRef = admin
.database()
.ref('/node/counter');
return counterRef
.transaction(current_value => {
return (current_value || 0) + 1;
});
});
Note that, in this second case, I have used a Realtime Database trigger as an example of trigger.
Related
I'm using flutter and firebase. I use pagination, max 5 documents per page. How do I know if there are more documents left to get from a firestore collection. I want to use this information to enable/disable a next page button presented to the user.
limit: 5 (5 documents each time)
orderBy: "date" (newest first)
startAfterDocument: latestDocument (just a variable that holds the latest document)
This is how I fetch the documents.
collection.limit(5).orderBy("date", descending: true).startAfterDocument(latestDocument).get()
I thought about checking if the number of docs received from firestore is equal to 5, then assume there are more docs to get. But this will not work if I there are a total of n * 5 docs in the collection.
I thought about getting the last document in the collection and store this and compare this to every doc in the batches I get, if there is a match then I know I've reach the end, but this means one excess read.
Or maybe I could keep on getting docs until I get an empty list and assume I've reached the end of the collection.
I still feel there are a much better solution to this.
Let me know if you need more info, this is my first question on this account.
There is no flag in the response to indicate there are more documents. The common solution is to request one more document than you need/display, and then use the presence of that last document as an indicator that there are more documents.
This is also what the database would have to do to include such a flag in its response, which is probably why this isn't an explicit option in the SDK.
You might also want to check the documentation on keeping a distributed count of the number of documents in a collection as that's another way to determine whether you need to enable the UI to load a next page.
here's a way to get a large data from firebase collection
let latestDoc = null; // this is to store the last doc from a query
//result
const dataArr = []; // this is to store the data getting from firestore
let loadMore = true; // this is to check if there's more data or no
const initialQuery = async () => {
const first = db
.collection("recipes-test")
.orderBy("title")
.startAfter(latestDoc || 0)
.limit(10);
const data = await first.get();
data.docs.forEach((doc) => {
// console.log("doc.data", doc.data());
dataArr.push(doc.data()); // pushing the data into the array
});
//! update latest doc
latestDoc = data.docs[data.docs.length - 1];
//! unattach event listeners if no more docs
if (data.empty) {
loadMore = false;
}
};
// running this through this function so we can actual await for the
//docs to get from firebase
const run = async () => {
// looping until we get all the docs
while (loadMore) {
console.log({ loadMore });
await initialQuery();
}
};
I'm looking for a way to prevent writing more than a given limit of documents to a (sub)collection in a given periode.
For example: Messenger A is not allowed to write more then 1000 Messages per 24 hours.
This should be done in the context of an Firebase Function API endpoint because it's called by third parties.
The endpoint
app.post('/message', async function(req:any, res:any) {
// get the messenger's API key
const key = req.query.key
// if no key was provided, return error
if(!key) {
res.status(400).send('Please provide a valid key')
return
}
// get the messenger by the provided key
const getMessengerResult = await admin.firestore().collection('messengers')
.where('key', '==', key).limit(1).get()
// if there is no result the messenger is not valid, return error
if (getMessengerResult.empty){
res.status(400).send('Please provide a valid key')
return
}
// get the messenger from the result
const messenger = getMessengerResult.docs[0]
// TODO: check if messenger hasn't reached limit of 1000 messages per 24 hours
// get message info from the body
const title:String = req.body.title
const body: String = req.body.body
// add message
await messenger.ref.collection('messages').add({
'title':title,
'body':body,
'timestamp': Timestamp.now()
})
// send response
res.status(201).send('The notification has been created');
})
One thing I've tried was the following piece of code in place of the TODO::
// get the limit message and validate its timestamp
const limitMessageResult = await messenger.ref.collection('messages')
.orderBy('timestamp',"desc").limit(1).offset(1000).get()
if(!limitMessageResult.empty) {
const limitMessage = limitMessageResult.docs[0]
const timestamp: Timestamp = limitMessage.data()['timestamp']
// create a date object for 24 hours ago
const 24HoursAgo = new Date()
24HoursAgo.setDate(24HoursAgo.getDate()-1)
if(24HoursAgo < timestamp.toDate()) {
res.status(405).send('You\'ve exceeded your messages limit, please try again later!')
return
}
}
This code works, but there is a big BUT. The offset does indeed skip the 1000 results, but Firebase will still charge you for it! So every time the messenger tries to add 1 message, 1000+ are read... and that's costly.
So I need a better (cheaper) way to do this.
One thing I've come up with, but haven't yet tried would be adding an index/counter field to a message that increases by 1 every message.
Then instead of doing:
const limitMessageResult = await messenger.ref.collection('messages')
.orderBy('timestamp',"desc").limit(1).offset(1000).get()
I could do something like:
const limitMessageResult = await messenger.ref.collection('messages')
.where('index','==', currentIndex-1000).limit(1).get()
But I was wondering if that would be a save way.
For example, what would happen if there are multiple request at the same time.
I would first need to get the current index from the last message and add the new message with index+1. But could two requests read, and thus write the same index? Or could this be handled with transactions?
Or is there a totally different way to solve my problem?
I have a strong aversion against using offset() in my server-side code, precisely because it makes it seem like it's skipping documents, where it's actually reading-and-discarding them.
The simplest way I can think of to implement your maximum-writes-per-day count is to keep a writes-per-day counter for each messenger, that you then update whenever they write a message.
For example, you could do the following whenever you write a message:
await messenger.ref.collection('messages').add({
'title':title,
'body':body,
'timestamp': Timestamp.now()
})
const today = new Date().toISOString().substring(0, 10); // e.g. "2020-04-11"
await messenger.ref.set({
[today]: admin.firestore.FieldValue.increment(1)
}, { merge: true })
So this adds an additional field to your messenger document for each day, whee it then keeps a count of the number of messages that messenger has written for that day.
You'd then use this count instead of your current limitMessageResult.
const messageCount = (await messenger.get()).data()[today] || 0;
if (messageCount < 1000) {
... post the message and increase the counter
}
else {
... reject the message, and return a message
}
Steps left to do:
You'll want to secure write access to the counter fields, as the messenger shouldn't be able to modify these on their own.
You may want to clean out older message counts periodically, if you're worried about the messenger's document becoming too big. I prefer to leave these types of counters, as they give an opportunity to provide some stats cheaply if needed.
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)
I want to clean up this userPublic by deleting all of its child node which has isTesting == true. I am using Firebase's cloud function. My approach would be :
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", dataSnapshot => {
// ???
})
Since I can only call .remove() on reference and not snapshot but to filter the child I want it returns snapshot, how can I get the reference from snapshot? (I would like to know the key XXX-XXX-XXX of each filtered child, so I can concatenate with userPublic and .remove() them one by one)
Also, even if I can get all the references that I want to remove I think deleting them one by one by calling .remove() then wait for promise, then call the next one does not sounds like an optimal way. Are there any way to remove all of them in one go?
If it involves calling .update() on the top userPublic node, I would have to fetch everything, remove the one with isTesting and then put the remaining back for update. This sounds like it is not efficient compared to the filtering way. As eventually the one with .isTesting is only about 5% of all data. Or is this actually the approach everyone is using?
You're almost there. All that's left is to create a single multi-location update from the results of your query:
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", snapshot => {
var updates = {};
snapshot.forEach(function(child) {
updates["/userPublic/"+child.key] = null;
});
userPublic.update(updates);
})
Doing this with promises would not be too different:
testsInUserPublic.once("value", snapshot => {
var promises = [];
snapshot.forEach(function(child) {
promises.push(child.ref.remove());
});
Promise.all(promises); // this returns a promise that resolves once all deletes are done, or that rejects once one of them fails
})
Performance of this will be very similar, since Firebase pipelines the requests over a single connection. See http://stackoverflow.com/questions/35931526/speed-up-fetching-posts-for-my-social-network-app-by-using-query-instead-of-obse/35932786#35932786
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)