I have a firebase functions which get triggered on onCreate for path /activites/{uid}/{workId}. This function then do some processing with the values and then updates the results (number value) to /users/{uid}/jobs.
// value read from the snapshot function gives
// processing on value
// .
// .
// .
db.ref('/users/{uid}/jobs').transaction(currentVal => {
return (currentVal || 0) + workIdValue;
})
My issue of inconsistent writes appears when let say function is triggered 40 times in span of 1 second per uid i.e. for the same path. This makes data inconsistent since each execution is having different currentValue from path /users/{uid}/jobs
I tired a solution of reading the value of the jobs from db to achieve consistency and then initiate transaction to update value but the result is still the same.
Other solution which I didn't tested is to use a lock mechanism by placing a boolean at the node /users/{uid}/jobs/locked and when trying to do update first check if boolean is false (meaning no other execution is modifying data) then set boolean true update the value and then release the lock setting value false.
Is there any better approach to solve this inconsistency?
Related
This question already has an answer here:
How to return failed task result in continuation task?
(1 answer)
Closed 2 years ago.
I'm writing my first app in Kotlin and am using Firestore & Firebase Storage. In the process of deleting a document, I want to delete all files in Storage that the document references (as it is the only reference to them in my case). If the Storage delete fails, I want to abort the document delete, in order to avoid orphan files in my Storage. I also want to do everything in "one Task", to allow showing a progress bar properly. My simplified code looks like this:
fun deleteItem(id: String): Task<Void>? {
val deleteTask = deleteTaleMedia(id)
continueWithTaskOrInNew(deleteTask) { task ->
if (task?.isSuccessful != false) { ... }
}
}
fun deleteItemMedia(id: String): Task<Void>? =
getItem(id)?.continueWithTask { task ->
if (task.isSuccessful)
task.result?.toObject(ItemModel::class.java)?.let { deleteFiles(it.media) }
else ???
}
fun deleteFiles(filesList: List<String>): Task<Void>? {
var deleteTask: Task<Void>? = null
for (file in filesList) deleteTask = continueWithTaskOrInNew(deleteTask) { task ->
if (task?.isSuccessful != false) deleteFile(file)
else task
}
return task
}
fun deleteFile(fileName: String) = Firebase.storage.getReferenceFromUrl(fileName).delete()
fun getItem(id: String): Task<DocumentSnapshot>? {
val docRef = userDocRef?.collection(COLLECTION_PATH)?.document(id)
return docRef?.get()
?.addOnCompleteListener { ... }
}
fun <ResultT, TContinuationResult> continueWithTaskOrInNew(
task: Task<ResultT>?,
continuation: (Task<ResultT>?) -> Task<TContinuationResult>?
) = task?.continueWithTask { continuation.invoke(task) } ?: continuation.invoke(null)
data class ItemModel(
#DocumentId val id: String = "",
var title: String = "",
var media: List<String> = listOf()
)
My problem comes in deleteItemMedia function (the "???" at the end). In case the get task failed, I want to return a task that will tell my deleteItem function to abort deletion (task.isSuccessful == false). I cannot return the get task itself (replace "???" with "task" in code), because it's type (Task<DocumentSnapshot>) differs from the type of the delete task (Task<Void>). I cannot return null, as null is returned in the case of no media at all, which is a valid case for me (document should be deleted). Is there a way to create a new "failed Task"?
In the process of deleting a document I want to delete all files in Storage that the document references (as it is the only reference to them in my case).
There is no API that's doing that. You have to perform both delete operations yourself.
I also want to do everything in "one Task", to allow showing a progress bar properly.
Unfortunately, this is not possible in a single go. If you think of an atomic operation, this also not possible because none of the Firebase services support this kind of cross-product transactional operations. What you need to do is, get the document, get the references to the files in the Storage, delete the document and as soon as the delete operation is complete, delete the files. You can definitely reduce the risk by trying to roll-back the data from the client, but you cannot do them atomic, in "one Task". However, at some point in time, there will be an Exception that the client can't rollback.
If the Storage delete fails, I want to abort the document delete, in order to avoid orphan files in my Storage.
To avoid that, first, try not to have incomplete data. For instance, when you read the document and you get the corresponding Storage URLs, don't blindly assume that all those files actually exist. A file can unavailable for many reasons (was previously deleted, for some reasons the service is unavailable, etc.)
Another approach might be to use Cloud Functions for Firebase, so you can delete the desired document, and use onDelete function to delete the corresponding files from the Storage. Meaning, when document delete fails, the files from the Storage won't be deleted. If the operation to delete the document is successful, the Cloud Function will be triggered and the images will be deleted from the Storage. This approach will drastically reduce the chances of having failures between the document delete operation and the deletion of the files from Storage, but it doesn't eliminate that chance 100%.
Besides that, the most common approach to avoid failures is to make your code as robust as you possibly can against failure and do frequent database cleanups.
For incrementing an int value in my database , I first get that value by using a listener , increment it by 1 and then set the new value to database. This works but I want to know if there is an easier way of doing this. This way seems like too much work.
Update: since early 2020 there actually is a server-side operation to increment values in the Realtime Database. See dfeverx's answer on this, or my own Q&A comparing performance between transactions and increments: How quickly can you atomically increment a value on the Firebase Realtime Database?.
There is no server-side way to increment values (or do other calculations) in the Firebase Database.
Your approach is one way of doing this, but it has the chance of leading to a race condition.
the value in the database is 12
client 1 reads the value 12
client 2 read the value 12
client 1 writes its incremented value 13
client 2 writes its incremented value 13
The result is now likely incorrect (it depends on your use-case).
In that case, one way to make it work is to use Firebase transactions, which combine the reading and writing into a single function.
Transactions are covered in the Firebase Database documentation. I highly recommend reading it. A few hours spent there will prevent many problems down the road. From the docs:
postRef.runTransaction(new Transaction.Handler() {
#Override
public Transaction.Result doTransaction(MutableData mutableData) {
Long value = mutableData.getValue(Long.class);
if (value == null) {
mutableData.setValue(0);
}
else {
mutableData.setValue(value + 1);
}
return Transaction.success(mutableData);
}
#Override
public void onComplete(DatabaseError databaseError, boolean b,
DataSnapshot dataSnapshot) {
Log.d(TAG, "transaction:onComplete:" + databaseError);
}
});
From firebase JavaScript SDK v7.14.0 you can use ServerValue.increment()
to increment value in firebase
Method 1
var userRef= firebase.database().ref("user/user_id_123");
userRef.push({
likes: firebase.database.ServerValue.increment(1)
});
Method 2
firebase.database()
.ref('user')
.child('user_id_123')
.child('likes')
.set(firebase.database.ServerValue.increment(1))
You can also pass float or negative value to the increment function
You can find the reference here: https://firebase.google.com/docs/reference/js/v8/firebase.database.ServerValue
I've one firebase database instance and I would like to add a counter to a certain node.
Everytime the users run an specific action I would like to increment the node value. How to do that without getting synchronization problems? How to use google functions to do that?
Ex.:
database{
node {
counter : 0
}
}
At certain time 3 different users read the value on counter, and try to increment it. As they read at exact same time all of them read "0" and incremented to "1", but the desired value at end of execution should be "3" since it was read 3 times
==================update===================
#renaud pointed to use transactions to keep synchronization on of the saved data, but i have another scenario where i need the synchronization done on read side also:
ex.
the user read the actual value, acording to it does a different action and finishing by incrementing one...
in a sql like enviorement i would write a procedure for doing that, because doesn't matter what user will do with the info i will finish always by incrementing one
If i did understand #renaud answer right, in that scenario 4 different users reading the database at same time would get 0 as current value, then on transaction update the final stored value would be 4, but on client side each of them just read 0
You have to use a Transaction in this case, see https://firebase.google.com/docs/database/web/read-and-write#save_data_as_transactions and also https://firebase.google.com/docs/reference/js/firebase.database.Reference#transaction
A Transaction will "ensure there are no conflicts with other clients writing to the same location at the same time."
In a Cloud Function you could write your code along the following lines, for example:
....
const counterRef = admin
.database()
.ref('/node/counter');
return counterRef
.transaction(current_value => {
return (current_value || 0) + 1;
})
.then(counterValue => {
if (counterValue.committed) {
//For example update another node in the database
const updates = {};
updates['/nbrOfActionsExecuted'] = counterValue.snapshot.val();
return admin
.database()
.ref()
.update(updates);
}
})
or simply the following if you just want to update the counter (Since a transaction returns a Promise, as explained in the second link referred to above):
exports.testTransaction = functions.database.ref('/path').onWrite((change, context) => {
const counterRef = admin
.database()
.ref('/node/counter');
return counterRef
.transaction(current_value => {
return (current_value || 0) + 1;
});
});
Note that, in this second case, I have used a Realtime Database trigger as an example of trigger.
So I have a cloud function that is triggered each time a transaction is liked/unliked. This function increments/decrements the likesCount. I've used firestore transactions to achieve the same. I think the problem is the Code inside the Transaction block is getting executed multiple times, which may be correct as per the documentation.
But my Likes count are being updated incorrectly at certain times.
return firestore.runTransaction(function (transaction) {
return transaction.get(transRef).then(function (transDoc) {
let currentLikesCount = transDoc.get("likesCount");
if (event.data && !event.data.previous) {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 1 : transDoc.get("likesCount") + 1;
} else {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 0 : transDoc.get("likesCount") - 1;
}
transaction.update(transRef, { likesCount: newLikesCount });
});
});
Anyone had similar experience
Guys finally found out the cause for this unexpected behaviour.
Firestore isn't suitable for maintaining counters if your application is going to be traffic intensive. They have mentioned it in their documentation. The solution they suggest is to use a Distributed counter.
Many realtime apps have documents that act as counters. For example,
you might count 'likes' on a post, or 'favorites' of a specific item.
In Cloud Firestore, you can only update a single document about once
per second, which might be too low for some high-traffic applications.
https://cloud.google.com/firestore/docs/solutions/counters
I wasn't convinced with that approach as it's too complex for a simple use case, which is when I stumbled across the following blog
https://medium.com/evenbit/on-collision-course-with-cloud-firestore-7af26242bc2d
These guys used a combination of Firestore + Firebase thereby eliminating their weaknesses.
Cloud Firestore is sitting conveniently close to the Firebase Realtime
Database, and the two are easily available to use, mix and match
within an application. You can freely choose to store data in both
places for your project, if that serves your needs.
So, why not use the Realtime database for one of its strengths: to
manage fast data streams from distributed clients. Which is the one
problem that arises when trying to aggregate and count data in the
Firestore.
Its not correct to say that Firestore is an upgrade to the Realtime database (as it is advertised) but a different database with different purposes and both can and should coexist in a large scale application. That's my thought.
It might have something to do with what you're returning from the function, as you have
return transaction.get(transRef).then(function (transDoc) { ... })
And then another return inside that callback, but no return inside the inner-most nested callback. So it might not be executing the transaction.update. Try removing the first two return keywords and add one before transaction.update:
firestore.runTransaction(function (transaction) {
transaction.get(transRef).then(function (transDoc) {
let currentLikesCount = transDoc.get("likesCount");
if (event.data && !event.data.previous) {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 1 : transDoc.get("likesCount") + 1;
} else {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 0 : transDoc.get("likesCount") - 1;
}
return transaction.update(transRef, { likesCount: newLikesCount });
});
});
Timeouts
First of all, check your Cloud Functions logs to see if you get any timeout messages.
Function execution took 60087 ms, finished with status: 'timeout'
If so, sort out your function so that it returns a Promise.resolve(). And shows
Function execution took 344 ms, finished with status: 'ok'
Idempotency
Secondly, write your data so that the function is idempotent. When your function runs, write a value to the document that you are reading. You can then check if that value exists before running the function again.
See this example for ensuring that functions are only run once.
I have a problem with transactions. The data in the transaction is always null and the update handler is called only singe once. The documentation says :
To accomplish this, you pass transaction() an update function which is
used to transform the current value into a new value. If another
client writes to the location before your new value is successfully
written, your update function will be called again with the new
current value, and the write will be retried. This will happen
repeatedly until your write succeeds without conflict or you abort the
transaction by not returning a value from your update function
Now I know that there is no other client accessing the location right now. Secondly if I read the documentation correctly the updateCounters function should be called multiple times should it fail to retrieve and update data.
The other thing - if I take out the condition if (counters === null) the execution will fail as counters is null but on a subsequent attempt the transaction finishes fine - retrieves data and does the update.
simple once - set on this location work just fine but it is not safe.
Please what do I miss?
here is the code
self.myRef.child('counters')
.transaction(function updateCounters(counters){
if (counters === null) {
return;
}
else {
console.log('in transaction counters:', counters);
counters.comments = counters.comments + 1;
return counters;
}
}, function(error, committed, ss){
if (error) {
console.log('transaction aborted');
// TODO error handling
} else if (!committed){
console.log('counters are null - why?');
} else {
console.log('counter increased',ss.val());
}
}, true);
here is the data in the location
counters:{
comments: 1,
alerts: 3,
...
}
By returning undefined in your if( ... === null ) block, you are aborting the transaction. Thus it never sends an attempt to the server, never realizes the locally cached value is not the same as remote, and never retries with the updated value (the actual value from the server).
This is confirmed by the fact that committed is false and the error is null in your success function, which occurs if the transaction is aborted.
Transactions work as follows:
pass the locally cached value into the processing function, if you have never fetched this data from the server, then the locally cached value is null (the most likely remote value for that path)
get the return value from the processing function, if that value is undefined abort the transaction, otherwise, create a hash of the current value (null) and pass that and the new value (returned by processing function) to the server
if the local hash matches the server's current hash, the change is applied and the server returns a success result
if the server transaction is not applied, server returns the new value, client then calls the processing function again with the updated value from the server until successful
when ultimately successful, and unrecoverable error occurs, or the transaction is aborted (by returning undefined from the processing function) then the success method is called with the results.
So to make this work, obviously you can't abort the transaction on the first returned value.
One workaround to accomplish the same result--although it is coupled and not as performant or appropriate as just using the transactions as designed--would be to wrap the transaction in a once('value', ...) callback, which would ensure it's cached locally before running the transaction.