I have a Firebase web Realtime DB with users, each of whom has a jobs attribute whose value is an object:
{
userid1:
jobs:
guid1: {},
guid2: {},
userid2:
jobs:
guid1: {},
guid2: {},
}
I want to query to get the n users with the most jobs. Is there an orderby trick I can use to order the users by the number of values the given user has in their jobs attribute?
I specifically don't want to store an integer count of the number of jobs each user has because I need to update users' jobs attribute as a part of atomic updates that update other user attributes concurrently and atomically, and I don't believe transactions (like incrementing/decrementing counters) can be a part of those atomic transactions.
Here's an example of the kind of atomic update I'm doing. Note I don't have the user that I'm modifying in memory when I run the following update:
firebase.database().ref('/').update({
[`/users/${user.guid}/pizza`]: true,
[`/users/${user.guid}/jobs/${job.guid}/scheduled`]: true,
})
Any suggestions on patterns that would work with this data would be hugely appreciated!
Realtime Database transactions run on a single node in the JSON tree, so it would be quite difficult to integrate the update of a jobCounter node within your atomic update to several nodes (i.e. to /users/${user.guid}/pizza and /users/${user.guid}/jobs/${job.guid}/scheduled). We would need to update at /users/${user.guid} level and calculate the counter value, etc...
An easier approach is to use a Cloud Function to update a user's jobCounter node each time there is a change to one of the jobs nodes that implies a change in the counter. In other words, if a new job node is added or removed, the counter is updated. If an existing node is only modified, the counter is not updated, since there were no change in the number of jobs.
exports.updateJobsCounter = functions.database.ref('/users/{userId}/jobs')
.onWrite((change, context) => {
if (!change.after.exists()) {
//This is the case when no more jobs exist for this user
const userJobsCounterRef = change.before.ref.parent.child('jobsCounter');
return userJobsCounterRef.transaction(() => {
return 0;
});
} else {
if (!change.before.val()) {
//This is the case when the first job is created
const userJobsCounterRef = change.before.ref.parent.child('jobsCounter');
return userJobsCounterRef.transaction(() => {
return 1;
});
} else {
const valObjBefore = change.before.val();
const valObjAfter = change.after.val();
const nbrJobsBefore = Object.keys(valObjBefore).length;
const nbrJobsAfter = Object.keys(valObjAfter).length;
if (nbrJobsBefore !== nbrJobsAfter) {
//We update the jobsCounter node
const userJobsCounterRef = change.after.ref.parent.child('jobsCounter');
return userJobsCounterRef.transaction(() => {
return nbrJobsAfter;
});
} else {
//No need to update the jobsCounter node
return null;
}
}
}
});
Related
Objective
User on click a purchase button on the web frontend, it will send a POST request to the backend to create a purchase order. First, it will check the number of available stocks. If available is greater than 0, reduce available by 1 and then create the order.
The setup
Backend (NestJS) queries the Firestore for the latest available value, and reduce available by 1. For debugging, I will return the available value.
let available;
try {
await runTransaction(firestore, async (transaction) => {
const sfDocRef = doc(collection(firestore, 'items_available'), documentId);
const sfDoc = await transaction.get(sfDocRef);
if (!sfDoc.exists()) {
throw 'Document does not exist!';
}
const data = sfDoc.data();
available = data.available;
if(available>0){
transaction.update(sfDocRef, {
available: available-1,
});
}
});
} catch (e) {
console.log('Transaction failed: ', e);
}
return { available };
My stress test setup
Our goal is to see all API requests having different available value, this would mean that Firestore Transactions is reducing the value even though there are multiple requests coming in.
I wrote a simple multi-threaded program that queries the backend's create order API, it will query the available value and return the available value. This program will save the available value returned for each API request.
The stress test performed is about 10 transactions per second, as I have 10 concurrent processes querying the backend. Each process will http.get 20 queries:
const http = require('http');
function call(){
http.get('http://localhost:5000/get_item_available', res => {
let data = [];
res.on('data', chunk => {
data.push(chunk);
});
res.on('end', () => {
console.log('Response: ', Buffer.concat(data).toString());
});
}).on('error', err => {
console.log('Error: ', err.message);
});
}
for (var i=0; i<20; i++){
call();
}
The problem
Unfortunately, the available values I got from the requests contains repeated values, that is, having same available values instead of having unique available values.
What is wrong? Isn't Firestore Transactions meant to handle race conditions? Any suggestions on what I could change to handle multiple requests hitting the server and return a new value for each request?
You have a catch clause to handle when the transaction fails, but then still end up returning a value to the caller return { available }. In that situation you should return an error to the caller.
I have a problem with transactions in my web application created in Flutter. For database I use Firebase Firestore where I save documents via transaction.
Dependency:
cloud_firestore: 3.1.1
StudentGroup is my main document. It has 4 stages and each of them has 3-5 tasks. (Everything is in 1 document). I have to store game timer, so every 10 seconds I make an request to save time for current stage. (Every stage has different timer). I have a problem with saving task, because "Sometimes" when 2 requests are made in the same time, then I get some weird state manipulation.
Task is updated and "isFinished" is set to true
Timer is updated to correct value (with this update somehow previous task update is lost, "isFinished" is set to false
This is how I save task.
Future<Result> saveTask({required String sessionId, required String studentGroupId,
required Task task}) async {
print("trying to save task <$task>.");
try {
return await _firebaseFirestore.runTransaction((transaction) async {
final studentGroupRef = _getStudentGroupDocumentReference(
sessionId: sessionId,
studentGroupId: studentGroupId
);
final sessionGroupDoc = await studentGroupRef.get();
if (!sessionGroupDoc.exists) {
return Result.error("student group not exists");
}
final sessionGroup = StudentGroup.fromSnapshot(sessionGroupDoc);
sessionGroup.game.saveTask(task);
transaction.set(studentGroupRef, sessionGroup.toJson());
})
.then((value) => taskFunction(true))
.catchError((error) => taskFunction(false));
} catch (error) {
return Result.error("Error couldn't save task");
}
}
This is how I save my time
Future<Result> updateTaskTimer({required String sessionId,
required String studentGroupId, required Duration duration}) async {
print("trying to update first task timer");
try {
return await _firebaseFirestore.runTransaction((transaction) async {
final studentGroupRef = _getStudentGroupDocumentReference(
sessionId: sessionId,
studentGroupId: studentGroupId
);
final sessionGroupDoc = await studentGroupRef.get();
if (!sessionGroupDoc.exists) {
return Result.error("student group not exists");
}
final sessionGroup = StudentGroup.fromSnapshot(sessionGroupDoc);
switch (sessionGroup.game.gameStage) {
case GameStage.First:
sessionGroup.game.stages.first.duration = duration.inSeconds;
break;
case GameStage.Second:
sessionGroup.game.stages[1].duration = duration.inSeconds;
break;
case GameStage.Third:
sessionGroup.game.stages[2].duration = duration.inSeconds;
break;
case GameStage.Fourth:
sessionGroup.game.stages[3].duration = duration.inSeconds;
break;
case GameStage.Fifth:
sessionGroup.game.stages[4].duration = duration.inSeconds;
break;
}
transaction.set(
studentGroupRef,
sessionGroup.toJson(),
SetOptions(merge: true)
);
print("Did I finish task 4? ${sessionGroup.game.stages.first.tasks[3].isFinished}");
})
.then((value) => timerFunction(true))
.catchError((error) => timerFunction(false));
} catch (error) {
return Result.error("Error couldn't update task timer");
}
}
timerFunction and taskFunction print some messages in console and return Result.error or Result.success (for now it returns bool)
I don't know If I am doing something wrong with Firebase Firestore Transaction. I would like to have atomic operations for reading and writing data.
Transactions ensure atomicity - which means that if the transaction succeeds then all the reads and writes occur in a non-overlapping way with other transactions. This prevents the type of problem you are describing.
But this doesn't work if you spread your reads and writes over multiple transactions. In particular, it looks to me like you are writing a task which was obtained from outside the transaction. Instead, you should use ids or similar to track which documents you need to update, then do a read and a write inside the transaction.Alternatively firebase also provides Batched Writes, which specify the specific properties you want to update. These will ensure that any other properties are not changed.For batch writes example you can refer to the link
I have two collections. (applyJobs and Jobs and users). When users apply for a job, I store that record inside applyJobs collection. Like this:
applyId:****,
jobId:*****,
userId:*****
Now, I want to show all apply for jobs by a user.
First: Get logged user id, I store locally logged user id. So, I can get loggged user id.
Second: I filter Apply Jobs by that id. like this, var ref = _db.collection('applyJobs').where('userId',isEqualTo: uid);. I here I didn't call users collection to get uid. because I already store uid on locally. Is it best practice?
Third: I store result here List<ApplyJobsModelClass>. I want to get all jobs by a list of id. How do I filter it?
This is way I tried it. But this is not list of IDs. only one id.
streamApplyJob(List<String> jobId) {
Collection('jobs').document(jobId);
}
And I tried this way too.
Stream<List<JobModel>> streamApplyJob(List<String> jobId) {
var ref = _db.collection('jobs').where('jobId',isEqualTo: jobId);
return ref.snapshots().map((list) =>
list.documents.map((doc) => JobModel.fromFirestore(doc)).toList());
}
tried to get length, but result is 0
db.streamApplyJob(jobIds).listen((v)=>{
print(v.length)
});
Full Code
Database side
///Get a stream of apply jobs
Stream<List<ApplyJobModel>> streamApplyJobs(String uid) {
var ref = _db.collection('applyJobs').where('userId',isEqualTo: uid);
return ref.snapshots().map((list) =>
list.documents.map((doc) => ApplyJobModel.fromFirestore(doc)).toList());
}
///Get a stream of a single document
Stream<List<JobModel>> streamApplyJob(List<String> jobId) {
var ref = _db.collection('jobs').where('jobId',isEqualTo: jobId);
return ref.snapshots().map((list) =>
list.documents.map((doc) => JobModel.fromFirestore(doc)).toList());
}
calling
List<String> jobIds = [];
void getData() {
db.streamApplyJobs(widget.uid).listen((listApplies) => {
for (int i = 0; i < listApplies.length; i++)
{jobIds.add(listApplies[i].jobId)},
});
db.streamApplyJob(jobIds).listen((v)=>{
print(v.length)
});
}
Solution(It's working now)- Is it best practice or are there other best way to do this?
Future<List<JobModel>> getJobs() async {
await db.streamJobs(true).listen((jobs) {
setState(() {
jobModel = jobs;
});
});
return jobModel;
}
Future getData() async {
await getJobs();
db.streamApplyJobs(widget.uid).listen((apply) => {
for (int j = 0; j < jobModel.length; j++)
{
for (int i = 0; i < apply.length; i++)
{
if (apply[i].jobId == jobModel[j].jobId)
{
jobModelNew.add(jobModel[j]),
}
}
}
});
}
I want to get all jobs by a list of id. How do I filter it?
There currently is no way to pass in a list of IDs to a Firestore query and get documents matching all those IDs. See Google Firestore - how to get document by multiple ids in one round trip? (which talks about doing this with document IDs), and Firebase Firestore - OR query (which talks about filtering for multiple values on a single field).
Unless your use-case happens to match the workaround mentioned in that second answer, you'll have to perform a separate query for each value, and merge the results in your application code.
Not sure if it is documented anywhere officially, but this is possible now!
.where(admin.firestore.FieldPath.documentId(), "in", [array, of, ids])
Found here: https://stackoverflow.com/a/52252264/10562805
Please take a look at this example. It binds a CollectionReference to a List.
Let me know if this is helpful.
Is there a way to retrieve the updated value of a document field updated using firestore.FieldValue.increment without asking for the document?
var countersRef = db.collection('system').doc('counters');
await countersRef.update({
nextOrderCode: firebase.firestore.FieldValue.increment(1)
});
// Get the updated nextOrderCode without asking for the document data?
This is not cost related, but for reliability. For example if I want to create a code that increases for each order, there is no guaranty that if >= 2 orders happen at the same time, will have different codes if I read the incremental value right after the doc update resolves, because if >= 2 writes happen before the first read, then at least 2 docs will have the same code even if the nextOrderCode will have proper advance increment.
Update
Possible now, check other answer.
It's not possible. You will have to read the document after the update if you want to know the value.
If you need to control the value of the number to prevent it from being invalid, you will have to use a transaction instead to make sure that the increment will not write an invalid value. FieldValue.increment() would not be a good choice for this case.
We can do it by using Firestore Transactions, like incremental worked before Field.increment feature:
try {
const orderCodesRef = admin.firestore().doc('system/counters/order/codes');
let orderCode = null;
await admin.firestore().runTransaction(async transaction => {
const orderCodesDoc = await transaction.get(orderCodesRef);
if(!orderCodesDoc.exists) {
throw { reason: 'no-order-codes-doc' };
}
let { next } = orderCodesDoc.data();
orderCode = next++;
transaction.update(orderCodesRef, { next });
});
if(orderCode !== null) {
newOrder.code = orderCode;
const orderRef = await admin.firestore().collection('orders').add(newOrder);
return success({ orderId: orderRef.id });
} else {
return fail('no-order-code-result');
}
} catch(error) {
console.error('commitOrder::ERROR', error);
throw errors.CantWriteDatabase({ error });
}
Had the same question and looks like Firestore Python client
doc_ref.update() returns WriteResult that has transform_results attribute with the updated field value
When I use transaction() to update a location, data at that location is returning null even though the location having some data.
I tried transaction() after reading data at the same location that time it is giving all data at that location.
How can I use transaction() if the case is like the above?
Transactions work in the manner of Amazon's SimpleDB or a sharded cluster of databases. That is to say, they are "eventually consistent" rather than guaranteed consistent.
So when you are using transactions, the processing function may get called more than once with a local value (in some cases null if it's never been retrieved) and then again with the synced value (whatever is on the server).
Example:
pathRef.transaction(function(curValue) {
// this part is eventually consistent and may be called several times
}, function(error, committed, ss) {
// this part is guaranteed consistent and will match the final value set
});
This is really the mindset with which you must approach transaction anyways. You should always expect multiple calls, since the first transaction may collide with another change and be rejected. You can't use a transaction's processing method to fetch the server value (although you could read it out of the success callback).
Preventing the locally triggered event
When the transaction happens, a local event is triggered before it reaches the server for latency compensation. If the transaction fails, then the local event will be reverted (a change or remove event is triggered).
You can use the applyLocally property on transactions to override this behavior, which makes the local results slower but ensures that only the server value is triggered locally.
pathRef.transaction(function(curValue) {
// this is still called multiple times
}, function(error, committed, ss) {
// this part is guaranteed consistent and will match the final value set
},
// by providing a third argument of `true`, no local event
// is generated with the locally cached value.
true);
You need to follow this pattern:
var pinRef = firebase.database().ref('vm-pin-generator');
pinRef.transaction(function(oldPin) {
// Check if the result is NOT NULL:
if (oldPin != null) {
return localPinIncrementor(oldPin);
} else {
// Return a value that is totally different
// from what is saved on the server at this address:
return 0;
}
}, function(error, committed, snapshot) {
if (error) {
console.log("error in transaction");
} else if (!committed) {
console.log("transaction not committed");
} else {
console.log("Transaction Committed");
}
}, true);
Firebase usually returns a null value while retrieving a key for the first time but while saving it checks if the new value is similar to older value or not. If not, firebase will run the whole process again, and this time the correct value is returned by the server.
Adding a null check and returning a totally unexpected value (0 in this case) will make firebase run the cycle again.
Simply showing an example implementation to elaborate on #Kato accepted answer above with a custom upsert function:
/**
* Transactional insert or update record
* #param {String} type - object type (table or index) to build lookup path
* #param {String} id - object ID that will be concat with path for lookup
* #param {Object} data - new object (or partial with just edited fields)
* #return {Object} new version of object
*/
const upsert = (type, id, data) => {
return new Promise((resolve, reject) => {
if (!type) {
log.error('Missing db object type')
reject(new TypeError('Missing db object type'))
}
if (!id) {
log.error('Missing db object id')
reject(new TypeError('Missing db object id'))
}
if (!data) {
log.error('Missing db data')
reject(new TypeError('Missing db data'))
}
// build path to resource
const path = `${type}/${id}`
log.debug(`Upserting record '${path}' to database`)
try {
const ref = service.ref(path)
ref.transaction(record => {
if (record === null) {
log.debug(`Creating new record`) // TODO: change to debug
return data
} else if (record) {
log.debug(`Updating existing record`) // TODO: change to debug
const updatedRecord = Object.assign({}, record, data)
return updatedRecord
} else {
return record
}
}, (error, committed, snapshot) => {
if (error) {
log.error(`Error upserting record in database`)
log.error(error.message)
reject(error)
} else if (committed) {
log.debug(`Saved update`)
} else {
log.debug(`Record unchanged`)
}
if (snapshot) {
resolve(snapshot.val())
} else {
log.info(`No snapshot found in transaction so returning original data object`)
resolve(data)
}
})
} catch (error) {
log.error(error)
reject(error)
}
})
}