How can I use AQL with multiple queries that using the result of one another? - graph

I have 2 vertices and an edge named user, device, ownership respectively.
My business logic is when I receive device information, I upsert it with dateCreated and dateUpdated fields added. If I inserted that device then I insert new user with default values and create edge connection to it. If I update I simple return already connected user as a result.
Without losing atomicity how can I achieve this?
I tried single AQL query but without condition it is not possible it seems and traversal also is not supported with insert/update operation.
I can do separate queries but that loses atomicity.
var finalQuery = aql`
UPSERT ${deviceQuery}
INSERT MERGE(${deviceQuery},{dateCreated:DATE_NOW()})
UPDATE MERGE(${deviceQuery},{dateUpdated:DATE_NOW()})
IN ${this.DeviceModel}
RETURN { doc: NEW, type: OLD ? 'update' : 'insert' }`;
var cursor = await db.query(finalQuery);
var result = await cursor.next();
if (result.type == 'insert') {
console.log('Inserted documents')
finalQuery = aql`
LET user=(INSERT {
"_key":UUID(),
"name": "User"
} INTO user
RETURN NEW)
INSERT {
_from:${result.doc._id},
_to:user[0]._id,
"type": "belongs"
}INTO ownership
return user[0]`;
cursor = await db.query(finalQuery);
result = await cursor.next();
console.log('New user:',result);
}

You can try something like this
Upsert ....
FILTER !OLD
Let model = NEW
LET user= First(INSERT {
"_key":UUID(),
"name": "User"
} INTO user
RETURN NEW)
INSERT {
_from:model._id,
_to:user._id,
"type": "belongs"
}INTO ownership
return user

I end up separating the modification and selection queries.
var finalQuery = aql`
LET device=(
UPSERT ${deviceQuery}
INSERT MERGE(${deviceQuery},{dateCreated:DATE_NOW()})
UPDATE MERGE(${deviceQuery},{dateUpdated:DATE_NOW()})
IN ${this.DeviceModel}
RETURN { doc: NEW, type: OLD ? 'update' : 'insert' })
FILTER device[0].type=='insert'
LET user=(INSERT {
"_key":UUID(),
"name": "User"
} INTO user
RETURN NEW)
INSERT {
_from:device[0].doc._id,
_to:user[0]._id,
"type": "belongs"
}INTO ownership
return user[0]`;
var cursor = await db.query(finalQuery);
var result = await cursor.next();
if (result == null) {
const deviceId=this.DeviceModel.name+"/"+queryParams._key;
finalQuery = aql`
FOR v,e,p IN 1..1
OUTBOUND ${deviceId} ownership
FILTER e.type=="belongs"
RETURN v `;
cursor = await db.query(finalQuery);
result = await cursor.next();
isUpdate=true;
}
This way I ensure the atomicity. There are improvements for controling if cursor.extra.stats.writesExecuted true etc.

Related

How to prevent Firestore write race conditions for a reservation button

Summary
I'm developing an application where users can reserve and cancel reservations for classes. In a ReservationButtonView I two buttons that add and remove a user to a workout class respectively. Currently the button I show is based off whether the user's Firebase Auth uid is listed in a Firestore document.
I was having issues when rapidly tapping on the reservation button. Specifically, the reservationCnt would become inaccurate by showing more or less than the actual users reserved for a class.
The only way I have found to resolve this is be using a Firestore transaction that checks to see if a user is in a workout class already. If they are, addReservation() now does nothing. If they aren't, removeReservation() would also do nothing.
At first I thought I could just disable the button and via the logic still in place the code below (.disabled()), but that alone didn't work as I ran into the above described race conditions. What I found out is that arrayUnion and arrayRemove still succeed even when the object I'm looking to add is there and not there respectively. Meaning it is possible for my transaction to not remove a reservedUser that isn't there and also decrease the reservationCnt which can leave me with say no reserved users and a reservationCnt of -1
The Ask
Is there a better way to handle this reservation process? Can I accomplish this without a transaction for at least the removal of users in some way. Ideally, I'd like to have a spinner replace the button as I add or remove a user's reservation to indicate to the user that the app is processing the request. Perhaps I need two variables to manage the disabled() state instead of one?
MVVM Code Snippets
NOTE: I pulled out some button styling to make the code a bit less verbose
ReservationButtonView
struct ReservationButtonView: View {
var workoutClass: WorkoutClass
#ObservedObject var viewModel: WorkoutClassViewModel
#EnvironmentObject var authViewModel: AuthViewModel
var body: some View {
if checkIsReserved(uid: authViewModel.user?.uid ?? "", reservedUsers: workoutClass.reservedUsers ?? []) {
Button(action: {
viewModel.isDisabled = true
viewModel.removeReservation(
documentId: workoutClass.id!,
reservedUserDetails: ["uid": authViewModel.user?.uid as Any, "photoURL": authViewModel.user?.photoURL?.absoluteString ?? "" as Any, "displayName": authViewModel.user?.displayName ?? "Bruin Fitness Member" as Any],
uid: authViewModel.user?.uid ?? "")
}){
Label(
title: { Text("Cancel Reservation")
.font(.title) },
icon: { Image(systemName: "person.badge.minus")
.font(.title) }
)
}.disabled(viewModel.isDisabled)
} else{
Button(action: {
viewModel.isDisabled = true
viewModel.addReservation(
documentId: workoutClass.id!,
reservedUserDetails: ["uid": authViewModel.user?.uid as Any, "photoURL": authViewModel.user?.photoURL?.absoluteString ?? "" as Any, "displayName": authViewModel.user?.displayName ?? "Bruin Fitness Member" as Any],
uid: authViewModel.user?.uid ?? "")
}){
Label(
title: { Text("Reserve")
.font(.title) },
icon: { Image(systemName: "person.badge.plus")
.font(.title) }
)
}
.disabled(viewModel.isDisabled)
}
}
}
func checkIsReserved(uid: String, reservedUsers: [reservedUser]) -> Bool {
return reservedUsers.contains { $0.uid == uid }
}
WorkoutClassModel
struct reservedUser: Codable, Identifiable {
var id: String = UUID().uuidString
var uid: String
var photoURL: URL?
var displayName: String?
enum CodingKeys: String, CodingKey {
case uid
case photoURL
case displayName
}
}
struct WorkoutClass: Codable,Identifiable {
#DocumentID var id: String?
var reservationCnt: Int
var time: String
var workoutType: String
var reservedUsers: [reservedUser]?
enum CodingKeys: String, CodingKey {
case id
case reservationCnt
case time
case workoutType
case reservedUsers
}
}
WorkoutClassViewModel
class WorkoutClassViewModel: ObservableObject {
#Published var isDisabled = false
private var db = Firestore.firestore()
func addReservation(documentId: String, reservedUserDetails: [String: Any], uid: String){
let incrementValue: Int64 = 1
let increment = FieldValue.increment(incrementValue)
let addUser = FieldValue.arrayUnion([reservedUserDetails])
let classReference = db.document("schedules/Redwood City/dates/\(self.stateDate.dbDateFormat)/classes/\(documentId)")
db.runTransaction { transaction, errorPointer in
let classDocument: DocumentSnapshot
do {
print("Getting classDocument for docId: \(documentId) in addReservedUser()")
try classDocument = transaction.getDocument(classReference)
} catch let fetchError as NSError {
errorPointer?.pointee = fetchError
return nil
}
guard let workoutClass = try? classDocument.data(as: WorkoutClass.self) else {
let error = NSError(
domain: "AppErrorDomain",
code: -3,
userInfo: [
NSLocalizedDescriptionKey: "Unable to retrieve workoutClass from snapshot \(classDocument)"
]
)
errorPointer?.pointee = error
return nil
}
let isReserved = self.checkIsReserved(uid: uid, reservedUsers: workoutClass.reservedUsers ?? [])
if isReserved {
print("user is already in class so therefore can't be added again")
return nil
} else {
transaction.updateData(["reservationCnt": increment, "reservedUsers": addUser], forDocument: classReference)
return nil
}
} completion: { object, error in
if let error = error {
print(error.localizedDescription)
self.isDisabled = false
} else {
print("Successfully ran transaction with object: \(object ?? "")")
self.isDisabled = false
}
}
}
func removeReservation(documentId: String, reservedUserDetails: [String: Any], uid: String){
let decrementValue: Int64 = -1
let decrement = FieldValue.increment(decrementValue)
let removeUser = FieldValue.arrayRemove([reservedUserDetails])
let classReference = db.document("schedules/Redwood City/dates/\(self.stateDate.dbDateFormat)/classes/\(documentId)")
db.runTransaction { transaction, errorPointer in
let classDocument: DocumentSnapshot
do {
print("Getting classDocument for docId: \(documentId) in addReservedUser()")
try classDocument = transaction.getDocument(classReference)
} catch let fetchError as NSError {
errorPointer?.pointee = fetchError
return nil
}
guard let workoutClass = try? classDocument.data(as: WorkoutClass.self) else {
let error = NSError(
domain: "AppErrorDomain",
code: -3,
userInfo: [
NSLocalizedDescriptionKey: "Unable to retrieve reservedUsers from snapshot \(classDocument)"
]
)
errorPointer?.pointee = error
return nil
}
let isReserved = self.checkIsReserved(uid: uid, reservedUsers: workoutClass.reservedUsers ?? [] )
if isReserved {
transaction.updateData(["reservationCnt": decrement, "reservedUsers": removeUser], forDocument: classReference)
return nil
} else {
print("user not in class so therefore can't be removed")
return nil
}
} completion: { object, error in
if let error = error {
print(error.localizedDescription)
self.isDisabled = false
} else {
print("Successfully ran removeReservation transaction with object: \(object ?? "")")
self.isDisabled = false
}
}
}
func checkIsReserved(uid: String, reservedUsers: [reservedUser]) -> Bool {
return reservedUsers.contains { $0.uid == uid }
}
}
App screenshot
Reservation button is the green/grey button at the bottom of the view
As this is a race condition, You have already acknowledged the use of Transactions for the update which is the most desirable as this can ensure the update is successful before allowing the App to change button status.
I.e. by using a transaction and only updating the UI Button state on success, which is explained here
The recommendation is to keep the state of the button mapped to what is in the document, therefore you are likely to exceed rate limits by updating the same field continuously based on the flipping of the button.
Another way to handle this tracking of the state of enrollment is to add a new document that indicates the state of the enrollment for the user to a collection that is the class they are enrolling in.
I.e. Rather than having the class user enrolling into being a document, make that a collection and each time the enrollment state changes, write a new document. This will allow for updates to occur without using transactions and the current state of enrollments is contained within the latest document. This latest document can be read and used as the status of the button within the App with the added benefit that the state will always update to the status contained within Firestore.
I ended up resolving this by adding a disable check conditional before the conditional that decides whether to show the "Reserve" or "Cancel" button.
This way when my Firestore transaction is running the user will see a spinner instead and can't monkey test the button. The spinner helps to show that the reservation operation is in progress. When the transaction hits its completion block I disable the isDisabled Bool and the listener is in sync (the user then sees the newly toggled button state)
if workoutClassVM.isDisabled {
ProgressView()
.progressViewStyle(CircularProgressViewStyle(tint: Color("bruinGreenColor")))
} else if checkIsReserved(uid: authVM.user?.uid ?? "", reservedUsers: workoutClass.reservedUsers ?? []) {
...

2 updates within one Firestore transaction

I am new to Firestore transaction, and would like to update a document field based the current data of the document.
My planned transaction is given below:
const cityRef = db.collection('cities').doc('SF');
try {
await db.runTransaction(async (t) => {
const doc = await t.get(cityRef);
let status = doc.data().myStatus;
if (status == "one") {
throw "err";
} else {
// run some function - next status is based on the return
let output = await someFunction();
if (output) {
await t.update(cityRef, { myStatus: "two" });
return output;
} else {
await t.update(cityRef, { myStatus: "three" });
return output;
}
}
});
console.log("transaction successful");
} catch (err) {
console.log("Alreadu updated");
output = "one";
return output;
}
My queries are given below:
As per the documentation I have returned the data after update, however it does not seem to be working as expected.
Can we have 2 updates within one single transaction (both are updating the same field in the firestore)?
Thank you
You make the following clarification in the comments above:
someFunction() does some processing on other firestore
collection/documents (not the one I am updating) and returns either
true or false.
As you read in the doc on Transactions, "Read operations must come before write operations". If you want to read some docs in the transaction, you need to use the get() method of the Transaction, like you did with the first document. You cannot call a function that is using other Firestore methods like the get() method of a DocumentReference.

Query size limits in DynamoDB

I don't get the concept of limits for query/scan in DynamoDb.
According to the docs:
A single Query operation can retrieve a maximum of 1 MB of data.This
limit applies before any FilterExpression is applied to the results.
Let's say I have 10k items, 250kb per item, all of them fit query params.
If I run a simple query, I get only 4 items?
If I use ProjectionExpression to retrieve only single attribute (1kb
in size), will I get 1k items?
If I only need to count items (select: 'COUNT'), will it count all
items (10k)?
If I run a simple query, I get only 4 items?
Yes
If I use ProjectionExpression to retrieve only single attribute (1kb in size), will I get 1k items?
No, filterexpressions and projectexpressions are applied after the query has completed. So you still get 4 items.
If I only need to count items (select: 'COUNT'), will it count all items (10k)?
No, still just 4
The thing that you are probably missing here is that you can still get all 10k results, or the 10k count, you just need to get the results in pages. Some details here. Basically when you complete your query, check the LastEvaluatedKey attribute, and if its not empty, get the next set of results. Repeat this until the attribute is empty and you know you have all the results.
EDIT: I should say some of the SDKs abstract this away for you. For example the Java SDK has query and queryPage, where query will go back to the server multiple times to get the full result set for you (i.e. in your case, give you the full 10k results).
For any operation that returns items, you can request a subset of attributes to retrieve; however, doing so has no impact on the item size calculations. In addition, Query and Scan can return item counts instead of attribute values. Getting the count of items uses the same quantity of read capacity units and is subject to the same item size calculations. This is because DynamoDB has to read each item in order to increment the count.
Managing Throughput Settings on Provisioned Tables
Great explanation by #f-so-k.
This is how I am handling the query.
import AWS from 'aws-sdk';
async function loopQuery(params) {
let keepGoing = true;
let result = null;
while (keepGoing) {
let newParams = params;
if (result && result.LastEvaluatedKey) {
newParams = {
...params,
ExclusiveStartKey: result.LastEvaluatedKey,
};
}
result = await AWS.query(newParams).promise();
if (result.count > 0 || !result.LastEvaluatedKey) {
keepGoing = false;
}
}
return result;
}
const params = {
TableName: user,
IndexName: 'userOrder',
KeyConditionExpression: 'un=:n',
ExpressionAttributeValues: {
':n': {
S: name,
},
},
ConsistentRead: false,
ReturnConsumedCapacity: 'NONE',
ProjectionExpression: ALL,
};
const result = await loopQuery(params);
Edit:
import AWS from 'aws-sdk';
async function loopQuery(params) {
let keepGoing = true;
let result = null;
let list = [];
while (keepGoing) {
let newParams = params;
if (result && result.LastEvaluatedKey) {
newParams = {
...params,
ExclusiveStartKey: result.LastEvaluatedKey,
};
}
result = await AWS.query(newParams).promise();
if (result.count > 0 || !result.LastEvaluatedKey) {
keepGoing = false;
list = [...list, ...result]
}
}
return list;
}
const params = {
TableName: user,
IndexName: 'userOrder',
KeyConditionExpression: 'un=:n',
ExpressionAttributeValues: {
':n': {
S: name,
},
},
ConsistentRead: false,
ReturnConsumedCapacity: 'NONE',
ProjectionExpression: ALL,
};
const result = await loopQuery(params);

Most efficient way to ensure user owns document on update?

I'm using Meteor methods to update documents so I can share them easier and have more control. However i've ran into a problem with checking ownership.
How should I check to make sure the user calling the update method is the owner of the document? Currently i'm grabbing the document first then running the update.
Is there a better pattern to accomplish this?
Meteor.methods({
'Listing.update': function(docId, data) {
var doc = db.listings.findOne({_id: docId}) || {};
if (doc.userId !== this.userId) {
throw new Meteor.Error(504, "You don't own post");
}
// ensure data is the type we expect
check(data, {
title: String,
desc: String
});
return db.listings.update(docId, {$set: data});
}
});
You don't need the additional db call to fetch the original doc, just make the userId an additional criteria in the update selector. If no doc exists with the correct _id and userId no update will be done. update returns the number of docs updated so it will return 1 on success and 0 on failure.
like this:
'Listing.update': function(docId, data) {
var self = this;
check(data, {
title: String,
desc: String
});
if ( ! self.userId )
throw new Meteor.Error(500, 'Must be logged in to update listing');
res = db.listings.update({_id: docId, userId: self.userId}, {$set: data});
if ( res === 0 )
throw new Meteor.Error( 504, "You do not own a post with that id" );
return res;
}
Also, if you use findOne to check a document's existence, use the fields option to limit what you return from the db. Usually just {fields: {_id:1}}.

async nodejs querying and processing results

I have an array of objects taken from mongodb. Every element in the array is a post, with author as user_id. Now i wish to find the user info related to the user_id.
Since node uses async methods to find the data from db, the forEach loop finishes before the callbacks finish.
docs.forEach(function(doc, index){
//get the user for this doc
User.find({_id: mongo.BSONPure.ObjectID(doc.user_id)}, {name: 1, username: 1, email: 1}).skip(0).limit(1).toArray(function(err, user){
user = user[0]
if(err) {
throw new Error(err)
} else {
docs[index].user = user
if(doc.for_id) {
User.find({_id: mongo.BSONPure.ObjectID(doc.for_id)}, {name: 1, username: 1, email: 1}).skip(0).limit(1).toArray(function(err, for_user){
for_user = for_user[0]
if(err) {
throw new Error(err)
} else {
docs[index].for_user = for_user
}
})
}
}
})
})
So at the end of this loop, if i send a cb(docs), docs do not have the user and for_user attribute. How do I overcome this?
Use Step for node.js. It will run your functions in serial order
var Step = require('step');
Step( docs.forEach(...), function() { cb(docs); } );
Or if you know the total number of records, you can call the callback when you're done processing the last one. Something like this
var count = docs.count(); // or something
var processed = 0;
docs.forEach(... if (++processed == count) cb(docs); );

Resources