How to implement transactions in Meteor Method calls - meteor

Suppose I have 2 collections "PlanSubscriptions" and "ClientActivations". I am serially doing a insert on both the collections.
Later one depends on previous one, if any of the transaction fails then the entire operation must rollback.
How can I achieve that in Meteor 1.4?

Since MongoDB doesn't support atomicity, you will have to manage it with Method Chaining.
You can write a method, say, transaction where you will call PlanSubscriptions.insert(data, callback). Then in the callback function you will call ClientActivations.insert(data, callback1) if the first insertion is success and in callback1 return truthy if second insertion is succes, otherwise falsy. If the first insertion returns error you don't need to do anything, but if the second insertion returns error then remove the id got from the insertion in first collection.
I can suggest following structure:
'transaction'(){
PlanSubscriptions.insert(data, (error, result)=>{
if(result){
// result contains the _id
let id_plan = result;
ClientActivations.insert(data, (error, result)=>{
if(result){
// result contains the _id
return true;
}
else if(error){
PlanSubscriptions.remove(id_plan);
return false;
}
})
}
else if(error){
return false;
}
})
}

There is no way to do that in Meteor, since mongodb is not an ACID-compliant database. It has a single-document update atomicity, but not a multiple-document one, which is your case with the two collections.
From the mongo documentation:
When a single write operation modifies multiple documents, the modification of each document is atomic, but the operation as a whole is not atomic and other operations may interleave.
A way to isolate the visibility of your multi-document updates is available, but it's probably not what you need.
Using the $isolated operator, a write operation that affects multiple documents can prevent other processes from interleaving once the write operation modifies the first document. This ensures that no client sees the changes until the write operation completes or errors out.
An isolated write operation does not provide “all-or-nothing” atomicity. That is, an error during the write operation does not roll back all its changes that preceded the error.
However, there are a couple of libraries which try to tackle the problem at the app-level. I recommend taking a look at fawn
In your case, where you have exactly two dependent collections, it's possible to take advantage of the two phase commits technique. Read more about it here: two-phase-commits

Well I figured it out myself.
I added a package babrahams:transactions
At server side Meteor Method call, I called tx Object that is globally exposed by the package. The overall Server Side Meteor.method({}) looks like below.
import { Meteor } from 'meteor/meteor';
import {PlanSubscriptions} from '/imports/api/plansubscriptions/plansubscriptions.js';
import {ClientActivations} from '/imports/api/clientactivation/clientactivations.js';
Meteor.methods({
'createClientSubscription' (subscriptionData, clientActivationData) {
var txid;
try {
txid = tx.start("Adding Subscription to our database");
PlanSubscriptions.insert(subscriptionData, {tx: true})
ClientActivations.insert(activation, {tx: true});
tx.commit();
return true;
} catch(e){
tx.undo(txid);
}
return false;
}
});
With every insert I had added {tx : true}, this concluded it to be a apart of transaction.
Server Console Output:
I20170523-18:43:23.544(5.5)? Started "Adding Subscription to our database" with
transaction_id: vdJQvFgtyZuWcinyF
I20170523-18:43:23.547(5.5)? Pushed insert command to stack: vdJQvFgtyZuWcinyF
I20170523-18:43:23.549(5.5)? Pushed insert command to stack: vdJQvFgtyZuWcinyF
I20170523-18:43:23.551(5.5)? Beginning commit with transaction_id: vdJQvFgtyZuWcinyF
I20170523-18:43:23.655(5.5)? Executed insert
I20170523-18:43:23.666(5.5)? Executed insert
I20170523-18:43:23.698(5.5)? Commit reset transaction manager to clean state
For more Information you can goto link : https://github.com/JackAdams/meteor-transactions
NOTE: I am using Meteor 1.4.4.2

Just sharing this link for future readers:
https://forums.meteor.com/t/solved-transactions-with-mongodb-meteor-methods/48677
import { MongoInternals } from 'meteor/mongo';
// utility async function to wrap async raw mongo operations with a transaction
const runTransactionAsync = async asyncRawMongoOperations => {
// setup a transaction
const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
const session = await client.startSession();
await session.startTransaction();
try {
// running the async operations
let result = await asyncRawMongoOperations(session);
await session.commitTransaction();
// transaction committed - return value to the client
return result;
} catch (err) {
await session.abortTransaction();
console.error(err.message);
// transaction aborted - report error to the client
throw new Meteor.Error('Database Transaction Failed', err.message);
} finally {
session.endSession();
}
};
import { runTransactionAsync } from '/imports/utils'; // or where you defined it
Meteor.methods({
async doSomething(arg) {
// remember to check method input first
// define the operations we want to run in transaction
const asyncRawMongoOperations = async session => {
// it's critical to receive the session parameter here
// and pass it to every raw operation as shown below
const item = await collection1.rawCollection().findOne(arg, { session: session });
const response = await collection2.rawCollection().insertOne(item, { session: session });
// if Mongo or you throw an error here runTransactionAsync(..) will catch it
// and wrap it with a Meteor.Error(..) so it will arrive to the client safely
return 'whatever you want'; // will be the result in the client
};
let result = await runTransactionAsync(asyncRawMongoOperations);
return result;
}
});

Related

Problem with saving data in Transaction in Firebase Firestore for Flutter

I have a problem with transactions in my web application created in Flutter. For database I use Firebase Firestore where I save documents via transaction.
Dependency:
cloud_firestore: 3.1.1
StudentGroup is my main document. It has 4 stages and each of them has 3-5 tasks. (Everything is in 1 document). I have to store game timer, so every 10 seconds I make an request to save time for current stage. (Every stage has different timer). I have a problem with saving task, because "Sometimes" when 2 requests are made in the same time, then I get some weird state manipulation.
Task is updated and "isFinished" is set to true
Timer is updated to correct value (with this update somehow previous task update is lost, "isFinished" is set to false
This is how I save task.
Future<Result> saveTask({required String sessionId, required String studentGroupId,
required Task task}) async {
print("trying to save task <$task>.");
try {
return await _firebaseFirestore.runTransaction((transaction) async {
final studentGroupRef = _getStudentGroupDocumentReference(
sessionId: sessionId,
studentGroupId: studentGroupId
);
final sessionGroupDoc = await studentGroupRef.get();
if (!sessionGroupDoc.exists) {
return Result.error("student group not exists");
}
final sessionGroup = StudentGroup.fromSnapshot(sessionGroupDoc);
sessionGroup.game.saveTask(task);
transaction.set(studentGroupRef, sessionGroup.toJson());
})
.then((value) => taskFunction(true))
.catchError((error) => taskFunction(false));
} catch (error) {
return Result.error("Error couldn't save task");
}
}
This is how I save my time
Future<Result> updateTaskTimer({required String sessionId,
required String studentGroupId, required Duration duration}) async {
print("trying to update first task timer");
try {
return await _firebaseFirestore.runTransaction((transaction) async {
final studentGroupRef = _getStudentGroupDocumentReference(
sessionId: sessionId,
studentGroupId: studentGroupId
);
final sessionGroupDoc = await studentGroupRef.get();
if (!sessionGroupDoc.exists) {
return Result.error("student group not exists");
}
final sessionGroup = StudentGroup.fromSnapshot(sessionGroupDoc);
switch (sessionGroup.game.gameStage) {
case GameStage.First:
sessionGroup.game.stages.first.duration = duration.inSeconds;
break;
case GameStage.Second:
sessionGroup.game.stages[1].duration = duration.inSeconds;
break;
case GameStage.Third:
sessionGroup.game.stages[2].duration = duration.inSeconds;
break;
case GameStage.Fourth:
sessionGroup.game.stages[3].duration = duration.inSeconds;
break;
case GameStage.Fifth:
sessionGroup.game.stages[4].duration = duration.inSeconds;
break;
}
transaction.set(
studentGroupRef,
sessionGroup.toJson(),
SetOptions(merge: true)
);
print("Did I finish task 4? ${sessionGroup.game.stages.first.tasks[3].isFinished}");
})
.then((value) => timerFunction(true))
.catchError((error) => timerFunction(false));
} catch (error) {
return Result.error("Error couldn't update task timer");
}
}
timerFunction and taskFunction print some messages in console and return Result.error or Result.success (for now it returns bool)
I don't know If I am doing something wrong with Firebase Firestore Transaction. I would like to have atomic operations for reading and writing data.
Transactions ensure atomicity - which means that if the transaction succeeds then all the reads and writes occur in a non-overlapping way with other transactions. This prevents the type of problem you are describing.
But this doesn't work if you spread your reads and writes over multiple transactions. In particular, it looks to me like you are writing a task which was obtained from outside the transaction. Instead, you should use ids or similar to track which documents you need to update, then do a read and a write inside the transaction.Alternatively firebase also provides Batched Writes, which specify the specific properties you want to update. These will ensure that any other properties are not changed.For batch writes example you can refer to the link

Am I doing Firestore Transactions correct?

I've followed the Firestore documentation with relation to transactions, and I think I have it all sorted correctly, but in testing I am noticing issues with my documents not getting updated properly sometimes. It is possible that multiple versions of the document could be submitted to the function in a very short interval, but I am only interested in only ever keeping the most recent version.
My general logic is this:
New/Updated document is sent to cloud function
Check if document already exists in Firestore, and if not, add it.
If it does exist, check that it is "newer" than the instance in firestore, if it is, update it.
Otherwise, don't do anything.
Here is the code from my function that attempts to accomplish this...I would love some feedback if this is correct/best way to do this:
const ocsFlight = req.body;
const procFlight = processOcsFlightEvent(ocsFlight);
try {
const ocsFlightRef = db.collection(collection).doc(procFlight.fltId);
const originalFlight = await ocsFlightRef.get();
if (!originalFlight.exists) {
const response = await ocsFlightRef.set(procFlight);
console.log("Record Added: ", JSON.stringify(procFlight));
res.status(201).json(response); // 201 - Created
return;
}
await db.runTransaction(async (t) => {
const doc = await t.get(ocsFlightRef);
const flightDoc = doc.data();
if (flightDoc.recordModified <= procFlight.recordModified) {
t.update(ocsFlightRef, procFlight);
console.log("Record Updated: ", JSON.stringify(procFlight));
res.status(200).json("Record Updated");
return;
}
console.log("Record isn't newer, nothing changed.");
console.log("Record:", JSON.stringify("Same Flight:", JSON.stringify(procFlight)));
res.status(200).json("Record isn't newer, nothing done.");
return;
});
} catch (error) {
console.log("Error:", JSON.stringify(error));
res.status(500).json(error.message);
}
The Bugs
First, you are trusting the value of req.body to be of the correct shape. If you don't already have type assertions that mirror your security rules for /collection/someFlightId in processOcsFlightEvent, you should add them. This is important because any database operations from the Admin SDKs will bypass your security rules.
The next bug is sending a response to your function inside the transaction. Once you send a response back the client, your function is marked inactive - resources are severely throttled and any network requests may not complete or crash. As a transaction may be retried a handful of times if a database collision is detected, you should make sure to only respond to the client once the transaction has properly completed.
You use set to write the new flight to Firestore, this can lead to trouble when working with transactions as a set operation will cancel all pending transactions at that location. If two function instances are fighting over the same flight ID, this will lead to the problem where the wrong data can be written to the database.
In your current code, you return the result of the ocsFlightRef.set() operation to the client as the body of the HTTP 201 Created response. As the result of the DocumentReference#set() is a WriteResult object, you'll need to properly serialize it if you want to return it to the client and even then, I don't think it will be useful as you don't seem to use it for the other response types. Instead, a HTTP 201 Created response normally includes where the resource was written to as the Location header with no body, but here we'll pass the path in the body. If you start using multiple database instances, including the relevant database may also be useful.
Fixing
The correct way to achieve the desired result would be to do the entire read->check->write process inside of a transaction and only once the transaction has completed, then respond to the client.
So we can send the appropriate response to the client, we can use the return value of the transaction to pass data out of it. We'll pass the type of the change we made ("created" | "updated" | "aborted") and the recordModified value of what was stored in the database. We'll return these along with the resource's path and an appropriate message.
In the case of an error, we'll return a message to show the user as message and the error's Firebase error code (if available) or general message as the error property.
// if not using express to wrangle requests, assert the correct method
if (req.method !== "POST") {
console.log(`Denied ${req.method} request`);
res.status(405) // 405 - Method Not Allowed
.set("Allow", "POST")
.end();
return;
}
const ocsFlight = req.body;
try {
// process AND type check `ocsFlight`
const procFlight = processOcsFlightEvent(ocsFlight);
const ocsFlightRef = db.collection(collection).doc(procFlight.fltId);
const { changeType, recordModified } = await db.runTransaction(async (t) => {
const flightDoc = await t.get(ocsFlightRef);
if (!flightDoc.exists) {
t.set(ocsFlightRef, procFlight);
return {
changeType: "created",
recordModified: procFlight.recordModified
};
}
// only parse the field we need rather than everything
const storedRecordModified = flightDoc.get('recordModified');
if (storedRecordModified <= procFlight.recordModified) {
t.update(ocsFlightRef, procFlight);
return {
changeType: "updated",
recordModified: procFlight.recordModified
};
}
return {
changeType: "aborted",
recordModified: storedRecordModified
};
});
switch (changeType) {
case "updated":
console.log("Record updated: ", JSON.stringify(procFlight));
res.status(200).json({ // 200 - OK
path: ocsFlightRef.path,
message: "Updated",
recordModified,
changeType
});
return;
case "created":
console.log("Record added: ", JSON.stringify(procFlight));
res.status(201).json({ // 201 - Created
path: ocsFlightRef.path,
message: "Created",
recordModified,
changeType
});
return;
case "aborted":
console.log("Outdated record discarded: ", JSON.stringify(procFlight));
res.status(200).json({ // 200 - OK
path: ocsFlightRef.path,
message: "Record isn't newer, nothing done.",
recordModified,
changeType
});
return;
default:
throw new Error("Unexpected value for 'changeType': " + changeType);
}
} catch (error) {
console.log("Error:", JSON.stringify(error));
res.status(500) // 500 - Internal Server Error
.json({
message: "Something went wrong",
// if available, prefer a Firebase error code
error: error.code || error.message
});
}
References
Cloud Firestore Transactions
Cloud Firestore Node SDK Reference
HTTP Event Cloud Functions

Add new function in promise.all runtime in node

I am not sure about this question if this can be implemented or not.
I am using node.js with express.js and MySQL database.
I have a few records in MySQL database. These records are updating continues.
So, suppose I fetch some records from MySQL and start operations on each record with Promise.all using demoFunction function which is returned promise.
In this function, I am trying to check for new records in MySQL database. If I got new records then I want to push this new record's operation into current Promise.all queue. Is this possible? If not possible then how can I achieve this goal with continues execution?
So, my code is like,
const demoFunction = (arg1, arg2) => {
checkForNewData();
return new Promise((resolve, reject) => {
// Rest of my code is here for this function
// This function will be take around 5 to 10 mins
});
};
const dataFromDatabase = "Here i'm getting some data into array of object from SQL database";
let allPromises = dataFromDatabase.map((obj) => demoFunction(obj.arg1, obj.arg1));
const checkForNewData = () => {
const newDataFromDatabase = "Here i'm getting some new data into array of object from SQL database";
for (let i = 0; i < newDataFromDatabase.length; i++) {
allPromises.push(demoFunction(newDataFromDatabase[i].arg1, newDataFromDatabase[i].arg2));
}
};
return Promise.all(allPromises)
.then(() => {
// response
})
.catch((e) => {
console.log(e);
})
In this function, I am trying to check for new records in MySQL database. If I got new records then I want to push this new record's operation into current Promise.all queue. Is this possible?
Nope, Promise.all takes a finite and set number of promises and waits for all of them to complete.
If not possible then how can I achieve this goal with continues execution?
Well, a promise is just a value - if you have a promise for something then execution has already started somewhere else. You can always execute a second .all but what happens if records were added in the meantime?
It's fine to do:
Promise.all(allPromises).then(() => Promise.all(allPromises)).then(() => {
});
But at that point you're better off just waiting for the checkNewData call to finish before calling the Promise.all since otherwise you're introducing a race between checkAllData and the Promise.all
A promise is a "one time" thing, consider using an async iterator if you want to process results (note, this requires Node 12):
async function* getRecordData() {
for await(const item in getPromisesOfInitDataFromDatabase()) {
yield item; // or process it
}
while(true) { // or how often you want
for await(const item of getNewDastaFromDatabase()) {
yield item; // or process it
}
await sleep(3000); // or some sleep timeout to not constantly poll
}
}
Then elsewhere:
(async () => {
for await(const item of getRecordData()) {
// items are available here one by one, including new items in the database
}
})();

Is there a way to prevent having to await an async method returning a stream?

We currently have a method that returns a Future<Stream<Position>> just because internally we have to await the result of a method returning a Future before we can call another method that returns the Stream<Position> which we are actually interested in. Here is the code:
Future<Stream<Position>> getPositionStream(
[LocationOptions locationOptions = const LocationOptions()]) async {
PermissionStatus permission = await _getLocationPermission();
if (permission == PermissionStatus.granted) {
if (_onPositionChanged == null) {
_onPositionChanged = _eventChannel
.receiveBroadcastStream(
Codec.encodeLocationOptions(locationOptions))
.map<Position>(
(element) => Position._fromMap(element.cast<String, double>()));
}
return _onPositionChanged;
} else {
_handleInvalidPermissions(permission);
}
return null;
}
So what happens here is:
We await the _getLocationPermission() method so that we can test if the user grants us permission to access to the location services on their device (Android or iOS);
If the user grants us permission we return a Stream<Position> which will update every time the device registers a location change.
I have the feeling we can also handle this without doing an await and returning a Future. Something along the lines of:
Manually create and return an instance of the Stream<Position> class;
Handle the logic of checking the permissions and calling the _eventChannel.receiveBroadcastStream in the then() method of the Future<PermissionStatus> returned from the _getLocationPermission() method (so we don't have to await it);
Copy the events send on the stream from the _eventChannel.receiveBroadcastStream onto the earlier created (and returned) stream.
Somehow this seems to be possible, but also includes some overhead in managing the stream and make sure it closes and is cleaned up correctly during the live cycle of the plugin or when the user unsubscribes pass through the events to the _eventChannel etc.
So I guess the question would be, what would be the best way to approach this situation?
You can write the code as an async* function, which will return a Stream and still allows await in the body:
Stream<Position> getPositionStream(
[LocationOptions locationOptions = const LocationOptions()]) async* {
PermissionStatus permission = await _getLocationPermission();
if (permission == PermissionStatus.granted) {
if (_onPositionChanged == null) {
_onPositionChanged = _eventChannel
.receiveBroadcastStream(
Codec.encodeLocationOptions(locationOptions))
.map<Position>(
(element) => Position._fromMap(element.cast<String, double>()));
}
yield* _onPositionChanged;
} else {
_handleInvalidPermissions(permission);
}
}
Alternatively, if you are using a non-async function, you can also use StreamCompleter from package:async.
It allows you to return a Stream now, even if you only get the real stream later. When that happens, you "complete" the StreamCompleter with the real stream, and the original stream will behave as if it was the real stream.

Firebase Function Get Single Value From Database

I want to get a single value from Firebase Database within Firebase function. However, the Promise never returns and the chained method never executes. Here is the method that fetches a value from the database
function getFcmToken(username){
return admin.database().ref('tokens/{username}/fcmToken').once('value').then(snap => {
if(snap.exists()){
const token = Object.keys(snap.val());
console.log("FCM Token", token);
return token;
}
return [];
});
}
The above method was supposed to return a token, but I am not sure it is, so the method below does not get executed.
function sendNotification(keyword, username){
return getFcmToken(username).then(token => {
if(token.length > 0){
//Notification Detail
let payload = {
data:{
keyword: keyword
}
};
return admin.messaging().sendToDevice(token, payload);
}
});
}
In the console log, all I see is Promise pending.
How can I update the code above to return a single value, it appears it is returning an array?
Thanks
Your database ref path is wrong. You might wanted to replace username in path, but single quoted won't do that.
Firebase is listening on tokens/{username}/fcmToken, which doesn't exists. Hence on value event will not be triggered and so downline callback will not be executed.
You can use Template literals for building dynamic strings.
Try ref path as
`tokens/${username}/fcmToken`
Code:
function getFcmToken(username){
return admin.database().ref(`tokens/${username}/fcmToken`).once(...)
}

Resources