I want to upload csv file and insert it into db after huge validation and some process. I am using fibers for asynchronous call. But the fiber process keep executing even fiber returns future.wait();.
Meteor.methods({
uploadCSV: (calender) {
if (Meteor.isServer) {
var Future = Npm.require('fibers/future');
var future = new Future();
calender = parse(calender);
future["return"](saveCalender(calender)); //huge process and validation
console.log(future); // { value: 1, resolved: true }
return future.wait();
}
}
});
saveCalender function executes every time. Did I missed anything to stop fiber execution or how do I stop fiber execution?
Related
Suppose I have 2 collections "PlanSubscriptions" and "ClientActivations". I am serially doing a insert on both the collections.
Later one depends on previous one, if any of the transaction fails then the entire operation must rollback.
How can I achieve that in Meteor 1.4?
Since MongoDB doesn't support atomicity, you will have to manage it with Method Chaining.
You can write a method, say, transaction where you will call PlanSubscriptions.insert(data, callback). Then in the callback function you will call ClientActivations.insert(data, callback1) if the first insertion is success and in callback1 return truthy if second insertion is succes, otherwise falsy. If the first insertion returns error you don't need to do anything, but if the second insertion returns error then remove the id got from the insertion in first collection.
I can suggest following structure:
'transaction'(){
PlanSubscriptions.insert(data, (error, result)=>{
if(result){
// result contains the _id
let id_plan = result;
ClientActivations.insert(data, (error, result)=>{
if(result){
// result contains the _id
return true;
}
else if(error){
PlanSubscriptions.remove(id_plan);
return false;
}
})
}
else if(error){
return false;
}
})
}
There is no way to do that in Meteor, since mongodb is not an ACID-compliant database. It has a single-document update atomicity, but not a multiple-document one, which is your case with the two collections.
From the mongo documentation:
When a single write operation modifies multiple documents, the modification of each document is atomic, but the operation as a whole is not atomic and other operations may interleave.
A way to isolate the visibility of your multi-document updates is available, but it's probably not what you need.
Using the $isolated operator, a write operation that affects multiple documents can prevent other processes from interleaving once the write operation modifies the first document. This ensures that no client sees the changes until the write operation completes or errors out.
An isolated write operation does not provide “all-or-nothing” atomicity. That is, an error during the write operation does not roll back all its changes that preceded the error.
However, there are a couple of libraries which try to tackle the problem at the app-level. I recommend taking a look at fawn
In your case, where you have exactly two dependent collections, it's possible to take advantage of the two phase commits technique. Read more about it here: two-phase-commits
Well I figured it out myself.
I added a package babrahams:transactions
At server side Meteor Method call, I called tx Object that is globally exposed by the package. The overall Server Side Meteor.method({}) looks like below.
import { Meteor } from 'meteor/meteor';
import {PlanSubscriptions} from '/imports/api/plansubscriptions/plansubscriptions.js';
import {ClientActivations} from '/imports/api/clientactivation/clientactivations.js';
Meteor.methods({
'createClientSubscription' (subscriptionData, clientActivationData) {
var txid;
try {
txid = tx.start("Adding Subscription to our database");
PlanSubscriptions.insert(subscriptionData, {tx: true})
ClientActivations.insert(activation, {tx: true});
tx.commit();
return true;
} catch(e){
tx.undo(txid);
}
return false;
}
});
With every insert I had added {tx : true}, this concluded it to be a apart of transaction.
Server Console Output:
I20170523-18:43:23.544(5.5)? Started "Adding Subscription to our database" with
transaction_id: vdJQvFgtyZuWcinyF
I20170523-18:43:23.547(5.5)? Pushed insert command to stack: vdJQvFgtyZuWcinyF
I20170523-18:43:23.549(5.5)? Pushed insert command to stack: vdJQvFgtyZuWcinyF
I20170523-18:43:23.551(5.5)? Beginning commit with transaction_id: vdJQvFgtyZuWcinyF
I20170523-18:43:23.655(5.5)? Executed insert
I20170523-18:43:23.666(5.5)? Executed insert
I20170523-18:43:23.698(5.5)? Commit reset transaction manager to clean state
For more Information you can goto link : https://github.com/JackAdams/meteor-transactions
NOTE: I am using Meteor 1.4.4.2
Just sharing this link for future readers:
https://forums.meteor.com/t/solved-transactions-with-mongodb-meteor-methods/48677
import { MongoInternals } from 'meteor/mongo';
// utility async function to wrap async raw mongo operations with a transaction
const runTransactionAsync = async asyncRawMongoOperations => {
// setup a transaction
const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
const session = await client.startSession();
await session.startTransaction();
try {
// running the async operations
let result = await asyncRawMongoOperations(session);
await session.commitTransaction();
// transaction committed - return value to the client
return result;
} catch (err) {
await session.abortTransaction();
console.error(err.message);
// transaction aborted - report error to the client
throw new Meteor.Error('Database Transaction Failed', err.message);
} finally {
session.endSession();
}
};
import { runTransactionAsync } from '/imports/utils'; // or where you defined it
Meteor.methods({
async doSomething(arg) {
// remember to check method input first
// define the operations we want to run in transaction
const asyncRawMongoOperations = async session => {
// it's critical to receive the session parameter here
// and pass it to every raw operation as shown below
const item = await collection1.rawCollection().findOne(arg, { session: session });
const response = await collection2.rawCollection().insertOne(item, { session: session });
// if Mongo or you throw an error here runTransactionAsync(..) will catch it
// and wrap it with a Meteor.Error(..) so it will arrive to the client safely
return 'whatever you want'; // will be the result in the client
};
let result = await runTransactionAsync(asyncRawMongoOperations);
return result;
}
});
I have an event triggering a Metor.call():
Meteor.call("runCode", myCode, function(err, response) {
Session.set('code', response);
console.log(response);
});
But my runCode function inside the server's Metheor.methods has inside it a callback too and I can't find a way to make it return something to response in the above code.
runCode: function(myCode) {
var command = 'pwd';
child = exec(command, function(error, stdout, stderr) {
console.log(stdout.toString());
console.log(stderr.toString());
// I Want to return stdout.toString()
// returning here causes undefined because runCode doesn't actually return
});
// I can't really return here because I don't have yet the valuer of stdout.toString();
}
I'd like a way to have the exec callback return something as runCode without setInterval which would work, but as a hacky way in my opinion.
You should use Future from fibers.
See docs here : https://npmjs.org/package/fibers
Essentially, what you want to do is wait until some asynchronous code is run, then return the result of it in a procedural fashion, this is exactly what Future does.
You will find out more here : https://www.eventedmind.com/feed/Ww3rQrHJo8FLgK7FF
Finally, you might want to use the Async utilities provided by this package : https://github.com/arunoda/meteor-npm, it will make your like easier.
// load future from fibers
var Future=Npm.require("fibers/future");
// load exec
var exec=Npm.require("child_process").exec;
Meteor.methods({
runCode:function(myCode){
// this method call won't return immediately, it will wait for the
// asynchronous code to finish, so we call unblock to allow this client
// to queue other method calls (see Meteor docs)
this.unblock();
var future=new Future();
var command=myCode;
exec(command,function(error,stdout,stderr){
if(error){
console.log(error);
throw new Meteor.Error(500,command+" failed");
}
future.return(stdout.toString());
});
return future.wait();
}
});
In Meteor, I'm writing a method that will have to check a certain path's subdirectories for new files.
I first would like to just list the subdirectories within Meteor after which I child_process.exec a simple bash script that lists files added since the last time it executed.
I'm having some issues getting the directory discovery to be async (Error: Can't wait without a fiber). I've written a synchronous version but having both fs.readdir and fs.stat in stead of their synchronous alternatives allows me to catch errors.
Here's the code:
function listDirs(dir, isDir){
var future1 = new Future();fs.readdir(dir, function(err, files){
if (err)
throw new Meteor.error(500, "Error listing files", err);
var dirs = _.map(files, function(file){
var future2 = new Future();
var resolve2 = future2.resolver();
fs.stat(dir+file, function(err, stats){
if (err)
throw new Meteor.error(500, "Error statting files", err);
if (stats.isDirectory() == isDir && file.charAt(0) !== '.')
resolve2(err, file);
});
return future2;
});
Future.wait(dirs);
//var result = _.invoke(dirs, 'get');
future1['return'](_.compact(dirs));
});
return future1.wait();
}
The error Error: Can't wait without a fiber has to do with future2.
When I comment out Future.wait(dirs) the server doesn't crash anymore, but that's about as far as I got in trying to solve this. :/
Another _.map function I use in another part of the method works fine with futures. (see also https://gist.github.com/possibilities/3443021 where I found my inspiration)
Wrap your callback into Meteor.bindEnvironment, example:
fs.readdir(dir, Meteor.bindEnvironment(function (err, res) {
if (err) future.throw(err);
future.return(res);
}, function (err) { console.log("couldn't wrap the callback"); });
Meteor.bindEnvironment does a lot of things and one of them is to make sure callback is running in a Fiber.
Another thing that could be helpful is var funcSync = Meteor._wrapAsync(func) which utilizes futures and allows use to call a function in synchronous style (but it is still async).
Watch these videos on evented mind if you want to know more: https://www.eventedmind.com/posts/meteor-dynamic-scoping-with-environment-variables https://www.eventedmind.com/posts/meteor-what-is-meteor-bindenvironment
is there a way to simulate lag with Meteor? Perhaps something that would delay all calls by say, 300ms?
You can do it in publish using:
Meteor._sleepForMs(5000); // sleeps for 5 seconds
I guess I'm a bit late for the party, but here's a better solution:
There are basically two parts to this question. One is, how to delay Meteor WebSocket (SockJS) writes and one is how to delay HTTP traffic (connect). You'll need to add both of the following snippets to your server-side code in order to delay all traffic sent from the Meteor server.
WebSocket
The hard part was overwriting the WebSocket write to delay it with a setTimeout:
(function () {
// Set the delay you want
var timeout = 3000
// stream_server abstracts sockJS the DDP-Server talks to it.
var streamServer = Meteor.server.stream_server
// The connect event listener
var standardConnect = streamServer.server._events.connection
// Overwrite the default event-handler
streamServer.server._events.connection = function (socket) {
// Overwrite the writes to the socket
var write = socket.write
socket.write = function () {
var self = this
var args = arguments
// Add a delay
setTimeout(function () {
// Call the normal write methods with the arguments passed to this call
write.apply(self, args)
}, timeout)
}
// Call the normal handler after overwritting the socket.write function
standardConnect.apply(this, arguments)
}
})()
HTTP
With connect it's pretty straight forward:
// Add a simple connect handler, wich calls the next handler after a delay
WebApp.rawConnectHandlers.use(function (req, res, next) {
return setTimeout(next, timeout)
})
Not sure about all calls, but you can use Futures to add a lag on the server, that way you can see latency compensation in action.
In a meteor method for example, you can
Meteor.methods({
post: function(post) {
post.title = post.title + (this.isSimulation ? '(client)' : '(server)');
// wait for 5 seconds
if (! this.isSimulation) {
var Future = Npm.require('fibers/future');
var future = new Future();
Meteor.setTimeout(function() {
future.ret();
}, 5 * 1000); // 5 seconds
future.wait();
}
var postId = Posts.insert(post);
return postId;
}
});
This will show the post being inserted with (client) appended to the end, and then 5 seconds later will get the update from the server and post's title will end with (server)
If you want simulate the lag in the subscriptions you can do the next:
Meteor.publish('collection', function(params) {
Meteor._sleepForMs(2000); // Sleep for 2 seconds
return CollectionX.find(params.query,params.projection);
});
I've been trying to access the this.userId variable from within a Meteor.methods call, but it doesn't seem to work when I try to call the method via Meteor.setTimeout or Meteor.setInterval.
This is what I've got:
if (Meteor.is_server) {
Meteor.methods({
getAccessToken : function() {
try {
console.log(this.userId);
return Meteor.users.findOne({_id: this.userId}).services.facebook.accessToken;
} catch(e) {
return null;
}
}
});
var fetch_feed = function() {
console.log(Meteor.call("getAccessToken"));
[...] // A bunch of other code
};
Meteor.startup(function() {
Meteor.setInterval(fetch_feed, 60000); // fetch a facebook group feed every minute
Meteor.setTimeout(fetch_feed, 3000); // initially fetch the feed after 3 seconds
});
}
Watching the terminal log, the this.userId always returns a null. But if I try calling the method from the client side, or through the console, it returns the correct ID.
How come this doesn't work from within a Meteor.setInterval? Is it a bug or am I doing something wrong?
Meteor userId's are associated with client connections. The server may interact with many clients and this.userId inside a method will tell you which client has asked for the method to be run.
If the server uses Meteor.call() to run a method then it will not have a userId since it is not running for any client.
The methods allow clients to call for functions to be run on the server. For things the server will trigger itself a javascript function will do.
There is a solution I used - sometimes you do not want to make the method a function but really want it to remain a method. In that case, a hack to make this work:
var uniqueVar_D8kMWHtMMZJRCraiJ = Meteor.userId();
Meteor.setTimeout(function() {
// hack to make Meteor.userId() work on next async
// call to current method
if(! Meteor._userId) Meteor._userId = Meteor.userId;
Meteor.userId = function() {
return Meteor._userId() || uniqueVar_D8kMWHtMMZJRCraiJ
};
Meteor.apply(methodName, args);
}
, 100);
Some brief explanation: we save Meteor.userId in Meteor._userId and overwrite Meteor.userId with a function that returns Meteor._userId() if it is true and otherwise the historic value of Meteor.userId() before any of this happened. That historic value is saved in an impossible to occur twice var name so that no context conflicts can happen.