A number of my meteor methods have mysteriously slowed down recently. Whereas they used to be quite snappy, many are taking 10 or so seconds.
Things not causing the slowdown:
Additional functionality, the slowed sections of the codebase haven't been changed significantly
Machine load (cpu load hovering around 30%)
Additional DB load (no new queries added)
transfer time of the return data (returns undefined)
blocking method (I've tried with this.unblock() within the method)
I did debugging by using console.time() / console.timeEnd() on both the server and client side. The server side code takes about .3 seconds to run, but the client doesnt get the callback until about 11 seconds after the meteor.call() ...
This is the server method:
function cancelSomething(somethingId, reason) {
console.time('cancelSomething method');
check(somethingId, String);
check(reason, String);
if (!AuthChecks()))
throw new Meteor.Error(401, 'Not Authorized');
var something = MySomethings.findOne({'_id': somethingId});
if (!something)
throw new Meteor.Error(404, 'Something not found');
var returnVal = SomethingService.cancel(something, reason);
console.timeEnd('cancelSomething method'); // <--- prints "cancelSomething 350ms" or there abouts
return returnVal;
}
clientSide:
console.time('meteorCall');
Meteor.call('cancelSomething', this._id, reason, function(err) {
if (err) {
console.log('an error occurred', err);
}
console.timeEnd('meteorCall'); // <--- prints "meteorCall 11500" or so
});
EDIT:
I have noticed there is some correlation with the quantity of docs within the "somethings" in the db. w/ 500 documents it takes about 1 second to receive the return on the client, with 5000 it takes about 8.5 seconds ...
Interesting. I think this one is hard to answer without knowing more about your app, but I have some suggestions that could help steer you in the right direction.
blocking
If we can assume that the timers are working properly, what could be happening is that the server is unable to begin execution of the method because another method call by the same client is already in progress. Have a look at unblock in the docs. The answer may be as simple as putting a this.unblock at the top of one of your methods.
syncing
If SomethingService.cancel does something which results in a lot of DDP traffic, that could tie up the client for a while and make it unable to execute your callback. For example if it:
modified or created a lot of documents which then had to be synced to the client
indirectly caused an expensive subscription to be rerun (my money is on this one)
Updated Answer
It seems the problem had to to with a call to observe, which makes a lot of sense given your high CPU load. Frankly, it's a little silly that I didn't suggest this because I have already answered a similar question here. If you want to try to keep the observe, here are some additional suggestions:
Use oplog tailing.
Try to narrow the scope of the observe as much as possible using selectors which are currently supported by the oplog observe driver. The supported selectors should improve at or before meteor 1.0.
Limit the fields to only those you need.
Check that oplog tailing is working by adding the facts package.
Related
Is it possible to stop (kill) asynchronous Call?
In my app I have at client side sth like:
Meteor.call('doCalculation', function(err, result) {
//do sth with result
});
'doCalculation' may take long time (this is ok) I dont want user to start new call when he/she has already one running call, I want to allow user to stop current call and submit new one. How correctly do this?
The only idea I have is to communicate between client and server using mongo. In some place in 'doCalculation' function I can observe some mongo document/collection and based on this do sth in the function (e.g. call exception). Do you have any better ideas?
You can use a semaphore for this purpose. When the semaphore is 1, requests are allowed to be sent. When the semaphore is 0, requests are not allowed to be sent. The semaphore should be 1 by default and just before you send the request, you need to set it to 0. When a response is successful, you set the semaphore back to 1.
As about the timeout: You could use a time out using setTimeout after sending the request, like this:
if (semaphore) {
var isTimedOut = false;
var isSuccess = false;
semaphore = 0; //No need to use var keyword, as this should be declared outside of this scope
Meteor.call('doCalculation', function(err, result) {
isSuccess = true;
//do sth with result
});
setTimeout(function() {
if (!isSuccess) {
isTimeout = true;
//do something else, to handle the time out state
}
}, 10000);
}
This is tricky, because you cannot generally set timeouts from the client's point of view. You don't need to, for a bunch of architectural reasons. The most important thing is that if you lose network connectivity or the server crashes (two cases timeouts are designed to manage), the client is aware immediately because it is disconnected. You can use Meteor.status().connected if this happens often.
It sounds like you're running a long calculation on the server. My suggestion is to return a calculationId immediately, and then update a collection with progress, e.g., CalculationProgresses.update(calculationId, {$set: {progress: currentProgress}}) as you calculate. Your UI can then update the progress reactively, in the most convenient way possible.
Note, that when you do run long calculations on the server, you need to occasionally "yield," giving the chance for other work to happen. Node, on which Meteor is based, is tricky for long calculations if you don't master this notion of yielding. In Meteor, you can yield easily by updating a collection (e.g., your progress collection). This will solve lots of problems you're probably experiencing as you write your application.
i think you need a server-side solution for this. if you go with a client-side solution, you don't handle 2 cases:
the user reloads their browser
the user uses 2 browsers
i would create these methods:
isCalculationActive() -- this checks if the user already has a calculation active. on the server, you can either keep that fact in memory or write it to the db. on the client, if this returns false, then you can proceed to call doCalculation(). if true, you can give the user a popup or alert or something to ask if they want to cancel and proceed.
doCalculation() -- this cancels any outstanding calculation by that user and starts a new one.
with these implemented, the user can reload their browser w/o affecting either the running calculation or correct behavior. and if they try a 2nd browser, everything should still work as expected.
if you want to give the user the option to simply stop the job and not start a new one, then you can simply create:
cancelCalculation() -- this cancels any outstanding calculation by that user.
I'm attempting to use observechanges with iron router but they don't seem to be compatible at all.
Router.route('/gaming', {
waitOn: function() {
return Meteor.subscribe('chat', function() {
window.chatmessagesCache = new ReactiveVar;
chatmessagesCache.set([]);
return chat.find().observeChanges({
added: function(id, doc) {
var tmpArr;
tmpArr = chatmessagesCache.get();
tmpArr.push(doc);
return chatmessagesCache.set(tmpArr);
}
});
});
}
If I leave the route and come back to it, observechanges begins being handled as many times as I've navigated away and returned, for each new record. What's the deal?
If I use subs manager it works as expected, but I don't understand why Meteor.subscribe inside waitOn is so cache/subscription unaware when it ALREADY gets called multiple times per load. Why!? I can't decipher what's causing this behavior at all.
Also, what I'm trying to accomplish is simple. I want to let chat messages that the user's client has received remain on the page even if the chat cursor is no longer publishing them (I'm publishing the last 10 chat messages)
Iron router has reactivity built in, which means when something inside your route function is invalidated, it will repeat the function as well as anything reactive with a Router.current(). These unexpected invalidation runs are a primary reason why folks made the exodus to flow router.
To solve this, you'll want to abstract your code away from the router. You can leave the sub, but I'd suggest you remove the sub's callback from the waitOn and move it into an onRendered callback. If you don't want the history loaded in chunks, you can first do a var collectionCount = chat.find({},{reactive:false}).count() on how many docs are in the collection & then in the added callback you can do something like if (++currentCount === collectionCount) /* add stuff */ to add al the records to the history when it reaches the last record.
On a bigger picture level, consider eliminating the observeChanges & just do an #each over the chat collection in spacebars to show your messages. Fewer cycles, cleaner code.
Iron router just has no management of observations you created yet it manages subscriptions itself, hence the multiple additions.
I figured this out by using a window level variable to check if I'm observing. Even in cases when the subscription is unhooked by iron, if I go back and never re-add the handler, the original observation hook still runs (!). ALSO, if you navigate away and drop the subscription, the handler is no longer called--which is the behavior I want in this scenario (This is all very insane behavior but at least it's now predictable to me )
This is caused by the fact that subscriptions != collections and the API for observations doesn't seem to expose any metadata, unfortunately, so I don't know how the iron router maintainers would account for this. Not to mention you return iron router a subscription, not a collection.
#Matt K if you were correct, this would always be an infinite loop (which admittedly I had a bunch of while trying to solve this) but the posted code is adding too many handlers, not looping indefinitely. Thanks for the post though.
This is what I settled on
Router.route('/gaming',
waitOn: ->
Meteor.subscribe('chat', ->
window.chatmessagesCache = new ReactiveVar(chat.find().fetch().reverse())
if !window.chatListening
window.chatListening = true
after = chat.find().count()
chat.find().observe(
added: _.after(after + 1,(doc) ->
tmpArr = chatmessagesCache.get()
tmpArr.push(doc)
chatmessagesCache.set(tmpArr))
changed : (id, doc) ->
))
I really just wanted to test out a pattern of locally "disconnected" documents. I still may use subs manager because it keeps subscriptions and their handlers alive without rerunning them constantly (which was rerunning the sub handler, which was adding multiple observations)
I was looking at some meteor code, and I saw this :
Template.notifications.helpers({
notifications: function() {
return Notifications.find({userId: Meteor.userId(), read: false});
},
notificationCount: function(){
return Notifications.find({userId: Meteor.userId(), read: false}).count();
}
});
So I was wondering, Is this optimized ?
I mean, will the mongo database execute two queries ? The server part ? the client part ? (mini mongo then)
Is it possible to use the previous result in the second function ? I tried with
notificationCount = function(){
this.notifications.length;
....
But it doesn't work, but maybe meteor remembers the previous result and uses it ?
I will definitely, in my template, return a something.find() to have a cursor and afterwards return other variable with, for example : count, or filter it with fields or other stuff so I'm intereted by this question.
Any expert to explain me ? Thanks a lot meteor community :) !
You're not performing 2 queries on the server, which ends up being the most critical place. When you subscribe to data, the data goes to a local database in the browser called MiniMongo. You can run as many queries as you want on the client, the data set (usually and let's hope so) is small and there isn't a noticeable performance penalty.
If you've some performance issues you can save the results of Notifications.find({userId: Meteor.userId(), read: false}) to Session or to another Reactive Dictionary. This will slightly improve the performance because you save the query time of Minimongo: parsing, searching, etc.
On the server, you should be as careful as possible. A bottleneck in the server might mean that your entire application isn't going to be as fast as wanted.
Read more about mini in memory databases: https://www.meteor.com/mini-databases
yes. on the client side, not on the server side.
you can use {{notifications.count}} on template.
I've actually been toying with Meteor for a little bit now, but I realized that I still lack some (or a lot!) comprehension on the topic.
For example, here is a tutorial that uses node.js/express/socket.io to make a simple real-time chat: http://net.tutsplus.com/tutorials/javascript-ajax/real-time-chat-with-nodejs-socket-io-and-expressjs/
In that above example, through socket.io, the webserver receives some data and passes it onto all of the connected clients -- all without any database accesses.
With Meteor, in all the examples that I've seen, clients are updated by writing to the mongodb, which then updates all the clients. But what if I don't need to write data to the database? It seems like an expensive step to pass data to all clients.
I am sure I am missing something here. What would be the Meteor way of updating all the clients (say, like with a simple chat app), but without needing the expense of writing to a database first?
Thank you!
At the moment there isn't an official way to send data to clients without writing it to a collection. Its a little tricker in meteor because the step to send data to multiple clients when there isn't a place to write to comes from when multiple meteor's are used together. I.e items sent from one meteor won't come to clients subscribed on the other.
There is a temporary solution using Meteor Streams (http://meteorhacks.com/introducing-meteor-streams.html) that can let you do what you want without writing to the database in the meanwhile.
There is also a pretty extensive discussion about this on meteor-talk (https://groups.google.com/forum/#!topic/meteor-talk/Ze9U9lEozzE) if you want to understand some of the technical details. This will actually become possible when the linker branch is merged into master, for a single server
Here's a bit of way to have a 'virtual collection, its not perfect but it can do until Meteor has a more polished way of having it done.
Meteor.publish("virtual_collection", function() {
this.added("virtual_coll", "some_id_of_doc", {key: "value"});
//When done
this.ready()
});
Then subscribe to this on the client:
var Virt_Collection = new Meteor.Collection("virtual_coll");
Meteor.subscribe("virtual_collection");
Then you could run this when the subscription is complete:
Virt_Collection.findOne();
=> { _id: "some_id_of_doc", key: "value"}
This is a bit messy but you could also hook into it to update or remove collections. At least this way though you won't be using any plugins or packages.
See : https://www.eventedmind.com/posts/meteor-how-to-publish-to-a-client-only-collection for more details and a video example.
The publish function on the server sends data to clients. It has some convenient shortcuts to publish query results from the database but you do not have to use these. The publish function has this.added(), this.removed(), and this.changed() that allow you to publish anything you choose. The client then subscribes and receives the published data.
For example:
if ( Meteor.isClient ){
var someMessages = new Meteor.Collection( "messages" ); //messages is name of collection on client side
Meteor.subscribe ( "messagesSub" ); //messagesSub tells the server which publish function to request data from
Deps.autorun( function(){
var message = someMessages.findOne({});
if ( message ) console.log( message.m ); // prints This is not from a database
});
}
if (Meteor.isServer ) {
Meteor.publish( "messagesSub", function(){
var self = this;
self.added ( "messages", "madeUpId1", { m: "This is not from a database"} ); //messages is the collection that will be published to
self.ready();
});
}
There is an example in meteor docs explained here and another example here. I also have an example that shares data between clients without ever using a database just to teach myself how the publish and subscribe works. Nothing used but basic meteor.
It's possible to use Meteor's livedata package (their DDP implementation) without the need of a database on the server. This was demoed by Avital Oliver and below I'll point out the pertinent part.
The magic happens here:
if (Meteor.isServer) {
TransientNotes = new Meteor.Collection("transientNotes", {connection: null});
Meteor.publish("transientNotes", function () {
return TransientNotes.find();
});
}
if (Meteor.isClient) {
TransientNotes = new Meteor.Collection("transientNotes");
Meteor.subscribe("transientNotes");
}
Setting connection: null specifies no connection (see Meteor docs).
Akshat suggested using streams. I'm unable to reply to his comment due to lack of reputation, so I will put this here. The package he links to is no longer actively maintained (see author's tweet). I recommend using the yuukan:streamy (look it up on Atmosphere) package instead or use the underlying SockJS lib used in Meteor itself—you can learn how to do this by going through the Meteor code and look how Meteor.server.Stream_server, and Meteor.connection._stream are used, which is what the Streamy package does.
I tested an implementation of the Streamy chat example and found performance to be negligibly different but this was only on rudimentary tests. Using the first approach you get the benefits of the minimongo implementation (e.g. finding) and Meteor reactivity. Reactivity is possible with Streamy, though is does through things like using ReactiveVar's.
There is a way! At least theoretically The protocol used by Meteor to sync between client and server is called DDP. The spec is here
And although there some examples here and here of people implementing their own DDP clients, I'm afraid haven't seen examples of implementations of DDP servers. But I think the protocol is straightforward and would guess it wouldn't be so hard to implement.
I have an EC2 instance running a small node script connecting to Firebase. Strangely enough, it happens quite often on a small instance that the set operation gets executed immeditely but the callback function only gets called much later (between 30s to 2 minutes). Do you see any reason why it would happen that way?
console.log('creating');
// Create workspace
rootRef.child('spaces').child(chid).set(req.space, function(error) {
var end = new Date().getTime();
var time = end - start;
console.log('- created', error, time);
});
The bug is directly related to node 0.11 (set() callback is only called the first name in my scenario). Just revert to 0.10.x and it's all fixed!
I've been facing the same issue. the "Set" callback is not being invoked at all. I noticed, however, that if I run a snippet code similar to yours in a standalone file, the callback is invoked very quickly.
It turned out that if you're installing listeners on the same Node you're calling the "set" function on (i.e., on('child_added'), on('child_removed') ... etc) and that Node has a huge number of records, it'll simply take ages.
I removed the listeners ( to test) and the "set" started to invoke the callback very quickly.
I hope this helps!