Meteor - Publications causing long loading time - meteor

I am trying to make a fairly big meteor app, and I noticed that it has gotten slower over the past few days and I read on the meteor forum that publications can cause slow loading times. After I refresh the page when making a change in the application itself (code change), it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
if(Meteor.isServer){
Meteor.publish('notes', function () {
return Notes.find()
});
Meteor.publish('users', function () {
return Meteor.users.find()
});
Meteor.publish("user", function(){
return Meteor.user()
})
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
}
document example:
let noteInfo = { title, subject, description, imageURL, userId, userEmail, createdAt }
let title = this.refs.title.value;
let subject = this.refs.subject.value;
let description = this.refs.description.value;
let allUrls = [this.refs.imageURL.value].concat(this.state.urls);
let imageURL = allUrls.filter(function(entry) { return entry.trim() != ''; });
let userId = Meteor.userId();
let userEmail = Meteor.user().emails[0].address;
let createdAt = Date.parse(new Date());

This is a very broad performance tuning question. You haven't told us how many documents are in your collections or how large these documents are. Some possible issues:
You are over-publishing either by publishing too many documents or because your documents are too big. For example when you do:
Meteor.publish('notes', function () {
return Notes.find()
});
If there are 100,000 Notes documents at 100 bytes each then that's 10 MB that needs to go over the network to the client. This is also the case if there are 1,000 notes documents at 10KB each.
Solution: Limit the number of documents with limit and/or reduce the number of fields transmitted with fields:
Meteor.publish('notes', function () {
return Notes.find({},{ limit: 100, fields: { key1: 1, key2: 1 }});
});
Your collection is missing one or more indexes. When you do:
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
if there are 1M notes documents but there is no index on the createdAt key then this will be terribly slow.
Solution: Adding an index on the createdAt key will make such a publication much faster.
You have an invalid publication.
Meteor.publish("user", function(){
return Meteor.user();
})
Is invalid because Meteor.user() is not a cursor and a publication must return either a cursor or an array of cursors. It is also redundant because Meteor.user() is automatically available on the client, albeit it doesn't include all the keys.
Solution: Remove this unnecessary publication altogether. If you want to publish some of the keys that are not available on the client for the current user you can do so as follows:
Meteor.publish("user", function(){
return Meteor.users.find(this.userId,{fields: {services: 1, emails: 1, profile: 1}})
});

After I refresh the page when making a change, it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
What does this mean? Do you mean when you change the code of your website, it takes 1-2 minutes for your changes to appear in a newly loaded page? That's to be expected: it takes time for Meteor to rebuild your application. Watch the terminal for the progress.
Using lots of packages and external code will slow down building. The speed of compilation is unrelated to your end user's experience.

A couple things I do with my meteor projects is that I send specific fields "profile.name" etc. I've also used collection.createIndex({}) which creates an index and inside it and you could put your sort in which is created on server startup and works well with your collection.find.sort. I think limiting your notes would be quite essential as you have a limit on one and not the other.

It happened to me too. At that time I checked with kadira (formerly still opensource but now it is not). Source code at kadira on github, but service is gone kadira.io
The problem is in pubication and subscribtion. When client request subscribtions, he opens up some kind of "connection" - i say that, and it applies to others who call the same publication. imagine how many connections are formed.
where the same data accessed by many people must server side must request data to mongodb, that same to all of number of such connections. This is what makes it slow.
Finally I added redis.io.
meteor npm install --save redis
meteor npm install --save hiredis
On server side :
import { Meteor } from "meteor/meteor";
var redis = require("redis");
var clientRedis = redis.createClient({
host: "YOURREDIS_IP",
port: "YOURREDIS_PORT"
});
clientRedis.setSync = Meteor.wrapAsync(clientRedis.set);
clientRedis.getSync = Meteor.wrapAsync(clientRedis.get);
setRedis_Object = function (keyREDIS, timeRefresh, valueREDIS) {
return clientRedis.setSync(keyREDIS + moment(new Date()).format(timeRefresh), JSON.stringify(valueREDIS));
};
getRedis_Object = function (keyREDIS, timeRefresh) {
return JSON.parse(clientRedis.getSync(keyREDIS + moment(new Date()).format(timeRefresh)));
};
Meteor.methods({
getRedis_YOURCOLLECTIONS: function(timeRefresh) {
var data = getRedis_Object(getRedis_YOURCOLLECTIONS, timeRefresh);
if (data != null) {
return data;
} else {
var data = YOURCOLLECTIONS.find().fetch();
setRedis_Object(key, timeRefresh, data);
return data
}
},
});
On Client Side
Meteor.call("getRedis_YOURCOLLECTIONS", "YYYYMMDD", function (error, result) {
if (error) {
console.log(error);
} else {
Session.set("getRedis_YOURCOLLECTIONS", result);
}
});
After i am add this REDIS consumtion Memory and CPU on server low, and app more Fast. I hope its work for you. Thanks

Related

Meteor Publication and Subscription fetch all records after use skip and limit

We have use publication and subscription with skip and limit in meteor and it fetch all records at client side.
Meteor.publish('jobOrders', function(skip, limit) {
return (jobOrders.find({}, {
skip: skip,
limit: limit
}));
});
Template.templateName.onCreated(function() {
this.subscribe("jobOrders", 0, 2);
});
Template.templateName.helpers({
methodName: function() {
var totalNoOfRecords = jobOrders.find({}).count();
console.og("records are", totalNoOfRecords);
return totalNoOfRecords;
}
});
The reason could be because you have other pub-subs in other templates which are fetching the data from jobOrders collection.
Please comment your subscription in your current template and see whether you are still able to fetch the records from this collection. If you are seeing all the records even after commenting the subscription, it should be because of some other subscription.
To overcome this problem, you have to either do one of the following:
Change the other subscription which is fetching all the records such that the data is available only with in that template by using
Template.instance().subscribe()
instead of using Meteor.subscribe()which will make the data available throughout client side across all templates.
Filter you records on the client side as well.
Template.templateName.helpers({
methodName: function() {
var totalNoOfRecords = jobOrders.find({skip:0, limit: 2}).count(); //Filter on the client side
console.og("records are", totalNoOfRecords);
return totalNoOfRecords;
}
});
Note: Another possibility could be that you did not remove the insecure package which will automatically publish all the records in all the collection.

Meteor-angular autocomplete from huge data

I have angular-meteor app that needs Material md-autocomplete from a collection with 53,296 documents with angularUtils.directives.dirPagination but this amount of data make my browser hang.
I'm publishing the collection with:
Meteor.publish('city', function (options, searchString) {
var where = {
'city_name': {
'$regex': '.*' + (searchString || '') + '.*' ,
'$options': 'i'
}
};
return City.find(where, options);
});
I subscribe with:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
and have pagination on controller :
$scope.currentPage = 1;
$scope.pageSize = 100;
$scope.sort = {city_name_sort : 1};
$scope.orderProperty = '1';
$scope.helpers({
city: function(){
return City.find({});
}
});
but it takes a long time to load and its make chrome stop working.
You already have most of the server-side searching done because your search is running inside a subscription. You should make sure that the city_name field is indexed in mongo! You should only return that field to minimize data transfer. You can also simplify your regex.
Meteor.publish('city', function (searchString) {
const re = new RegExp(searchString,'i');
const where = { city_name: { $regex: re }};
return City.find(where, {sort: {city_name: 1}, fields: {city_name: 1}});
});
What I've found helps with server-side auto-complete is:
Don't start searching until the user has typed 3 or 4 characters. This drastically narrows down the search results.
Throttle the search to only run every 500ms so that you're not sending every character to the server because then it has to keep re-executing the search. If the person is typing fast the search might only run every 2 or 3 characters.
Run the same .find() on the client that you're running on the server (instead of just querying for {}). That's just good practice since the client-side collection is the union of all subscriptions on that collection, there might be documents there that you don't want to list.
Lastly I don't know why you're subscribing twice here:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
only one of those Meteor.subscribe('city') calls is necessary.

How to prevent a client race condition between Meteor.userId() and subscription updates that depend on userId?

I am seeing a repeatable issue where a user authenticates ("logs in") with a Meteor server, and then a client subscription that depends on userId is updated (and dependent UI templates reactively update) before Meteor.userId() registers the successful login.
For example, in this code snippet, the assert will throw:
var coll = new Meteor.Collection("test");
if (Meteor.isServer) {
Meteor.publish('mineOrPublic', function () {
// Publish public records and those owned by subscribing user
return coll.find({owner: { $in: [ this.userId, null ]}});
});
}
if (Meteor.isClient) {
var sub = Meteor.subscribe('mineOrPublic');
var cursor = coll.find({});
cursor.observe({
added: function (doc) {
if (doc.owner) {
// This should always be true?!
assert(doc.owner === Meteor.userId());
}
}
});
}
Analogous to the added function above, if I write a template helper that checks Meteor.userId(), it will see a value of null, even when it is invoked with a data context of a document with an owner.
There is apparently a race condition between Meteor collection Pub/Sub and the Account userId update mechanisms. It seems to me that Meteor.userId() should always be updated before any subscriptions update based on a change in this.userId in a server publish function, but for some reason the opposite usually seems to be true (that is, the assert in the code above will usually throw).
The reason I care is because I have packages that depend on obtaining a valid Meteor Authentication token (using Accounts._storedLoginToken()) on the client for use in securing HTTP requests for files stored on the Meteor server. And the authentication token isn't correct until Meteor.userId() is. So the flow of events usually goes something like this:
User logs in
Publish function on server reruns based on the change in this.userId.
Client begins receiving new documents corresponding to the change in userId.
UI Template reactively updates to add DOM elements driven by new documents
Some of the DOM elements are <img> tags with src= values that depend on the data context.
HTTP requests are triggered and ultimately fail with 403 (forbidden) errors because the required authentication cookie hasn't been set yet.
Meteor.userId() finally updates on the client, and code reactively runs to set the authentication cookie
Helpers in the template that depend on a session variable set in the cookie update code are rerun, but the DOM doesn't change, because the URLs in the <img> tags don't change.
Because the DOM doesn't change, the tags don't retry their failed attempts to load the images.
Everything settles down, and the user has to manually reload the page to get their images to appear.
I've come up with two possible approaches to work around this issue:
In the template helper that generates the URL for the <img> tag, always append a dummy query string such as: "?time=" + new Date().getTime(). This causes the DOM to change every time the helper is called and fixes the problem, but it screws-up browser caching and if not coordinated will cause some assets to unnecessarily load multiple times, etc.
In every template helper that touches document data add a test of:
if (this.owner && this.owner !== Meteor.userId()) {
// Perhaps Meteor.loggingIn() could be used above?
// Invalid state, output placeholder
} else {
// Valid state, output proper value for template
}
I really hope someone knows of a less kludgy way to work around this. Alternatively, if consensus arises that this is a bug and Meteor's behavior is incorrect in this respect. I will happily file an issue on Github. I mostly really enjoy working with Meteor, but this is the kind of gritty annoyance that grinds in the gears of "it just works".
Thanks for any and all insights.
After trying lots of things, this variation on the example code in the OP seems to consistently solve the race condition, and I find this an acceptable resolution, unlike my initial attempted workarounds.
I still feel that this kind of logic should be unnecessary and welcome other approaches or opinions on whether Meteor's behavior in the OP sample code is correct or erroneous. If consensus emerges in the comments that Meteor's behavior is wrong, I will create an issue on Github for this.
Thanks for any additional feedback or alternative solutions.
var coll = new Meteor.Collection("test");
if (Meteor.isServer) {
Meteor.publish('mineOrPublic', function (clientUserId) {
if (this.userId === clientUserId) {
// Publish public records and those owned by subscribing user
return coll.find({owner: { $in: [ this.userId, null ]}});
} else {
// Don't return user owned docs unless client sub matches
return coll.find({owner: null});
}
});
}
if (Meteor.isClient) {
Deps.autorun(function () {
// Resubscribe anytime userId changes
var sub = Meteor.subscribe('mineOrPublic', Meteor.userId());
});
var cursor = coll.find({});
cursor.observe({
added: function (doc) {
if (doc.owner) {
// This should always be true?!
assert(doc.owner === Meteor.userId());
}
}
});
}
This code works by giving the server publish function the information it needs to recognize when it is running ahead of the client's own login state, thereby breaking the race condition.
I think this is something that Meteor should do automatically: clients should not see documents based on changes to this.userId in a publish function until after the client Meteor.userId() has been updated.
Do others agree?
I tried with this code that works on server too. In association with FileCollection package.
if (Meteor.isServer) {
CurrentUserId = null;
Meteor.publish(null, function() {
CurrentUserId = this.userId;
});
}
....
OrgFiles.allow({
read: function (userId, file) {
if (CurrentUserId !== file.metadata.owner) {
return false;
} else {
return true;
}
}
...

Publish documents in a collection to a meteor client depending on the existence of a specific document in another collection (publish-with-relations)

I have two collections
Offers (relevant fields: _id)
ShareRelations (relevant fields: receiverId and offerId)
and I'd like to publish only Offers to the logged in user which have been shared to him.
Actually, I'm doing this by using a helper array (visibleOffers) which I fill by looping for each ShareRelations and use this array later on the Offers.find as $in selector.
I wonder if this might be the meteor way to do this, or if I could do with less and/or prettier code?
My actual code to publish the Offers is the following:
Meteor.publish('offersShared', function () {
// check if the user is logged in
if (this.userId) {
// initialize helper array
var visibleOffers = [];
// initialize all shareRelations which the actual user is the receiver
var shareRelations = ShareRelations.find({receiverId: this.userId});
// check if such relations exist
if (shareRelations.count()) {
// loop trough all shareRelations and push the offerId to the array if the value isn't in the array actually
shareRelations.forEach(function (shareRelation) {
if (visibleOffers.indexOf(shareRelation.offerId) === -1) {
visibleOffers.push(shareRelation.offerId);
}
});
}
// return offers which contain the _id in the array visibleOffers
return Offers.find({_id: { $in: visibleOffers } });
} else {
// return no offers if the user is not logged in
return Offers.find(null);
}
});
Furthermore, the actual solution has the downside that if a new share relations is being created, the Offers collection on the client doesn't get updated with the newly visible offer instantly (read: page reload required. But I'm not sure if this is the case because of this publish method or because of some other code an this question is not primary because of this issue).
What you are looking for is a reactive join. You can accomplish this by directly using an observe in the publish function, or by using a library to do it for you. Meteor core is expected to have a join library at some point, but until then I'd recommend using publish-with-relations. Have a look at the docs, but I think the publish function you want looks something like this:
Meteor.publish('offersShared', function() {
return Meteor.publishWithRelations({
handle: this,
collection: ShareRelations,
filter: {receiverId: this.userId},
mappings: [{collection: Offers, key: 'offerId'}]
});
});
This should reactively publish all of the ShareRelations for the user, and all associated Offers. Hopefully publishing both won't be a problem.
PWR is a pretty legit package - several of us use it in production, and Tom Coleman contributes to it. The only thing I'll caution you about is that as of this writing, the current version in atmosphere (v0.1.5) has a bug which will result in a fairly serious memory leak. Until it gets bumped, see my blog post about how to run an updated local copy.
update 2/5/14:
The discover meteor blog has an excellent post on reactive joins which I highly recommend reading.
The way to do this is along the lines of this Question using observeChanges(). Still trying to figure out how to get it all working for my example, see Meteor, One to Many Relationship & add field only to client side collection in Publish?
You can use the reactive-publish package (I am one of authors):
Meteor.publish('offersShared', function () {
// check if the user is logged in
if (this.userId) {
this.autorun(function (computation) {
// initialize helper array
var visibleOffers = [];
// initialize all shareRelations which the actual user is the receiver
var shareRelations = ShareRelations.find({receiverId: this.userId}, {fields: {offerId: 1}});
// loop trough all shareRelations and push the offerId to the array if the value isn't in the array actually
shareRelations.forEach(function (shareRelation) {
if (visibleOffers.indexOf(shareRelation.offerId) === -1) {
visibleOffers.push(shareRelation.offerId);
}
});
// return offers which contain the _id in the array visibleOffers
return Offers.find({_id: { $in: visibleOffers } });
});
} else {
// return no offers if the user is not logged in
return Offers.find(null);
}
});
You can simply wrap your existing non-reactive code into an autorun and it will start to work. Just be careful to be precise which fields you query on because if you query on all fields then autorun will be rerun on any field change of ShareRelations, not just offerId.

How to get a published collection's total count, regardless of a specified limit, on the client?

I'm using the meteor-paginated-subscription package in my app. On the server, my publication looks like this:
Meteor.publish("posts", function(limit) {
return Posts.find({}, {
limit: limit
});
});
And on the client:
this.subscriptionHandle = Meteor.subscribeWithPagination("posts", 10);
Template.post_list.events = {
'click #load_more': function(event, template) {
template.subscriptionHandle.loadNextPage();
}
};
This works well, but I'd like to hide the #load_more button if all the data is loaded on the client, using a helper like this:
Template.post_list.allPostsLoaded = function () {
allPostsLoaded = Posts.find().count() <= this.subscriptionHandle.loaded();
Session.set('allPostsLoaded', allPostsLoaded);
return allPostsLoaded;
};
The problem is that Posts.find().count() is returning the number of documents loaded on the client, not the number available on the server.
I've looked through the Telescope project, which also uses the meteor-paginated-subscription package, and I see code that does what I want to do:
allPostsLoaded: function(){
allPostsLoaded = this.fetch().length < this.loaded();
Session.set('allPostsLoaded', allPostsLoaded);
return allPostsLoaded;
}
But I'm not sure if it's actually working. Porting their code into mine does not work.
Finally, it does look like Mongo supports what I want to do. The docs say that, by default, cursor.count() ignores the effects of limit.
Seems like all the pieces are there, but I'm having trouble putting them together.
None of the answers do what you really want becase none provide solution that is reactive.
This package does exactly what you want and also reactive.
publish-counts
I think you can see the demo: counts-by-room in meteor doc
It can help you publish the counts of your posts at server and get it at client
You can simply write this:
// server: publish the current size of your post collection
Meteor.publish("counts-by-room", function () {
var self = this;
var count = 0;
var initializing = true;
var handle = Posts.find().observeChanges({
added: function (id) {
count++;
if (!initializing)
self.changed("counts", 'postCounts', {count: count});
},
removed: function (id) {
count--;
self.changed("counts", postCounts, {count: count});
}
});
initializing = false;
self.added("counts", 'postCounts', {count: count});
self.ready();
self.onStop(function () {
handle.stop();
});
});
// client: declare collection to hold count object
Counts = new Mongo.Collection("counts");
// client: subscribe to the count for posts
Tracker.autorun(function () {
Meteor.subscribe("postCounts");
});
// client: simply use findOne, you can get the count object
Counts.findOne()
The idea of sub.loaded() is to help you with exactly this problem.
Posts.count() isn't going to return the right thing because, as you've guessed, on the client, Meteor has no way of knowing the real number of posts that live on the server. But what the client knows is how many posts it's tried to load. That's what that .loaded() tells you, and is why the line this.fetch().length < this.loaded() will tell you if there are more posts on the server or not.
What I would do is write a Meteor server side method that retrieves the count like so:
Meteor.methods({
getPostsCount: function () {
return Posts.find().count();
}
});
Then call it on the client, in observe to make it reactive:
function updatePostCount() {
Meteor.call('getPostsCount', function (err, count) {
Session.set('postCount', count);
});
}
Posts.find().observe({
added: updatePostCount,
removed: updatePostCount
});
Although this question is old, I thought I would provide an answer that ended up working for me. I did not create the solution, I found the basis for it here (so credit where credit is due): Discover Meteor
Anyway, in my case I was trying to get "size" of the database from client side, so I can determine when to hide the "load more" -button. I was using template level subscriptions. Oh and for this solution to work, you need to add reactive-var -package. Here is my (in short):
/*on the server we define the method which returns
the number of posts in total in the database*/
if(Meteor.isServer){
Meteor.methods({
postsTotal: function() {
return PostsCollection.find().count();
}
});
}
/*In the client side we first create the reactive variable*/
if(Meteor.isClient){
Template.Posts.onCreated(function() {
var self = this;
self.totalPosts = new ReactiveVar();
});
/*then in my case, when the user clicks the load more -button,
we call the postsTotal-method and set the returned value as
the value of the totalPosts-reactive variable*/
Template.Posts.events({
'click .load-more': function (event, instance){
Meteor.call('postsTotal', function(error, result){
instance.totalPosts.set(result);
});
}
});
}
Hope this helps someone (I recommend checking the link first). For template level subscriptions, I used this as my guide Discover Meteor - template level subscriptions. This was my first stacked-post and I am just learning Meteor, so please have mercy...:D
Ouch this post is old, anyway maybe it will help someone.
I had exactly the same issue. I managed to solve it with 2 simple lines...
Remember the :
handle = Meteor.subscribeWithPagination('posts', 10);
Well I used in client handle.loaded() and Posts.find().count(). Because when they are different it means that all the posts are loaded. So here is my code :
"click #nextPosts":function(event){
event.preventDefault();
handle.loadNextPage();
if(handle.loaded()!=Posts.find().count()){
$("#nextPosts").fadeOut();
}
}
I had the same problem, and using the publish-counts package didn't work with the subs-manager package. I created a package that can set a reactive server-to-client session, and keep the document count in this session. You can find an example here:
https://github.com/auweb/server-session/#getting-document-count-on-the-client-before-limit-is-applied
I'm doing something like this:
On cliente
Template.postCount.posts = function() {
return Posts.find();
};
Then you create a template:
<template name="postCount">
{{posts.count}}
</template>
Then, whatever you want to show the counter: {{> postCount}}
Much easier than any solution i have seen.

Resources