I have angular-meteor app that needs Material md-autocomplete from a collection with 53,296 documents with angularUtils.directives.dirPagination but this amount of data make my browser hang.
I'm publishing the collection with:
Meteor.publish('city', function (options, searchString) {
var where = {
'city_name': {
'$regex': '.*' + (searchString || '') + '.*' ,
'$options': 'i'
}
};
return City.find(where, options);
});
I subscribe with:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
and have pagination on controller :
$scope.currentPage = 1;
$scope.pageSize = 100;
$scope.sort = {city_name_sort : 1};
$scope.orderProperty = '1';
$scope.helpers({
city: function(){
return City.find({});
}
});
but it takes a long time to load and its make chrome stop working.
You already have most of the server-side searching done because your search is running inside a subscription. You should make sure that the city_name field is indexed in mongo! You should only return that field to minimize data transfer. You can also simplify your regex.
Meteor.publish('city', function (searchString) {
const re = new RegExp(searchString,'i');
const where = { city_name: { $regex: re }};
return City.find(where, {sort: {city_name: 1}, fields: {city_name: 1}});
});
What I've found helps with server-side auto-complete is:
Don't start searching until the user has typed 3 or 4 characters. This drastically narrows down the search results.
Throttle the search to only run every 500ms so that you're not sending every character to the server because then it has to keep re-executing the search. If the person is typing fast the search might only run every 2 or 3 characters.
Run the same .find() on the client that you're running on the server (instead of just querying for {}). That's just good practice since the client-side collection is the union of all subscriptions on that collection, there might be documents there that you don't want to list.
Lastly I don't know why you're subscribing twice here:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
only one of those Meteor.subscribe('city') calls is necessary.
Related
I am trying to make a fairly big meteor app, and I noticed that it has gotten slower over the past few days and I read on the meteor forum that publications can cause slow loading times. After I refresh the page when making a change in the application itself (code change), it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
if(Meteor.isServer){
Meteor.publish('notes', function () {
return Notes.find()
});
Meteor.publish('users', function () {
return Meteor.users.find()
});
Meteor.publish("user", function(){
return Meteor.user()
})
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
}
document example:
let noteInfo = { title, subject, description, imageURL, userId, userEmail, createdAt }
let title = this.refs.title.value;
let subject = this.refs.subject.value;
let description = this.refs.description.value;
let allUrls = [this.refs.imageURL.value].concat(this.state.urls);
let imageURL = allUrls.filter(function(entry) { return entry.trim() != ''; });
let userId = Meteor.userId();
let userEmail = Meteor.user().emails[0].address;
let createdAt = Date.parse(new Date());
This is a very broad performance tuning question. You haven't told us how many documents are in your collections or how large these documents are. Some possible issues:
You are over-publishing either by publishing too many documents or because your documents are too big. For example when you do:
Meteor.publish('notes', function () {
return Notes.find()
});
If there are 100,000 Notes documents at 100 bytes each then that's 10 MB that needs to go over the network to the client. This is also the case if there are 1,000 notes documents at 10KB each.
Solution: Limit the number of documents with limit and/or reduce the number of fields transmitted with fields:
Meteor.publish('notes', function () {
return Notes.find({},{ limit: 100, fields: { key1: 1, key2: 1 }});
});
Your collection is missing one or more indexes. When you do:
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
if there are 1M notes documents but there is no index on the createdAt key then this will be terribly slow.
Solution: Adding an index on the createdAt key will make such a publication much faster.
You have an invalid publication.
Meteor.publish("user", function(){
return Meteor.user();
})
Is invalid because Meteor.user() is not a cursor and a publication must return either a cursor or an array of cursors. It is also redundant because Meteor.user() is automatically available on the client, albeit it doesn't include all the keys.
Solution: Remove this unnecessary publication altogether. If you want to publish some of the keys that are not available on the client for the current user you can do so as follows:
Meteor.publish("user", function(){
return Meteor.users.find(this.userId,{fields: {services: 1, emails: 1, profile: 1}})
});
After I refresh the page when making a change, it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
What does this mean? Do you mean when you change the code of your website, it takes 1-2 minutes for your changes to appear in a newly loaded page? That's to be expected: it takes time for Meteor to rebuild your application. Watch the terminal for the progress.
Using lots of packages and external code will slow down building. The speed of compilation is unrelated to your end user's experience.
A couple things I do with my meteor projects is that I send specific fields "profile.name" etc. I've also used collection.createIndex({}) which creates an index and inside it and you could put your sort in which is created on server startup and works well with your collection.find.sort. I think limiting your notes would be quite essential as you have a limit on one and not the other.
It happened to me too. At that time I checked with kadira (formerly still opensource but now it is not). Source code at kadira on github, but service is gone kadira.io
The problem is in pubication and subscribtion. When client request subscribtions, he opens up some kind of "connection" - i say that, and it applies to others who call the same publication. imagine how many connections are formed.
where the same data accessed by many people must server side must request data to mongodb, that same to all of number of such connections. This is what makes it slow.
Finally I added redis.io.
meteor npm install --save redis
meteor npm install --save hiredis
On server side :
import { Meteor } from "meteor/meteor";
var redis = require("redis");
var clientRedis = redis.createClient({
host: "YOURREDIS_IP",
port: "YOURREDIS_PORT"
});
clientRedis.setSync = Meteor.wrapAsync(clientRedis.set);
clientRedis.getSync = Meteor.wrapAsync(clientRedis.get);
setRedis_Object = function (keyREDIS, timeRefresh, valueREDIS) {
return clientRedis.setSync(keyREDIS + moment(new Date()).format(timeRefresh), JSON.stringify(valueREDIS));
};
getRedis_Object = function (keyREDIS, timeRefresh) {
return JSON.parse(clientRedis.getSync(keyREDIS + moment(new Date()).format(timeRefresh)));
};
Meteor.methods({
getRedis_YOURCOLLECTIONS: function(timeRefresh) {
var data = getRedis_Object(getRedis_YOURCOLLECTIONS, timeRefresh);
if (data != null) {
return data;
} else {
var data = YOURCOLLECTIONS.find().fetch();
setRedis_Object(key, timeRefresh, data);
return data
}
},
});
On Client Side
Meteor.call("getRedis_YOURCOLLECTIONS", "YYYYMMDD", function (error, result) {
if (error) {
console.log(error);
} else {
Session.set("getRedis_YOURCOLLECTIONS", result);
}
});
After i am add this REDIS consumtion Memory and CPU on server low, and app more Fast. I hope its work for you. Thanks
Clearly, I am doing something wrong with ReactiveVar because I cannot get it to work as I expect it should.
I am trying to set the value of an ReactiveVar by calling a Meteor.call method which returns the list of usernames. But it does not update when the usernames get changed in another part of the app.
I tried both:
Template.qastatistics.created = function () {
this.trackUsernames = new ReactiveVar(false);
var instance = Template.instance();
Meteor.call('trackUsernames', function (err, data) {
instance.trackUsernames.set(data);
});
};
and:
Template.qastatistics.helpers({
users: function () {
var usernames,
instance = Template.instance();
if (instance.trackUsernames.get() === false) {
Meteor.call('trackUsernames', function (err, data) {
instance.trackUsernames.set(data);
});
}
usernames = instance.trackUsernames.get();
...
But neither updates the list of usernames when these change in the database.
Is this even possible with ReactiveVars or have I completely misunderstood them?
EDIT: The usernames I mention are not from Meteor.users collection, but rather a distinct call from another collection that has usernames in it.
Fist of all I would use the onCreated function instead of defining created. That's a little more extendable and it's the new API. created is just kept around for backwards compatibility.
About your problem. You are right, you seem to have misunderstood what ReactiveVars do. They are a reactive data source. That means that when you call myReactiveVar.get in some Tracker.autorun (aka. reactive computation), the computation will rerun whenever myReactiveVar.set is called.
You got the first part right. Spacebars helpers always run inside their own computation. What you got wrong is thinking that a method call is a reactive action. That means, that you could call trackUsernames and set the trackUsernames ReativeVar again and the value in your template would update itself. But a method is only run once. It doesn't do anything fancy with reactivity.
A method call only transfers data once. When you publish a set of documents (like all users) on the other hand, they will be updated dynamically. Whenever a change happens inside that set of published documents, it will be synced to the client. So in general, it's a better idea to use publications and subscriptions to sync data reactively. If you'd want to use a method for the same thing you'd need to do some kind of polling (so your back in the stone-age again).
The easiest way to implement what you are trying to do is to use Meteor.users.find().fetch(). As it says in the docs fetch registers dependencies for all the documents you are fetching if it's being called from within a reactive computation.
First you'll need to properly set up your publications, so that users can see other users usernames. I'll leave that to you. Then you need to reimplement your helper
Template.qastatistics.helpers({
users: function () {
var usernames = _.pluck(Meteor.users.find().fetch(), 'username');
...
Thanks to suggestions from #kyll, I managed to get what I wanted by publishing the data I need:
server:
cope.publish.usernamesID = Random.id();
Meteor.publish("itemsusernames", function () {
self = this;
var initializing = true;
var handle = Items.find().observeChanges({
added: function (id) {
!initializing && self.changed(
"itemsusernames",
cope.publish.usernamesID,
Items.distinct("p4User"));
},
changed: function (id) {
!initializing && self.changed(
"itemsusernames",
cope.publish.usernamesID,
Items.distinct("p4User"));
},
removed: function (id) {
!initializing && self.changed(
"itemsusernames",
cope.publish.usernamesID,
Items.distinct("p4User"));
}
});
initializing = false;
self.added("itemsusernames", cope.publish.usernamesID, Items.distinct("p4User"));
self.ready();
self.onStop(function () {
handle.stop();
});
});
client:
users: function () {
var usernames = [],
oUsernames = ItemsUsernames.find().fetch();
if (!oUsernames[0]) return [];
usernames = $.map(oUsernames[0], function (value, index) {
if (!isNaN(index)) {
return [value];
}
});
...
And ofcourse: ItemsUsernames = new Mongo.Collection("itemsusernames");
Meteor Collections have a transform ability that allows behavior to be attached to the objects returned from mongo.
We want to have autopublish turned off so the client does not have access to the database collections, but we still want the transform functionality.
We are sending data to the client with a more explicit Meteor.publish/Meteor.subscribe or the RPC mechanism ( Meteor.call()/Meteor.methods() )
How can we have the Meteor client automatically apply a transform like it will when retrieving data directly with the Meteor.Collection methods?
While you can't directly use transforms, there is a way to transform the result of a database query before publishing it. This is what the "publish the current size of a collection" example describes here.
It took me a while to figure out a really simple application of that, so maybe my code will help you, too:
Meteor.publish("publicationsWithHTML", function (data) {
var self = this;
Publications
.find()
.forEach(function(entry) {
addSomeHTML(entry); // this function changes the content of entry
self.added("publications", entry._id, entry);
});
self.ready();
});
On the client you subscribe to this:
Meteor.subscribe("publicationsWithHTML");
But your model still need to create a collection (on both sides) that is called 'publications':
Publications = new Meteor.Collection('publications');
Mind you, this is not a very good example, as it doesn't maintain the reactivity. But I found the count example a bit confusing at first, so maybe you'll find it helpful.
(Meteor 0.7.0.1) - meteor does allow behavior to be attached to the objects returned via the pub/sub.
This is from a pull request I submitted to the meteor project.
Todos = new Meteor.Collection('todos', {
// transform allows behavior to be attached to the objects returned via the pub/sub communication.
transform : function(todo) {
todo.update = function(change) {
Meteor.call('Todos_update', this._id, change);
},
todo.remove = function() {
Meteor.call('Todos_remove', this._id);
}
return todo;
}
});
todosHandle = Meteor.subscribe('todos');
Any objects returned via the 'todos' topic will have the update() and the remove() function - which is exactly what I want: I now attach behavior to the returned data.
Try:
let transformTodo = (fields) => {
fields._pubType = 'todos';
return fields;
};
Meteor.publish('todos', function() {
let subHandle = Todos
.find()
.observeChanges({
added: (id, fields) => {
fields = transformTodo(fields);
this.added('todos', id, fields);
},
changed: (id, fields) => {
fields = transformTodo(fields);
this.changed('todos', id, fields);
},
removed: (id) => {
this.removed('todos', id);
}
});
this.ready();
this.onStop(() => {
subHandle.stop();
});
});
Currently, you can't apply transforms on the server to published collections. See this question for more details. That leaves you with either transforming the data on the client, or using a meteor method. In a method, you can have the server do whatever you want to the data.
In one of my projects, we perform our most expensive query (it joins several collections, denormalizes the documents, and trims unnecessary fields) via a method call. It isn't reactive, but it greatly simplifies our code because all of the transformation happens on the server.
To extend #Christian Fritz answer, with Reactive Solution using peerlibrary:reactive-publish
Meteor.publish("todos", function() {
const self = this;
return this.autorun(function(computation) {
// Loop over each document in collection
todo.find().forEach(function(entry) {
// Add function to transform / modify each document here
self.added("todos", entry._id, entry);
});
});
});
I have two collections
Offers (relevant fields: _id)
ShareRelations (relevant fields: receiverId and offerId)
and I'd like to publish only Offers to the logged in user which have been shared to him.
Actually, I'm doing this by using a helper array (visibleOffers) which I fill by looping for each ShareRelations and use this array later on the Offers.find as $in selector.
I wonder if this might be the meteor way to do this, or if I could do with less and/or prettier code?
My actual code to publish the Offers is the following:
Meteor.publish('offersShared', function () {
// check if the user is logged in
if (this.userId) {
// initialize helper array
var visibleOffers = [];
// initialize all shareRelations which the actual user is the receiver
var shareRelations = ShareRelations.find({receiverId: this.userId});
// check if such relations exist
if (shareRelations.count()) {
// loop trough all shareRelations and push the offerId to the array if the value isn't in the array actually
shareRelations.forEach(function (shareRelation) {
if (visibleOffers.indexOf(shareRelation.offerId) === -1) {
visibleOffers.push(shareRelation.offerId);
}
});
}
// return offers which contain the _id in the array visibleOffers
return Offers.find({_id: { $in: visibleOffers } });
} else {
// return no offers if the user is not logged in
return Offers.find(null);
}
});
Furthermore, the actual solution has the downside that if a new share relations is being created, the Offers collection on the client doesn't get updated with the newly visible offer instantly (read: page reload required. But I'm not sure if this is the case because of this publish method or because of some other code an this question is not primary because of this issue).
What you are looking for is a reactive join. You can accomplish this by directly using an observe in the publish function, or by using a library to do it for you. Meteor core is expected to have a join library at some point, but until then I'd recommend using publish-with-relations. Have a look at the docs, but I think the publish function you want looks something like this:
Meteor.publish('offersShared', function() {
return Meteor.publishWithRelations({
handle: this,
collection: ShareRelations,
filter: {receiverId: this.userId},
mappings: [{collection: Offers, key: 'offerId'}]
});
});
This should reactively publish all of the ShareRelations for the user, and all associated Offers. Hopefully publishing both won't be a problem.
PWR is a pretty legit package - several of us use it in production, and Tom Coleman contributes to it. The only thing I'll caution you about is that as of this writing, the current version in atmosphere (v0.1.5) has a bug which will result in a fairly serious memory leak. Until it gets bumped, see my blog post about how to run an updated local copy.
update 2/5/14:
The discover meteor blog has an excellent post on reactive joins which I highly recommend reading.
The way to do this is along the lines of this Question using observeChanges(). Still trying to figure out how to get it all working for my example, see Meteor, One to Many Relationship & add field only to client side collection in Publish?
You can use the reactive-publish package (I am one of authors):
Meteor.publish('offersShared', function () {
// check if the user is logged in
if (this.userId) {
this.autorun(function (computation) {
// initialize helper array
var visibleOffers = [];
// initialize all shareRelations which the actual user is the receiver
var shareRelations = ShareRelations.find({receiverId: this.userId}, {fields: {offerId: 1}});
// loop trough all shareRelations and push the offerId to the array if the value isn't in the array actually
shareRelations.forEach(function (shareRelation) {
if (visibleOffers.indexOf(shareRelation.offerId) === -1) {
visibleOffers.push(shareRelation.offerId);
}
});
// return offers which contain the _id in the array visibleOffers
return Offers.find({_id: { $in: visibleOffers } });
});
} else {
// return no offers if the user is not logged in
return Offers.find(null);
}
});
You can simply wrap your existing non-reactive code into an autorun and it will start to work. Just be careful to be precise which fields you query on because if you query on all fields then autorun will be rerun on any field change of ShareRelations, not just offerId.
Background
I have "Lists" and "Products" collections, Products belong to a List
A List has a description, from which products are generated
On startup, a new List is created that's unique for that visitor
The List id is stored in the Session
What I Want
I want Products to be generated when the description of a List changes.
The first step is that when the list for the current visitor is changed, I want a new product to be inserted.
I get the feeling I'm going about this totally wrong...
The Problem
The product is inserted, appears in the browser for a split second, then vanishes. It's been removed by Meteor.
Code
Products = new Meteor.Collection("products");
Lists = new Meteor.Collection("lists");
if (Meteor.isClient) {
Meteor.startup(function () {
var my_list_id = Lists.insert({description: "Default list"});
Session.set("my_list", my_list_id);
var observed = Lists.find({_id: my_list_id}).observe({
changed: function (newDocument, oldDocument) {
Products.insert({list: newDocument._id, name: newDocument.description});
}
});
});
toggleElement = function (elementName) {
if(editedElementIs(elementName)) {
var newListDescription = $('textarea').val();
Lists.update(Session.get("my_list"), {description: newListDescription});
setEditedElement("");
} else {
setEditedElement(elementName);
}
};
// Including the rest in case I've misunderstood something.
// I don't see how any of this could cause the issue.
setEditedElement = function (elementName) {
return Session.set("edited_element", elementName);
};
editedElementIs = function (elementName) {
return Session.get("edited_element") == elementName;
};
Handlebars.registerHelper('editedElementIs', editedElementIs);
Handlebars.registerHelper('products', function() {
return Products.find({list: Session.get("my_list")});
});
Template.list_form.listDescription = function () {
return Lists.findOne({_id: Session.get("my_list")}).description;
};
Template.adminbar.events({
'click a#editlist' : function () {
toggleElement("list");
},
'click a#editsidebar' : function () {
toggleElement("sidebar");
}
});
}
if (Meteor.isServer) {
Meteor.startup(function () {
});
}
What I've Tried
Obviously, I can just do this:
if(editedElementIs(elementName)) {
var newListDescription = $('textarea').val();
Products.insert({list: Session.get("my_list"), name: newListDescription});
Lists.update(Session.get("my_list"), {description: newListDescription});
...
But that's writing clumsy update code that I'd like to house in an observer.
It looked like the product was being removed. So I've observed when a product is removed thus:
Products.find({list:my_list_id}).observe({
removed: function (oldDocument) {
throw error("wow");
console.log("Removed Product" + oldDocument);
}
})
and this observer is called immediately after the Product is inserted.
I get the stack trace:
at Object.Products.find.observe.removed (http://localhost:3000/ListyMeteor.js?2d867b7481df6389658be864b54d864151e87da5:22:15)
at Object.cursor.observeChanges.removed (http://localhost:3000/packages/minimongo/minimongo.js?daa88dc39d67b40b11d6d6809d72361f9ef6a760:909:52)
at http://localhost:3000/packages/minimongo/minimongo.js?daa88dc39d67b40b11d6d6809d72361f9ef6a760:275:15
at _.extend.runTask (http://localhost:3000/packages/meteor/fiber_stubs_client.js?52687e0196bc1d3184ae5ea434a8859275702d94:30:11)
at _.extend.flush (http://localhost:3000/packages/meteor/fiber_stubs_client.js?52687e0196bc1d3184ae5ea434a8859275702d94:58:10)
at _.extend.drain (http://localhost:3000/packages/meteor/fiber_stubs_client.js?52687e0196bc1d3184ae5ea434a8859275702d94:66:12)
at LocalCollection.remove (http://localhost:3000/packages/minimongo/minimongo.js?daa88dc39d67b40b11d6d6809d72361f9ef6a760:500:22)
at Object.self._connection.registerStore.update (http://localhost:3000/packages/mongo-livedata/collection.js?682caa185350aa26968d4ffc274579a33922f0e6:109:32)
at Object.store.(anonymous function) [as update] (http://localhost:3000/packages/livedata/livedata_connection.js?5d09753571656c685bb10c7970eebfbf23d35ef8:404:48)
at http://localhost:3000/packages/livedata/livedata_connection.js?5d09753571656c685bb10c7970eebfbf23d35ef8:984:19
It looks like Meteor is flushing the Products collection on the client side.
I'm clearly misunderstanding how Meteor works.
Any ideas on why this is happening?
Update 1
It looks like this is happening because insert is being called within an observer:
Why does meteor undo changes to collections nested in an observer method?
I'll post back here once I confirm.
Is autosubscribe turned on or off?
If you turn autosubscribe off, it could happen that your client updates the server copy and then on a subsequent update from the server - does not get all the items because its not subscribed to that collection.
Easiest way to check is to query the mongo db -
meteor mongo
Query the mongo db if your product has been added to the document.
If it has, then it is an autosubscribe issue -
You will have to create publish (on server) and subscribe (on client) methods as given here http://docs.meteor.com/#meteor_publish