I have a rather simple page with not so much data (about 20k documents in total), and it's VERY sluggish (complete load takes about 30-60 seconds). And what intrigues me most is that data appears gradually on page. Why is that? And why it's so slow?
Here is the page .jade file (it basically returns just order name and order's position name):
table(class="ui celled table")
thead
tr
th Order name
th Position name
tbody
each customerOrders
tr
td {{name}}
td
each customerOrderPositionModified
tr
td
td #{name}
The corresponding .js file is empty.
The router.js file (for Iron Router) is this:
this.route('/buyingList/:supplierUuid', {
data: function() {
var supplierUuid = this.params.supplierUuid;
var retOrd = [];
_.each(orders.find({checked: true}, {name: 1, "customerOrderPosition.name": 1, "customerOrderPosition.quantity": 1, "customerOrderPosition.supplierUuid": 1 }).fetch(), function (order) {
var ret = [];
_.each(order.customerOrderPosition, function (pos) {
var good = Goods.findOne({uuid: pos.goodUuid}, {name:1, supplierUuid:1});
if (good) {
if (good.supplierUuid == supplierUuid){
var company = Companies.findOne({uuid: good.supplierUuid}, {name: 1, uuid: 1});
var tt = {name: good.name, quantity: pos.quantity, companyName: (company ? company.name : "")};
ret.push(tt);
}
}
});
if (ret.length > 0) {
order.customerOrderPositionModified = ret;
retOrd.push(order);
}
});
return { customerOrders: retOrd };
},
name: 'buyingList'
});
The sizes of the collections:
Orders: 5k documents
Goods: 10k documents
Companies: 5k documents
The number of checked orders (which are returned in first query) is about 30.
What am I doing wrong? Or meteor is supposed to be so slow? I doubt it.
Thanks for help in advance.
summary
Meteor can be really slow if more than a few thousand docs are published at a time.
recommendation
Modify your publishers to accept the necessary parameters to publish only the subset of the database that you need to satisfy the UI requirements of the route your are on.
Related
I used a tutorial to help me sync my sheets to firebase with the use of a SYNC button that activates the script. The SYNC button currently sits just in the middle of the spreadsheet. I want to sync the data from sheets automatically to firebase when there are changes made.
function getFirebaseUrl(jsonPath) {
return (
'https://no-excusas.firebaseio.com/' +
jsonPath +
'.json?auth=' +
secret
)
}
function syncMasterSheet(sheetHeaders, sheetData) {
/*
We make a PUT (update) request,
and send a JSON payload
More info on the REST API here : https://firebase.google.com/docs/database/rest/start
*/
const outputData = [];
for(i = 0; i < sheetData.length; i++) {
var row = sheetData[i];
var newRow = {};
for(j = 0; j < row.length; j++) {
newRow[sheetHeaders[j]] = row[j];
}
outputData.push(newRow);
}
var options = {
method: 'put',
contentType: 'application/json',
payload: JSON.stringify(outputData)
}
var fireBaseUrl = getFirebaseUrl("UsersSheets")
UrlFetchApp.fetch(fireBaseUrl, options)
}
function startSync() {
//Get the currently active sheet
var sheet = SpreadsheetApp.getActiveSheet()
//Get the number of rows and columns which contain some content
var [rows, columns] = [sheet.getLastRow(), sheet.getLastColumn()]
// Get the data contained in those rows and columns as a 2 dimensional array.
// Get the headers in a separate array.
var headers = sheet.getRange(1, 1, 1, columns).getValues()[0]; // [0] to unwrap the
outer array
var data = sheet.getRange(2, 1, rows - 1, columns).getValues(); // skipping the header
row means we need to reduce rows by 1.
//Use the syncMasterSheet function defined before to push this data to the "masterSheet"
key in the firebase database
syncMasterSheet(headers, data)
}
Normally, it would be ok to just define an onEdit function in your code, like this:
function onEdit(event) {
startSync();
}
However, because you are making external requests via UrlFetchApp.fetch(), this will fail with an error about not having the https://www.googleapis.com/auth/script.external_request permission (gobs more detail about trigger authorization here).
Instead, you need to manually create an installable trigger
This is reasonably straightforward. In the edit menu for your code, go to your project's triggers:
Then, select "add a trigger" and create the on edit trigger, like so:
You should think about if you really want this running on every edit as the requests could be quite large (as it syncs the entire sheet) and run frequently (as you edit), however.
When you make a change to a spreadsheet, its onEdit event fires. So that's where you'd trigger that save with something like this:
function onEdit(event) {
startSync();
}
But since onEdit fires for each edit, this may end up saving a lot more than really necessary. So you may want to debounce to only save after some inactivity.
Something like this:
var timer;
function onEdit(event) {
// if we're counting down, stop the timer
if (timer) clearTimeout(timer);
// starting syncing after 2 seconds
timer = setTimeout(function() {
startSync();
}, 2000);
}
I have angular-meteor app that needs Material md-autocomplete from a collection with 53,296 documents with angularUtils.directives.dirPagination but this amount of data make my browser hang.
I'm publishing the collection with:
Meteor.publish('city', function (options, searchString) {
var where = {
'city_name': {
'$regex': '.*' + (searchString || '') + '.*' ,
'$options': 'i'
}
};
return City.find(where, options);
});
I subscribe with:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
and have pagination on controller :
$scope.currentPage = 1;
$scope.pageSize = 100;
$scope.sort = {city_name_sort : 1};
$scope.orderProperty = '1';
$scope.helpers({
city: function(){
return City.find({});
}
});
but it takes a long time to load and its make chrome stop working.
You already have most of the server-side searching done because your search is running inside a subscription. You should make sure that the city_name field is indexed in mongo! You should only return that field to minimize data transfer. You can also simplify your regex.
Meteor.publish('city', function (searchString) {
const re = new RegExp(searchString,'i');
const where = { city_name: { $regex: re }};
return City.find(where, {sort: {city_name: 1}, fields: {city_name: 1}});
});
What I've found helps with server-side auto-complete is:
Don't start searching until the user has typed 3 or 4 characters. This drastically narrows down the search results.
Throttle the search to only run every 500ms so that you're not sending every character to the server because then it has to keep re-executing the search. If the person is typing fast the search might only run every 2 or 3 characters.
Run the same .find() on the client that you're running on the server (instead of just querying for {}). That's just good practice since the client-side collection is the union of all subscriptions on that collection, there might be documents there that you don't want to list.
Lastly I don't know why you're subscribing twice here:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
only one of those Meteor.subscribe('city') calls is necessary.
I just want to check because I don't see it in the docs. Maybe I'm just missing it.
As far as I can tell if I subscribe with skip then I don't skip on the client. Correct?
I'm using iron router. I have code like this
Router.route('/docs/:_page', {
template: 'doclist',
subscriptions: function() {
var page = parseInt(this.params._page) - 1;
var skip = page * 10;
var limit = 10;
return Meteor.subscribe("pages", skip, limit);
},
});
The corresponding publish is this
Meteor.publish("pages", function (skip, limit) {
return Docs.find({}, {skip: skip, limit: limit});
});
But now in the template helper I don't use the skip AFAICT because there's only limit results in the MiniMongo
Template.doclist.helpers({
docs: function () {
var route = Router.current();
var pageId = parseInt(route.params._page) || 1;
var page = pageId - 1;
var skip = page * 10;
return Docs.find({}, {
// skip: skip
limit: limit,
});
},
});
It seems to work. If I comment in the skip line then I get no results on page 2.
Is that correct or am I doing something wrong?
You are correct - the client does not require a skip in this case. Let's say you have 100 documents in the DB and you skip the first 20 with a limit of 10. Then only 10 documents will exist on the client. Whenever you find on the client (in your templates), you are querying the local database (in this case 10 documents), so a skip would be inappropriate.
I'll caution that all of this is predicated on the notion that you have only one subscription for Docs. To extend the example above, if you had another 15 documents in the same collection on the client from another subscription, then you may need to do some additional filtering in order to show only the ones you are are interested in.
I have made a collection
var Words = new Meteor.Collection("words");
and published it:
Meteor.publish("words", function() {
return Words.find();
});
so that I can access it on the client. Problem is, this collection is going to get very large and I just want to publish a transform of it. For example, let's say I want to publish a summary called "num words by length", which is an array of ints, where the index is the length of a word and the item is the number of words of that length. So
wordsByLength[5] = 12;
means that there are 12 words of length 5. In SQL terms, it's a simple GROUP BY/COUNT over the original data set. I'm trying to make a template on the client that will say something like
You have N words of length X
for each length. My question boils down to "I have my data in form A, and I want to publish a transformed version, B".
UPDATE You can transform a collection on the server like this:
Words = new Mongo.Collection("collection_name");
Meteor.publish("yourRecordSet", function() {
//Transform function
var transform = function(doc) {
doc.date = new Date();
return doc;
}
var self = this;
var observer = Words.find().observe({
added: function (document) {
self.added('collection_name', document._id, transform(document));
},
changed: function (newDocument, oldDocument) {
self.changed('collection_name', oldDocument._id, transform(newDocument));
},
removed: function (oldDocument) {
self.removed('collection_name', oldDocument._id);
}
});
self.onStop(function () {
observer.stop();
});
self.ready();
});
To wrap transformations mentioned in other answers, you could use the package I developed, meteor-middleware. It provides a nice pluggable API for this. So instead of just providing a transform, you can stack them one on another. This allows for code reuse, permissions checks (like removing or aggregating fields based on permissions), etc. So you could create a class which allows you to aggregate documents in the way you want.
But for your particular case you might want to look into MongoDB aggregation pipeline. If there is really a lot of words you probably do not want to transfer all of them from the MongoDB server to the Meteor server side. On the other hand, aggregation pipeline lacks the reactivity you might want to have. So that published documents change counts as words come in and go.
To address that you could use another package I developed, PeerDB. It allows you to specify triggers which would be reactively called as data changes, and stored in the database. Then you could simply use normal publishing to send counts to the client. The downside is that all users should be interested in the same collection. It works globally, not per user. But if you are interested in counts of words per whole collection, you could do something like (in CoffeesScript):
class WordCounts extends Document
#Meta
name: 'WordCounts'
class Words extends Document
#Meta
name: 'Words'
triggers: =>
countWords: #Trigger ['word'], (newDocument, oldDocument) ->
# Document has been removed.
if not newDocument._id
WordCounts.update
length: oldDocument.word.length
,
$inc:
count: -1
# Document has been added.
else if not oldDocument._id
WordCounts.update
length: newDocument.word.length
,
$inc:
count: 1
# Word length has changed.
else if newDocument.word.length isnt oldDocument.word.length
WordCounts.update
length: oldDocument.word.length
,
$inc:
count: -1
WordCounts.update
length: newDocument.word.length
,
$inc:
count: 1
And then you could simply publish WordCounts documents:
Meteor.publish 'counts', ->
WordCounts.documents.find()
You could assemble the counts by going through each document in Words, (cursor for each)
var countingCursor = Words.find({});
var wordCounts = {};
countingCursor.forEach(function (word) {
wordCounts[word.length].count += 1;
wordCounts[word.length].words = wordCounts[word.length].words || []
wordCounts[word.length].words.push(word);
});
create a local collection,
var counts = new Meteor.Collection('local-counts-collection', {connection: null});
and insert your answers
var key, value;
for (key in wordCounts) {
value = object[key];
counts.insert({
length: key,
count: value.count,
members: value.words
});
}
Counts is now a collection, just not stored in Mongo.
Not tested!
I have a publication whose scope depends on a element property from another collection. Basically it looks like this on the server:
Meteor.publish('kids', function (parent) {
return Kids.find({ _id: { $in: parent.childrenIds } });
}
In the example above parent.childrenIds is an array containing the _id's of all the kids that are children of the parent. This works fine until I want to add a new child to the parent:
newKidId = Kids.insert({ name: 'Joe' });
Parents.update({ _id: parentId }, { $push: { childrenIds: newKidId } });
This works on the server for the Kids collection (i.e., the new kid is added) and it updates the parent's childrenIds array with the newKidId. BUT it does not update the above 'kids' publication (the cursor is not updated/modified). As a result, the client's Kids collection is not updated (and it looks like the change to Kids is rolled back on the client).
When the client refreshes, all publications are stopped/restarted and the new kid (Joe) is finally published to the client.
Is there a way to avoid refreshing the client and forcing the re-publication of the Kids collection (ideally only sending the new kid Joe to the client)?
One of the things that is often misunderstood in Meteor is that there is no reactivity on the server. Dynamic descriptions need to be handled by Deps.autorun blocks on the client. To do this, first make sure you are not including the autopublish package by using this command in project directory:
$ meteor remove autopublish
Second, set up an autorun block on the client like:
Meteor.startup(function(){
Meteor.subscribe('parents');
Deps.autorun(function() {
parent = Parents.findOne();
if (!parent) return;
Meteor.subscribe('kids', parent);
});
});
This will tear down and set up subscriptions as the parent object changes.
You can see a full working example at https://gist.github.com/jagill/5473599 .
In meantime there were quite some packages made to deal with reactive publish functions. I am an author of meteor-related, and at the end of the package's README I compare my package with few other packages:
meteor-reactive-publish
meteor-publish-with-relations
meteor-smart-publish
reywood:publish-composite
copleyjk:simple-publish
These days you can simply use reactive-publish package (I am one of authors):
Meteor.publish('kids', function (parentId) {
this.autorun(function (computation) {
var parent = Parents.findOne(parentId, {fields: {childrenIds: 1}});
return Kids.find({_id: {$in: parent.childrenIds}});
});
}
It is important to limit Parents' query fields only to childrenIds so that autorun does not rerun for any other changes to Parents document.
I think you need to use observe in the publish function if a published query relies on a second query. Deps.autorun on the client is not necessary.
See discussions on Meteor server reactivity and Reactive updates when query is filtered by another query.
This is some code based on http://docs.meteor.com 'counts-by-room' example.
Meteor.publish( "kids", function(parent_id){
var self = this;
Parents.find({_id: parent_id}, { childrenIds: 1 }).observe({
added: function (document){
document.childrenIds.forEach( function(kidId){
self.added("kids", kidId, Kids.findOne( { _id: kidId}, {name: 1, _id: 1} ));
});
},
removed: function (oldDocument){
oldDocument.childrenIds.forEach( function(kidId){
self.removed("kids", kidId, Kids.findOne( { _id: kidId}, {name: 1, _id: 1} ));
});
},
changed: function (newDocument, oldDocument){
var oldLength = oldDocument.childrenIds.length;
var newLength = newDocument.childrenIds.length;
if (newLength > oldLength){
self.added("kids",
newDocument.childrenIds[newLength-1],
Kids.findOne( { _id: newDocument.childrenIds[newLength-1] }, {name:1, _id:1}) );
}
else{
self.removed("kids",
oldDocument.childrenIds[oldLength-1],
Kids.findOne( { _id: oldDocument.childrenIds[oldLength-1] }, {name:1, _id:1}) );
}
}
});
self.ready();
});