I am writing an app that involves creating multiple subsets of the same collection, and publishing them under different record sets following this example.
Using this principle, I am creating ad hoc record sets. The publish code is in a method that gets called per template:
//Template
Template.item._item = function() {
Meteor.call('publishMethod', foo);
Meteor.subscribe('name-'+foo);
return someFunction(foo);
}
//Method
Meteor.methods({
'publishMethod' = function(foo) {
Meteor.publish('name-'+foo, function() { someFunction(foo); });
});
});
//Common area
someFunction = function(foo) {
return Collection.find({'foobar' : foo});
}
In this example, someFunction() sits in common area between client and server. someFunction() returns a subset of a collection based on foo.
I have some questions regarding the above approach:
When a method is called with the same foo value, Meteor prints "Ignoring duplicate publish named 'name-foo'". Is there any way to check if a record set exists?
There is concern that these record sets will continue to be published and not release memory. Are these record sets client side only? Or will they accumulate on the server?
This is the best approach I have found for dealing with multiple, complex queries on the same large dataset, and allows for specific fields to be sent per request and page. I am however open to suggestions.
Thanks in advance.
Related
I have built a small meteor app based on code generated by the excellent Meteor Kitchen project. This code works and renders the collection to the page, but there is one thing I am confused about.
A subset of the code is here:
router.js
this.route("articles", {path: "/articles", controller: "ArticlesController"});
ArticlesController
this.ArticlesController = RouteController.extend({
template: "Articles",
onBeforeAction: function() {
this.next();
},
action: function(){
if (this.isReady()) {
this.render();
} else {
this.render("loading");
}
},
isReady: function() {
var ready = true;
var subs = [ Meteor.subscribe('allArticles') ];
_.each(subs, function(sub) {
if(!sub.ready())
ready = false;
});
return ready;
},
data: function() {
return {
articles: Articles.find({})
};
}
});
server/pubs/articles.js
Meteor.publish('allArticles', function() {
return Articles.find({});
});
Meteor.publish('singleArticle', function(articleId) {
check(articleId, String);
return Articles.find({_id: articleId});
});
As I understand how this code is working, the following takes place:
(1) Collection is published via allArticles and singleArticle subscriptions
(2) ArticlesController subscribes to allArticles
(3) data function in the ArticlesController extracts the data (from the subscription?) to the articles array which is then exposed to the Blaze template.
Where I am confused:
Why do we need to do a Articles.find({}) in the data function? Where do we access the allArticles data ... it seems we are going back to the Articles collection directly and how is that possible if we have subscribed only to allArticles ?
While you don't show it, I'm assuming you have the following line in your code, defined somewhere that will execute it on both the server, and the client:
Articles = new Mongo.Collection('articles');
/*CollectionName = new Mongo.Collection('DBCollectionName');*/
Docs. When this is executed on the server a Collection is created, and assigned to the variable name Articles. 'articles' is the name used to store this collection in MongoDB.
When this is executed on the client, a mini mongo Collection is created. It initially will have no documents in it.
Next you have server only code:
Meteor.publish('allArticles', function() {
return Articles.find({});
});
Meteor.publish('singleArticle', function(articleId) {
check(articleId, String);
return Articles.find({_id: articleId});
});
Docs. This defines two publications, 'allArticles' and 'singleArticle'. These are not Collections themselves, but are rules that specify a set of data that the server will publish, and a client may subscribe to. While these two publications return data from the Server's Articles collection, publications can return data from one or more collections, or by directly using the underlying ddp protocol you can publish data that comes from another data source (not mongodb).
Next on the client you subscribe to a collection:
Meteor.subscribe('allArticles')
Docs. This call takes the name of a publication defined on the server ('allArticles'), and subscribes to it. The server then executes the publish function, and sends over ddp the set of data returned. This data is stored in the Client-side Mini Mongo Collection created above, and named Articles.
Also the server will monitor the Articles collection for changes, and if the resultset of the 'allArticles' publication changes, will send these changes as updates to the client.
So next you have the data function in your Controller (Client side).
data: function() {
return {
articles: Articles.find({})
};
}
Docs. This sets the data context for the render function.
The reason this calls Articles.find rather than allArticles.find is because allArticles is not a collection, but was instead the name of the publication the client used to request the server send data, that was stored in the clients mini mongo collection named Articles.
Where do we access the allArticles data ... it seems we are going back
to the Articles collection directly and how is that possible if we
have subscribed only to allArticles ?
You return this as part of your data object so that you can access it in your template. In your Articles template you can now use {{#each articles}} directly without a helper function because articles is part of your data context. You can also access the articles returned from your controllers data portion inside of your Articles template helpers by using this.articles.
Why do we need to do a Articles.find({}) in the data function?
These queries being performed in your controllers data function act on the clients minimongo Articles collection as opposed to the servers. Once the information is published from the server, and the client has subscribed to it, the client has this information available in their minimongo instance, but still needs to access it somehow. Basically, the publication makes the information available, but the Articles.find({}) accesses it for the client.
Accessing this information inside of the data function of your controller is simply to avoid doing it inside of your template.
I think that your misunderstanding comes from the third step that you have described:
data function in the ArticlesController extracts the data (from the
subscription?) to the articles array which is then exposed to the
Blaze template.
The data function extracts the data from minimongo on the client which contains the information from the subscription. Minimongo lies between the subscription and the data function.
I would need to know more about your app to answer the question. Are you viewing just a single article, or is there a page that lists them all?
If you are viewing a single article you would need to subscribe to the singleArticle publication.
If you were showing a list of articles, you would need to subscribe to allArticles. If there are a lot of articles, you could improve the speed of your app by limiting the number of fields with a query projection.
I have a custom publication on my server (which in some way join 2 collections).
This resulting set of this publication is exactly what I need but for performances issues I would like to avoid sending it entirely to the client.
If I did not care about performances, I would only subscribe to the
publication and do something like
theCollection.find({"my":"filter"})
I am therefore trying to find a way to publish a subset of the custom publication so that the filter would be applied on the custom publication on the server side.
Is there a way to chain or filter publications (server side) ?
For the question we can assume the custom publication to look like this and cannot be modified:
Meteor.publish('customPublication', function() {
var sub = this;
var aCursor = Resources.find({type: 'someFilter'});
Mongo.Collection._publishCursor(aCursor, sub, 'customPublication');
sub.ready();
});
if i understand the question right, you are looking for https://atmospherejs.com/reywood/publish-composite
It let's you "publish a set of related documents from various collections using a reactive join. This makes it easy to publish a whole tree of documents at once. The published collections are reactive and will update when additions/changes/deletions are made."
Ok I came to the following workaround. Instead of working on the publication, I simply added a new collection I update according to the other collections. In order to do so I am using the meteor hooks package
function transformDocument(doc)
{
doc.aField = "aValue"; // do what you want here
return doc;
}
ACollection.after.insert(function(userId, doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
ACollection.after.update(function(userId, doc, fieldNames, modifier, options)
{
var transformedDocument = transformDocument(doc);
delete transformedDocument._id;
AnotherCollection.update(doc._id,{$set:transformedDocument});
});
ACollection.after.remove(function(userId, doc)
{
AnotherCollection.remove(doc._id);
});
Then I have the new collection I can publish subsets the regular way
Benefits:
You can filter whatever you want into this db, no need to worry if the field is virtual or real
Only one operation every time a db changes. This avoid having several publication merging the same data
Cave eats:
This requires one more Collection = more space
The 2 db might not be always synchronised, there is few reasons for this:
The client manually changed the data of "AnotherCollection"
You had documents in "ACollection" before you added "AnotherCollection".
The transform function or source collection schema changed at some point
To fix this:
AnotherCollection.allow({
insert: function () {
return Meteor.isServer;
},
update: function () {
return Meteor.isServer;
},
remove: function () {
return Meteor.isServer;
}
});
And to synchronise at meteor startup (i.e. build the collection from scratch). Do this only once for maintenance or after adding this new collection.
Meteor.startup(function()
{
AnotherCollection.remove({});
var documents = ACollection.find({}).fetch();
_.each(documents, function(doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
});
I'd like to create a method that returns the count of a generic collection.
Calling the method would look something like this:
Meteor.call('getCollectionCount', 'COLLECTION_NAME');
And the result would be the collection count.
The server method code would look something like this:
getCollectionCount: function (collectionName) {
return window[collectionName].find().count();
}
This won't work because window isn't defined on the server, but is something similar possible?
Use global instead of window.
Note that this uses the variable name assigned to the collection object, not the name given to the collection. For this to work with Meteor.users you need to assign another variable name.
if (Meteor.isServer) {
users = Meteor.users;
}
if (Meteor.isClient) {
Meteor.call('count', 'users', function (err, res) {
// do something with number of users
});
}
Also probably a good idea to check that global[collectionName] is actually a collection.
I came up with this code which makes the following assumptions :
collections are declared in the global scope as top level objects.
collections are searched by collection name, not the collection variable identifier.
So client code should declare their collections like this :
MyCollection=new Meteor.Collection("my-collection");
And use the function like this :
var clientResult=Meteor.call("getCollectionCount","my-collection",function(error,result){
if(error){
console.log(error);
return;
}
console.log("actual server-side count is : ",result);
});
console.log("published subset count is : ",clientResult);
The method supports execution on the client (this is known as method stub or method simulation) but will only yield the count of the collection subset replicated client-side, to get the real count wait for server-side response using a callback.
/packages/my-package/lib/my-package.js
getCollection=function(collectionName){
if(collectionName=="users"){
return Meteor.users;
}
var globalScope=Meteor.isClient?window:global;
for(var property in globalScope){
var object=globalScope[property];
if(object instanceof Meteor.Collection && object._name==collectionName){
return object;
}
}
throw Meteor.Error(500,"No collection named "+collectionName);
};
Meteor.methods({
getCollectionCount:function(collectionName){
return getCollection(collectionName).find().count();
}
});
As Meteor.users is not declared as a top level variable you have to account for the special case (yes, this is ugly).
Digging into Meteor's collection handling code could provide a better alternative (getting access to a collection handle by collection name).
Final words on this : using a method call to count a collection documents is unfortunately non-reactive, so given the Meteor paradigm this might be of little use.
Most of the time you will want to fetch the number of documents in a collection for pagination purpose (something like a "Load more" button in a posts list for example), and as the rest of the Meteor architecture you'll want this to be reactive.
To count documents in a collection reactively you'll have to setup a slightly more complicated publication as showcased in the "counts-by-room" example in the docs.
http://docs.meteor.com/#meteor_publish
This is something you definitely want to read and understand.
This smart package is actually doing it right :
http://atmospherejs.com/package/publish-counts
It provides a helper function that is publishing the counts of any cursor.
Keep track of the collections on some other property that the server has access too. You could even call it window if you really wanted to.
var wow = new Meteor.Collection("wow");
collections["wow"] = wow;
getCollectionCount: function (collectionName) {
return collections[collectionName].find().count();
}
If you don't want the package users to change how they work with collections in the app then I think you should use MongoInternals to get collections by name from the db. Not tested but here is an example:
//on server
Meteor.methods({
count: function( name ){
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
var collection = db.collection( name );
return collection && collection.count({});
}
});
Another example of MongoInternals use is here. Documentation of the count() function available from the mongo driver is here.
I have an app that loads a Jobs collection on startup.
Deps.autorun(function(){
Meteor.subscribe('jobs', Session.get('currentIndustryOnet'));
});
I am setting the jobs like so for Handlebars:
Template.list.jobs = function() {
var jobs;
Deps.autorun(function(){
if(Session.get('currentIndustryOnet')) {
console.log(Jobs.find({onet: Session.get('currentIndustryOnet')}).count());
}
});
if(Session.get('currentIndustryOnet')) {
jobs = Jobs.find({onet: Session.get('currentIndustryOnet')});
} else {
jobs = Jobs.find({})
}
Session.set('jobCount', jobs.count());
return Pagination.collection(jobs.fetch());
}
Also, the Session.get('currentIndustryOnet') is set by a click event from a dropdown list
I am confused on why you should have to set the jobs in the template if it auto-updates in the Deps.autorun?
Meteor.subscribe('jobs', ...) asks the server to publish a set of jobs documents to the client-side database (minimongo) for use in your templates. This blog post gives a good explanation of how this works. It should be noted that the string 'jobs' does not need to have anything to do with the collection - it's simply an identifier which should match a publish function on the server side. For example, it could have been called, 'jobsForCurrentIndustryOnet' (that's probably a better name).
After subscribing for the needed subset of the database, the client will reactively redraw the jobs in your templates based on session variables and the result of Jobs.find calls.
Note that your template helpers should be primarily concerned with retrieving and manipulating data. They should not include calls to subscriptions and autoruns. In your case, you should remove the Deps.autorun inside of the jobs helper because a new autorun gets created every time the jobs helper gets evaluated (could be often depending on how your reactive variables are changing).
Say I have two view models that each have an observable property that represents different, but similar data.
function site1Model(username) {
this.username = ko.observable(username);
....
}
function site2Model(username) = {
this.username = ko.observable(username);
....
}
These view models are independent and not necessarily linked to each other, but in some cases, a third view model creates a link between them.
function site3Model(username) = {
this.site1 = new site1Model(username);
this.site2 = new site2Model(username);
// we now need to ensure that the usernames are kept the same between site1/2
...
}
Here are some options that I've come up with.
Use a computed observable that reads one and writes to both:
site3Model.username = ko.computed({
read: function() {
return this.site1.username(); // assume they are always the same
},
write: function(value) {
this.site1.username(value);
this.site2.username(value);
},
owner: site3Model
}
This will keep the values in sync as long as changes always come through the computed. But if an underlying observable is changed directly, it won't do so.
Use the subscribe method to update each from the other:
site3Model.site1.username.subscribe(function(value) {
this.site2.username(value);
}, site3Model);
site3Model.site2.username.subscribe(function(value) {
this.site1.username(value);
}, site3Model);
This works as long as the observables suppress notifications when the values are the same; otherwise you'd end up with an infinite loop. You could also do the check earlier: if (this.site1.username() !== value) this.site1.username(value); This also has a problem that the observables have to be simple (it won't work right if site1 and site2 themselves are observables).
Use computed to do the subscribe and updates:
site3Model.username1Updater = ko.computed(function() {
this.site1.username(this.site2.username());
}, site3Model);
site3Model.username2Updater = ko.computed(function() {
this.site2.username(this.site1.username());
}, site3Model);
This format allows us to have other dependencies. For example, we could make site1 and site2 observables and then use this.site1().username(this.site2().username()); This method also requires a check for equality to avoid an infinite loop. If we can't depend on the observable to do it, we could check within the computed, but would add another dependency on the observable we're updating (until something like observable.peek is available).
This method also has the downside of running the update code once initially to set up the dependencies (since that's how computed works).
Since I feel that all of these methods have a downside, is there another way to do this that would be simple (less than 10 lines of code), efficient (not run unnecessary code or updates), and flexible (handle multiple levels of observables)?
It is not exactly 10 lines of code (although you could strip it down to your liking), but I use pub/sub messages between view models for this situation.
Here is a small library that I wrote for it: https://github.com/rniemeyer/knockout-postbox
The basic idea is just to create a ko.subscribable and use topic-based subscriptions. The library extends subscribables to add subscribeTo, publishOn and syncWith (both publish and subscribe on a topic). These methods will set up the proper subscriptions for an observable to automatically participate in this messaging and stay synchronized with the topic.
Now your view models do not need to have direct references to each other and can communicate through the pubsub system. You can refactor your view models without breaking anything.
Like I said you could strip it down to less than 10 lines of code. The library just adds some extras like being able to unsubscribe, being able to have control over when publishing actually happens (equalityComparer), and you can specify a transform to run on incoming values.
Feel free to post any feedback.
Here is a basic sample: http://jsfiddle.net/rniemeyer/mg3hj/
Ryan and John, Thank you both for your answers. Unfortunately, I really don't want to introduce a global naming system that the pub/sub systems require.
Ryan, I agree that the subscribe method is probably the best. I've put together a set of functions to handle the subscription. I'm not using an extension because I also want to handle the case where the observables themselves might be dynamic. These functions accept either observables or functions that return observables. If the source observable is dynamic, I wrap the accessor function call in a computed observable to have a fixed observable to subscribe to.
function subscribeObservables(source, target, dontSetInitially) {
var sourceObservable = ko.isObservable(source)
? source
: ko.computed(function(){ return source()(); }),
isTargetObservable = ko.isObservable(target),
callback = function(value) {
var targetObservable = isTargetObservable ? target : target();
if (targetObservable() !== value)
targetObservable(value);
};
if (!dontSetInitially)
callback(sourceObservable());
return sourceObservable.subscribe(callback);
}
function syncObservables(primary, secondary) {
subscribeObservables(primary, secondary);
subscribeObservables(secondary, primary, true);
}
This is about 20 lines, so maybe my target of less than 10 lines was a bit unreasonable. :-)
I modified Ryan's postbox example to demonstrate the above functions: http://jsfiddle.net/mbest/vcLFt/
Another option is to create an isolated datacontext that maintains the models of observables. the viewmodels all look to the datacontext for their data and refer to the same objects, so when one updates, they all do. The VM's dependency is on the datacontext, but not on other VMs. I've been doing this lately and it has worked well. Although, it is much more complex than using pub/sub.
If you want simple pub/sub, you can use Ryan Niemyer's library that he mentioned or use amplify.js which has pub/sub messaging (basically a messenger or event aggregator) built in. Both are lightweight and decoupled.
In case anyone needed.
Another option is to create a reference object/observable.
This also handle object that contains multiple observable.
(function(){
var subscriptions = [];
ko.helper = {
syncObject: function (topic, obj) {
if(subscriptions[topic]){
return subscriptions[topic];
} else {
return subscriptions[topic] = obj;
}
}
};
})();
In your view models.
function site1Model(username) {
this.username = syncObject('username', ko.observable());
this.username(username);
....
}
function site2Model(username) = {
this.username = syncObject('username', ko.observable());
this.username(username);
....
}