I'm trying to work with associating documents in different collections (not embedded documents) and while there is an issue for that in Mongooose, I'm trying to work around it now by lazy loading the associated document with a virtual property as documented on the Mongoose website.
The problem is that the getter for a virtual takes a function as an argument and uses the return value for the virtual property. This is great when the virtual doesn't require any async calls to calculate it's value, but doesn't work when I need to make an async call to load the other document. Here's the sample code I'm working with:
TransactionSchema.virtual('notebook')
.get( function() { // <-- the return value of this function is used as the property value
Notebook.findById(this.notebookId, function(err, notebook) {
return notebook; // I can't use this value, since the outer function returns before we get to this code
})
// undefined is returned here as the properties value
});
This doesn't work since the function returns before the async call is finished. Is there a way I could use a flow control library to make this work, or could I modify the first function so that I pass the findById call to the getter instead of an anonymous function?
You can define a virtual method, for which you can define a callback.
Using your example:
TransactionSchema.method('getNotebook', function(cb) {
Notebook.findById(this.notebookId, function(err, notebook) {
cb(notebook);
})
});
And while the sole commenter appears to be one of those pedantic types, you also should not be afraid of embedding documents. Its one of mongos strong points from what I understand.
One uses the above code like so:
instance.getNotebook(function(nootebook){
// hey man, I have my notebook and stuff
});
While this addresses the broader problem rather than the specific question, I still thought it was worth submitting:
You can easily load an associated document from another collection (having a nearly identical result as defining a virtual) by using Mongoose's query populate function. Using the above example, this requires specifying the ref of the ObjectID in the Transaction schema (to point to the Notebook collection), then calling populate(NotebookId) while constructing the query. The linked Mongoose documentation addresses this pretty thoroughly.
I'm not familiar with Mongoose's history, but I'm guessing populate did not exist when these earlier answers were submitted.
Josh's approach works great for single document look-ups, but my situation was a little more complex. I needed to do a look-up on a nested property for an entire array of objects. For example, my model looked more like this:
var TransactionSchema = new Schema({
...
, notebooks: {type: [Notebook]}
});
var NotebookSchema = new Schema({
...
, authorName: String // this should not necessarily persist to db because it may get stale
, authorId: String
});
var AuthorSchema = new Schema({
firstName: String
, lastName: String
});
Then, in my application code (I'm using Express), when I get a Transaction, I want all of the notebooks with author last name's:
...
TransactionSchema.findById(someTransactionId, function(err, trans) {
...
if (trans) {
var authorIds = trans.notebooks.map(function(tx) {
return notebook.authorId;
});
Author.find({_id: {$in: authorIds}, [], function(err2, authors) {
for (var a in authors) {
for (var n in trans.notebooks {
if (authors[a].id == trans.notebooks[n].authorId) {
trans.notebooks[n].authorLastName = authors[a].lastName;
break;
}
}
}
...
});
This seems wildly inefficient and hacky, but I could not figure out another way to accomplish this. Lastly, I am new to node.js, mongoose, and stackoverflow so forgive me if this is not the most appropriate place to extend this discussion. It's just that Josh's solution was the most helpful in my eventual "solution."
As this is an old question, I figured it might use an update.
To achieve asynchronous virtual fields, you can use mongoose-fill, as stated in mongoose's github issue: https://github.com/Automattic/mongoose/issues/1894
Related
I have a custom publication on my server (which in some way join 2 collections).
This resulting set of this publication is exactly what I need but for performances issues I would like to avoid sending it entirely to the client.
If I did not care about performances, I would only subscribe to the
publication and do something like
theCollection.find({"my":"filter"})
I am therefore trying to find a way to publish a subset of the custom publication so that the filter would be applied on the custom publication on the server side.
Is there a way to chain or filter publications (server side) ?
For the question we can assume the custom publication to look like this and cannot be modified:
Meteor.publish('customPublication', function() {
var sub = this;
var aCursor = Resources.find({type: 'someFilter'});
Mongo.Collection._publishCursor(aCursor, sub, 'customPublication');
sub.ready();
});
if i understand the question right, you are looking for https://atmospherejs.com/reywood/publish-composite
It let's you "publish a set of related documents from various collections using a reactive join. This makes it easy to publish a whole tree of documents at once. The published collections are reactive and will update when additions/changes/deletions are made."
Ok I came to the following workaround. Instead of working on the publication, I simply added a new collection I update according to the other collections. In order to do so I am using the meteor hooks package
function transformDocument(doc)
{
doc.aField = "aValue"; // do what you want here
return doc;
}
ACollection.after.insert(function(userId, doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
ACollection.after.update(function(userId, doc, fieldNames, modifier, options)
{
var transformedDocument = transformDocument(doc);
delete transformedDocument._id;
AnotherCollection.update(doc._id,{$set:transformedDocument});
});
ACollection.after.remove(function(userId, doc)
{
AnotherCollection.remove(doc._id);
});
Then I have the new collection I can publish subsets the regular way
Benefits:
You can filter whatever you want into this db, no need to worry if the field is virtual or real
Only one operation every time a db changes. This avoid having several publication merging the same data
Cave eats:
This requires one more Collection = more space
The 2 db might not be always synchronised, there is few reasons for this:
The client manually changed the data of "AnotherCollection"
You had documents in "ACollection" before you added "AnotherCollection".
The transform function or source collection schema changed at some point
To fix this:
AnotherCollection.allow({
insert: function () {
return Meteor.isServer;
},
update: function () {
return Meteor.isServer;
},
remove: function () {
return Meteor.isServer;
}
});
And to synchronise at meteor startup (i.e. build the collection from scratch). Do this only once for maintenance or after adding this new collection.
Meteor.startup(function()
{
AnotherCollection.remove({});
var documents = ACollection.find({}).fetch();
_.each(documents, function(doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
});
I'd like to create a method that returns the count of a generic collection.
Calling the method would look something like this:
Meteor.call('getCollectionCount', 'COLLECTION_NAME');
And the result would be the collection count.
The server method code would look something like this:
getCollectionCount: function (collectionName) {
return window[collectionName].find().count();
}
This won't work because window isn't defined on the server, but is something similar possible?
Use global instead of window.
Note that this uses the variable name assigned to the collection object, not the name given to the collection. For this to work with Meteor.users you need to assign another variable name.
if (Meteor.isServer) {
users = Meteor.users;
}
if (Meteor.isClient) {
Meteor.call('count', 'users', function (err, res) {
// do something with number of users
});
}
Also probably a good idea to check that global[collectionName] is actually a collection.
I came up with this code which makes the following assumptions :
collections are declared in the global scope as top level objects.
collections are searched by collection name, not the collection variable identifier.
So client code should declare their collections like this :
MyCollection=new Meteor.Collection("my-collection");
And use the function like this :
var clientResult=Meteor.call("getCollectionCount","my-collection",function(error,result){
if(error){
console.log(error);
return;
}
console.log("actual server-side count is : ",result);
});
console.log("published subset count is : ",clientResult);
The method supports execution on the client (this is known as method stub or method simulation) but will only yield the count of the collection subset replicated client-side, to get the real count wait for server-side response using a callback.
/packages/my-package/lib/my-package.js
getCollection=function(collectionName){
if(collectionName=="users"){
return Meteor.users;
}
var globalScope=Meteor.isClient?window:global;
for(var property in globalScope){
var object=globalScope[property];
if(object instanceof Meteor.Collection && object._name==collectionName){
return object;
}
}
throw Meteor.Error(500,"No collection named "+collectionName);
};
Meteor.methods({
getCollectionCount:function(collectionName){
return getCollection(collectionName).find().count();
}
});
As Meteor.users is not declared as a top level variable you have to account for the special case (yes, this is ugly).
Digging into Meteor's collection handling code could provide a better alternative (getting access to a collection handle by collection name).
Final words on this : using a method call to count a collection documents is unfortunately non-reactive, so given the Meteor paradigm this might be of little use.
Most of the time you will want to fetch the number of documents in a collection for pagination purpose (something like a "Load more" button in a posts list for example), and as the rest of the Meteor architecture you'll want this to be reactive.
To count documents in a collection reactively you'll have to setup a slightly more complicated publication as showcased in the "counts-by-room" example in the docs.
http://docs.meteor.com/#meteor_publish
This is something you definitely want to read and understand.
This smart package is actually doing it right :
http://atmospherejs.com/package/publish-counts
It provides a helper function that is publishing the counts of any cursor.
Keep track of the collections on some other property that the server has access too. You could even call it window if you really wanted to.
var wow = new Meteor.Collection("wow");
collections["wow"] = wow;
getCollectionCount: function (collectionName) {
return collections[collectionName].find().count();
}
If you don't want the package users to change how they work with collections in the app then I think you should use MongoInternals to get collections by name from the db. Not tested but here is an example:
//on server
Meteor.methods({
count: function( name ){
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
var collection = db.collection( name );
return collection && collection.count({});
}
});
Another example of MongoInternals use is here. Documentation of the count() function available from the mongo driver is here.
Meteor Collections have a transform ability that allows behavior to be attached to the objects returned from mongo.
We want to have autopublish turned off so the client does not have access to the database collections, but we still want the transform functionality.
We are sending data to the client with a more explicit Meteor.publish/Meteor.subscribe or the RPC mechanism ( Meteor.call()/Meteor.methods() )
How can we have the Meteor client automatically apply a transform like it will when retrieving data directly with the Meteor.Collection methods?
While you can't directly use transforms, there is a way to transform the result of a database query before publishing it. This is what the "publish the current size of a collection" example describes here.
It took me a while to figure out a really simple application of that, so maybe my code will help you, too:
Meteor.publish("publicationsWithHTML", function (data) {
var self = this;
Publications
.find()
.forEach(function(entry) {
addSomeHTML(entry); // this function changes the content of entry
self.added("publications", entry._id, entry);
});
self.ready();
});
On the client you subscribe to this:
Meteor.subscribe("publicationsWithHTML");
But your model still need to create a collection (on both sides) that is called 'publications':
Publications = new Meteor.Collection('publications');
Mind you, this is not a very good example, as it doesn't maintain the reactivity. But I found the count example a bit confusing at first, so maybe you'll find it helpful.
(Meteor 0.7.0.1) - meteor does allow behavior to be attached to the objects returned via the pub/sub.
This is from a pull request I submitted to the meteor project.
Todos = new Meteor.Collection('todos', {
// transform allows behavior to be attached to the objects returned via the pub/sub communication.
transform : function(todo) {
todo.update = function(change) {
Meteor.call('Todos_update', this._id, change);
},
todo.remove = function() {
Meteor.call('Todos_remove', this._id);
}
return todo;
}
});
todosHandle = Meteor.subscribe('todos');
Any objects returned via the 'todos' topic will have the update() and the remove() function - which is exactly what I want: I now attach behavior to the returned data.
Try:
let transformTodo = (fields) => {
fields._pubType = 'todos';
return fields;
};
Meteor.publish('todos', function() {
let subHandle = Todos
.find()
.observeChanges({
added: (id, fields) => {
fields = transformTodo(fields);
this.added('todos', id, fields);
},
changed: (id, fields) => {
fields = transformTodo(fields);
this.changed('todos', id, fields);
},
removed: (id) => {
this.removed('todos', id);
}
});
this.ready();
this.onStop(() => {
subHandle.stop();
});
});
Currently, you can't apply transforms on the server to published collections. See this question for more details. That leaves you with either transforming the data on the client, or using a meteor method. In a method, you can have the server do whatever you want to the data.
In one of my projects, we perform our most expensive query (it joins several collections, denormalizes the documents, and trims unnecessary fields) via a method call. It isn't reactive, but it greatly simplifies our code because all of the transformation happens on the server.
To extend #Christian Fritz answer, with Reactive Solution using peerlibrary:reactive-publish
Meteor.publish("todos", function() {
const self = this;
return this.autorun(function(computation) {
// Loop over each document in collection
todo.find().forEach(function(entry) {
// Add function to transform / modify each document here
self.added("todos", entry._id, entry);
});
});
});
Suppose I write:
new Meteor.Collection("foos");
new Meteor.Collection("bars");
Is there an API for accessing those collections by name? Something like Meteor.Collection.get(name), where name is "foos" or "bars"? I know I could write something like
var MyCollections = {
foos: new Meteor.Collection("foos");
bars: new Meteor.Collection("bars");
}
and then use MyCollections[name], but I'd prefer to use an existing API if one exists.
Based on Shane Donelley's mongoinspector
https://github.com/shanedonnelly1/mongoinspector
getCollection = function (string) {
for (var globalObject in window) {
if (window[globalObject] instanceof Meteor.Collection) {
if (globalObject === string) {
return (window[globalObject]);
break;
};
}
}
return undefined; // if none of the collections match
};
I've just found that package : https://github.com/dburles/mongo-collection-instances/
It allow you to
Foo1 = new Mongo.Collection('foo'); // local
Foo2 = new Mongo.Collection('foo', { connection: connection });
Mongo.Collection.get('foo') // returns instance of Foo1
Mongo.Collection.get('foo', { connection: connection });
// returns instance of Foo2
Hope it will help
This feature was added to Meteor in Feb 2016: "Provide a way to access collections from stores on the client"
It works like this:
Meteor.connection._stores['tasks']._getCollection();
And I was using it as follows to test inserts using the javascript console:
Meteor.connection._stores['tasks']._getCollection().insert({text:'test'});
For the insert it required the insecure package to still be installed otherwise got an access denied message.
As far as I can see in the collection.js source there currently is no way in the api to get an existing Collection by name, once it has already been initialized on the server. It probably wouldn't be hard to add that feature.
So, why not fork Meteor and submit a patch or create a smart package and share it I'm sure there are others out there who'd like the same feature.
With https://github.com/dburles/mongo-collection-instances you can use Mongo.Collection.get('collectionname')
Note that the parameter you're inserting is the same one you use when creating the collection. So if you're using const Products = new Mongo.Collection('products') then you should use get('products') (lowercase).
Note that they have a return value, so you can just do
var Patterns = new Meteor.Collection("patterns");
and use Patterns everywhere.
And when you need to subscribe to server updates, provide "patterns" to Meteor.subscribe().
If you have the same code for multiple collections, the chance is high that you're doing something wrong from a software engineering viewpoint; why not use a single collection with a type field (or something else that differentiates the documents) and use that instead of using multiple collections?
Rather than looking, I've just been doing:
Foos = new Meteor.Collection("foos");
or possibly put it inside another object. I haven't really been making a Collections collection object.
It seems there is no way to get at the wrapped Meteor.Collection object without saving it at creation time, as others have mentioned.
But there is at least a way to list all created collections, and actually access the corresponding Mongo LocalCollection object. They are available from any Meteor Collection object, so to keep it generalistic you can create a dummy collection just for this. Use a method as such (CoffeeScript):
dummy = new Meteor.Collection 'dummy'
getCollection = (name) ->
dummy._driver.collections[name]
These objects do have all the find, findOne, update et al methods, and even some that Meteor doesn't seem to expose, like pauseObservers and resumeObservers which seem interesting. But I haven't tried fiddling with this mongo LocalCollection reference directly to knowif it will update the server collection accordingly.
var bars = new Meteor.Collection("foos");
Judging by what the collection.js does, the line we use to instantiate the collection object opens a connection to the database and looks for the collection matching the name we give. So in this case a connection is made and the collection 'foos' is bound to the Meteor.Collection object 'bars'. See collection.js AND remote_collection_driver.js within the mongo-livedata package.
As is the way with MongoDB, whilst you can, you don't have to explicitly create collections. As stated in the MongoDB documentation:
A collection is created when the first document is inserted.
So, I think what you're after is what you already have - unless I've totally misunderstood what you're intentions are.
You can always roll your own automatic collection getter.
Say you have a couple of collections called "Businesses" and "Clients". Put a reference each into some "collections" object and register a Handlebars helper to access those "collections" by collections["name"].
i.e. put something like this on the client-side main.js:
collections = collections || {};
collections.Businesses = Businesses;
collections.Clients = Clients;
Handlebars.registerHelper("getCollection", function(coll) {
return collections[coll].find();
});
Then in your HTML, just refer to the collection by name:
{{#each getCollection 'Businesses'}}
<div> Business: {{_id}} </div>
{{/each}}
{{#each getCollection 'Clients'}}
<div> Client: {{_id}} </div>
{{/each}}
Look ma, no more generic "list all records" boilerplate js required!
Say I have two view models that each have an observable property that represents different, but similar data.
function site1Model(username) {
this.username = ko.observable(username);
....
}
function site2Model(username) = {
this.username = ko.observable(username);
....
}
These view models are independent and not necessarily linked to each other, but in some cases, a third view model creates a link between them.
function site3Model(username) = {
this.site1 = new site1Model(username);
this.site2 = new site2Model(username);
// we now need to ensure that the usernames are kept the same between site1/2
...
}
Here are some options that I've come up with.
Use a computed observable that reads one and writes to both:
site3Model.username = ko.computed({
read: function() {
return this.site1.username(); // assume they are always the same
},
write: function(value) {
this.site1.username(value);
this.site2.username(value);
},
owner: site3Model
}
This will keep the values in sync as long as changes always come through the computed. But if an underlying observable is changed directly, it won't do so.
Use the subscribe method to update each from the other:
site3Model.site1.username.subscribe(function(value) {
this.site2.username(value);
}, site3Model);
site3Model.site2.username.subscribe(function(value) {
this.site1.username(value);
}, site3Model);
This works as long as the observables suppress notifications when the values are the same; otherwise you'd end up with an infinite loop. You could also do the check earlier: if (this.site1.username() !== value) this.site1.username(value); This also has a problem that the observables have to be simple (it won't work right if site1 and site2 themselves are observables).
Use computed to do the subscribe and updates:
site3Model.username1Updater = ko.computed(function() {
this.site1.username(this.site2.username());
}, site3Model);
site3Model.username2Updater = ko.computed(function() {
this.site2.username(this.site1.username());
}, site3Model);
This format allows us to have other dependencies. For example, we could make site1 and site2 observables and then use this.site1().username(this.site2().username()); This method also requires a check for equality to avoid an infinite loop. If we can't depend on the observable to do it, we could check within the computed, but would add another dependency on the observable we're updating (until something like observable.peek is available).
This method also has the downside of running the update code once initially to set up the dependencies (since that's how computed works).
Since I feel that all of these methods have a downside, is there another way to do this that would be simple (less than 10 lines of code), efficient (not run unnecessary code or updates), and flexible (handle multiple levels of observables)?
It is not exactly 10 lines of code (although you could strip it down to your liking), but I use pub/sub messages between view models for this situation.
Here is a small library that I wrote for it: https://github.com/rniemeyer/knockout-postbox
The basic idea is just to create a ko.subscribable and use topic-based subscriptions. The library extends subscribables to add subscribeTo, publishOn and syncWith (both publish and subscribe on a topic). These methods will set up the proper subscriptions for an observable to automatically participate in this messaging and stay synchronized with the topic.
Now your view models do not need to have direct references to each other and can communicate through the pubsub system. You can refactor your view models without breaking anything.
Like I said you could strip it down to less than 10 lines of code. The library just adds some extras like being able to unsubscribe, being able to have control over when publishing actually happens (equalityComparer), and you can specify a transform to run on incoming values.
Feel free to post any feedback.
Here is a basic sample: http://jsfiddle.net/rniemeyer/mg3hj/
Ryan and John, Thank you both for your answers. Unfortunately, I really don't want to introduce a global naming system that the pub/sub systems require.
Ryan, I agree that the subscribe method is probably the best. I've put together a set of functions to handle the subscription. I'm not using an extension because I also want to handle the case where the observables themselves might be dynamic. These functions accept either observables or functions that return observables. If the source observable is dynamic, I wrap the accessor function call in a computed observable to have a fixed observable to subscribe to.
function subscribeObservables(source, target, dontSetInitially) {
var sourceObservable = ko.isObservable(source)
? source
: ko.computed(function(){ return source()(); }),
isTargetObservable = ko.isObservable(target),
callback = function(value) {
var targetObservable = isTargetObservable ? target : target();
if (targetObservable() !== value)
targetObservable(value);
};
if (!dontSetInitially)
callback(sourceObservable());
return sourceObservable.subscribe(callback);
}
function syncObservables(primary, secondary) {
subscribeObservables(primary, secondary);
subscribeObservables(secondary, primary, true);
}
This is about 20 lines, so maybe my target of less than 10 lines was a bit unreasonable. :-)
I modified Ryan's postbox example to demonstrate the above functions: http://jsfiddle.net/mbest/vcLFt/
Another option is to create an isolated datacontext that maintains the models of observables. the viewmodels all look to the datacontext for their data and refer to the same objects, so when one updates, they all do. The VM's dependency is on the datacontext, but not on other VMs. I've been doing this lately and it has worked well. Although, it is much more complex than using pub/sub.
If you want simple pub/sub, you can use Ryan Niemyer's library that he mentioned or use amplify.js which has pub/sub messaging (basically a messenger or event aggregator) built in. Both are lightweight and decoupled.
In case anyone needed.
Another option is to create a reference object/observable.
This also handle object that contains multiple observable.
(function(){
var subscriptions = [];
ko.helper = {
syncObject: function (topic, obj) {
if(subscriptions[topic]){
return subscriptions[topic];
} else {
return subscriptions[topic] = obj;
}
}
};
})();
In your view models.
function site1Model(username) {
this.username = syncObject('username', ko.observable());
this.username(username);
....
}
function site2Model(username) = {
this.username = syncObject('username', ko.observable());
this.username(username);
....
}