Is there a way to store subscriptions of the same server collection in a different minimongo collection?
If not is there any best practice to work around?
I do have a summary table having 50k datasets with a lot of details in the documents.
// Server
var collection = new Meteor.Collection("collection");
Meteor.publish("detail", function (id) {
return collection.find({_id: id});
});
// A pager that does not include the data (fields:{data:0})
Meteor.publish("master", function (filter, sort, skip, limit) {
return collection.find({name: new RegExp("^" + filter + "|\\s" + filter, "i")},
{limit: limit,
skip: skip,
sort: options,
fields: {data: 0}
});
});
// Client
var collection = new Meteor.Collection("collection");
Deps.autorun(function () {
Meteor.subscribe("master",
Session.get("search"),
Session.get("sort"),
Session.get("skip"),
Session.get("limit")
);
Meteor.subscribe("detail", Session.get("selection"));
});
Problem above: both subscriptions are feed into the same collection.
This does not work well if the results of the finds are stored in the same local collection.
Having a local collection with the name of the subscription/publish would be great.
// Client
var detail = new Meteor.Collection("detail"),
master = new Meteor.Collection("master");
Any Ideas how to encourage subscriptions to use my own collections??
I found the solution through help of Andrews Hint the Discover Meteor book that shows a lot of publishing subscription scenarios.
Anyway: After reading I discovered that the question I was heading for is also answered in the Meteor documentation Meteor.publish
The last example basically creates a virtual collection "counts" for the "messages" collection. Well done already ;-)
"Wer lesen kann ist im Vorteil!"
Related
I have a custom publication on my server (which in some way join 2 collections).
This resulting set of this publication is exactly what I need but for performances issues I would like to avoid sending it entirely to the client.
If I did not care about performances, I would only subscribe to the
publication and do something like
theCollection.find({"my":"filter"})
I am therefore trying to find a way to publish a subset of the custom publication so that the filter would be applied on the custom publication on the server side.
Is there a way to chain or filter publications (server side) ?
For the question we can assume the custom publication to look like this and cannot be modified:
Meteor.publish('customPublication', function() {
var sub = this;
var aCursor = Resources.find({type: 'someFilter'});
Mongo.Collection._publishCursor(aCursor, sub, 'customPublication');
sub.ready();
});
if i understand the question right, you are looking for https://atmospherejs.com/reywood/publish-composite
It let's you "publish a set of related documents from various collections using a reactive join. This makes it easy to publish a whole tree of documents at once. The published collections are reactive and will update when additions/changes/deletions are made."
Ok I came to the following workaround. Instead of working on the publication, I simply added a new collection I update according to the other collections. In order to do so I am using the meteor hooks package
function transformDocument(doc)
{
doc.aField = "aValue"; // do what you want here
return doc;
}
ACollection.after.insert(function(userId, doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
ACollection.after.update(function(userId, doc, fieldNames, modifier, options)
{
var transformedDocument = transformDocument(doc);
delete transformedDocument._id;
AnotherCollection.update(doc._id,{$set:transformedDocument});
});
ACollection.after.remove(function(userId, doc)
{
AnotherCollection.remove(doc._id);
});
Then I have the new collection I can publish subsets the regular way
Benefits:
You can filter whatever you want into this db, no need to worry if the field is virtual or real
Only one operation every time a db changes. This avoid having several publication merging the same data
Cave eats:
This requires one more Collection = more space
The 2 db might not be always synchronised, there is few reasons for this:
The client manually changed the data of "AnotherCollection"
You had documents in "ACollection" before you added "AnotherCollection".
The transform function or source collection schema changed at some point
To fix this:
AnotherCollection.allow({
insert: function () {
return Meteor.isServer;
},
update: function () {
return Meteor.isServer;
},
remove: function () {
return Meteor.isServer;
}
});
And to synchronise at meteor startup (i.e. build the collection from scratch). Do this only once for maintenance or after adding this new collection.
Meteor.startup(function()
{
AnotherCollection.remove({});
var documents = ACollection.find({}).fetch();
_.each(documents, function(doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
});
I'm working on a simple app where a User can follow other users. Users can star posts. And a user's feed is composed of posts that have been starred by users they follow. Pretty simple actually. However, this all gets complicated in Mongo and Meteor...
There are basically two way of modeling this that I can think of:
A user has a property, following, which is an array of userIds that the user follows. Also, a post has a property, starrers, which is an array of userIds that have starred this post. The good thing about this method is that publications are relatively simple:
Meteor.publish 'feed', (limit) ->
Posts.find({starrers: {$in: Meteor.users.findOne(#userId).following}}, {sort: {date: -1}, limit:limit})
We aren't reactively listening to who the user is following, but thats not too bad for now. The main problem with this approach is that (1) the individual documents will become large and inefficient if 1000000 people star a post. Another problem is that (2) it would be pain to keep track of information like when a user started following another user or when a user starred a post.
The other way of doing this is having two more collections, Stars and Follows. If a user stars a post, then we create a document with properties userId and postId. If a user follows another user, then we create a document with properties userId and followId. This gives us the advantage of smaller document sizes for Users and Posts, but complicated things when it comes to querying, especially because Mongo doesn't handle joins!
Now, I did some research and people seem to agree that the second choice is the right way to go. Now the problem I'm having is efficiently querying and publishing. Based on the Discover Meteor chapter about Advanced Publications, I created a publication that publishes the posts that are starred by user's followers -- sorted, and limited.
# a helper to handle stopping observeChanges
observer = (sub, func) ->
handle = null
sub.onStop ->
handle?.stop?()
() ->
handle?.stop?()
handle = func()
Meteor.publish 'feed', (limit) ->
sub = this
userId = #userId
followIds = null
eventIds = null
publishFollows = observer sub, () ->
followIds = {}
Follows.find({userId:userId}).observeChanges
added: (id, doc) ->
followIds[id] = doc.followId
sub.added('follows', id, doc)
publishStars()
removed: (id) ->
delete followIds[id]
sub.removed('follows', id)
publishStars()
publishStars = observer sub, () ->
eventIds = {}
Stars.find({userId: {$in: _.keys(followIds)}).observeChanges
added: (id, doc) ->
eventIds[id] = null
sub.added('stars', id, doc)
publishEvents()
removed: (id) ->
delete eventIds[id]
sub.removed('stars', id)
publishEvents()
publishEvents = observer sub, () ->
Events.find({_id: {$in: _.keys(eventIds)}}, {sort: {name:1, date:-1}, limit:limit}).observeChanges
added: (id, doc) ->
sub.added('events', id, doc)
changed: (id, fields) ->
sub.changed('events', id, fields)
removed: (id) ->
sub.removed('events', id)
While this works, it seems very limited at scale. Particularly, we have to compile a list of every starred post by every follower. The size of this list will grow very quickly. Then we do a huge $in query against all posts.
Another annoyance is querying for the feed on the client after we subscribe:
Meteor.subscribe("feed", 20)
posts = null
Tracker.autorun ->
followers = _.pluck(Follows.find({userId: Meteor.userId()}).fetch(), "followId")
starredPostIds = _.pluck(Stars.find({userId: {$in: followers}}).fetch(), "postId")
posts = Posts.find({_id: {$in: starredPostIds}}, {sort: {date: -1}, limit: 20}).fetch()
Its like we're doing all this work twice. First we do all the work on the server to publish the feed. Then we need to go through the exact same logic again on the client to get those posts...
My question here is a matter of design over everything. How can I efficiently design this feed based on followers staring posts? What collection / collection schemas should I use? How should I create the appropriate publication? How can I query for the feed on the client?
So it turns out that Mongo and "non-relational" databases simply aren't designed for relational data. Thus, there is no solution here with Mongo. I've ended up using Neo4j, but SQL would work fine as well.
meteor add reywood:publish-composite
Meteor.publishComposite('tweets', function(username) {
return {
find: function() {
// Find the current user's following users
return Relationships.find({ follower: username });
},
children: [{
find: function(relationship) {
// Find tweets from followed users
return Tweets.find({user: relationship.following});
}
}]
}
});
Meteor.publish('ownTweets', function(username) {
return Tweets.find({user: username});
});
Meteor Collections have a transform ability that allows behavior to be attached to the objects returned from mongo.
We want to have autopublish turned off so the client does not have access to the database collections, but we still want the transform functionality.
We are sending data to the client with a more explicit Meteor.publish/Meteor.subscribe or the RPC mechanism ( Meteor.call()/Meteor.methods() )
How can we have the Meteor client automatically apply a transform like it will when retrieving data directly with the Meteor.Collection methods?
While you can't directly use transforms, there is a way to transform the result of a database query before publishing it. This is what the "publish the current size of a collection" example describes here.
It took me a while to figure out a really simple application of that, so maybe my code will help you, too:
Meteor.publish("publicationsWithHTML", function (data) {
var self = this;
Publications
.find()
.forEach(function(entry) {
addSomeHTML(entry); // this function changes the content of entry
self.added("publications", entry._id, entry);
});
self.ready();
});
On the client you subscribe to this:
Meteor.subscribe("publicationsWithHTML");
But your model still need to create a collection (on both sides) that is called 'publications':
Publications = new Meteor.Collection('publications');
Mind you, this is not a very good example, as it doesn't maintain the reactivity. But I found the count example a bit confusing at first, so maybe you'll find it helpful.
(Meteor 0.7.0.1) - meteor does allow behavior to be attached to the objects returned via the pub/sub.
This is from a pull request I submitted to the meteor project.
Todos = new Meteor.Collection('todos', {
// transform allows behavior to be attached to the objects returned via the pub/sub communication.
transform : function(todo) {
todo.update = function(change) {
Meteor.call('Todos_update', this._id, change);
},
todo.remove = function() {
Meteor.call('Todos_remove', this._id);
}
return todo;
}
});
todosHandle = Meteor.subscribe('todos');
Any objects returned via the 'todos' topic will have the update() and the remove() function - which is exactly what I want: I now attach behavior to the returned data.
Try:
let transformTodo = (fields) => {
fields._pubType = 'todos';
return fields;
};
Meteor.publish('todos', function() {
let subHandle = Todos
.find()
.observeChanges({
added: (id, fields) => {
fields = transformTodo(fields);
this.added('todos', id, fields);
},
changed: (id, fields) => {
fields = transformTodo(fields);
this.changed('todos', id, fields);
},
removed: (id) => {
this.removed('todos', id);
}
});
this.ready();
this.onStop(() => {
subHandle.stop();
});
});
Currently, you can't apply transforms on the server to published collections. See this question for more details. That leaves you with either transforming the data on the client, or using a meteor method. In a method, you can have the server do whatever you want to the data.
In one of my projects, we perform our most expensive query (it joins several collections, denormalizes the documents, and trims unnecessary fields) via a method call. It isn't reactive, but it greatly simplifies our code because all of the transformation happens on the server.
To extend #Christian Fritz answer, with Reactive Solution using peerlibrary:reactive-publish
Meteor.publish("todos", function() {
const self = this;
return this.autorun(function(computation) {
// Loop over each document in collection
todo.find().forEach(function(entry) {
// Add function to transform / modify each document here
self.added("todos", entry._id, entry);
});
});
});
Is there a way to store subscriptions of the same server collection in a different minimongo collection?
If not is there any best practice to work around?
I do have a summary table having 50k datasets with a lot of details in the documents.
// Server
var collection = new Meteor.Collection("collection");
Meteor.publish("detail", function (id) {
return collection.find({_id: id});
});
// A pager that does not include the data (fields:{data:0})
Meteor.publish("master", function (filter, sort, skip, limit) {
return collection.find({name: new RegExp("^" + filter + "|\\s" + filter, "i")}, {limit: limit, skip: skip, sort: options, fields: {data: 0}});
});
// Client
var collection = new Meteor.Collection("collection");
Deps.autorun(function () {
Meteor.subscribe("master",
Session.get("search"),
Session.get("sort"),
Session.get("skip"),
Session.get("limit")
);
Meteor.subscribe("detail", Session.get("selection"));
});
Problem above: both subscriptions are feed into the same collection.
This does not work well if the results of the finds are stored in the same local collection.
Having a local collection with the name of the subscription/publish would be great.
// Client
var detail = new Meteor.Collection("detail"),
master = new Meteor.Collection("master");
Any Ideas?
If you want your client side collection to have a different name from the server side collection you can not just return a collection cursor. This can be done in the publish function though like this:
Meteor.publish("details", function (id) { //details here matches the subscribe request
var self = this;
self.added( "details", id, collection.findOne({_id: id}); //details here tells the client which collection holds the data
self.ready();
});
This will not be reactive but can be made that way by using observe as in the counts by room example at http://docs.meteor.com which is explained in detail here How does the messages-count example in Meteor docs work?.
While this answers your question of how to get a specific name for a collection without having that collection on the server. I think you probably get what you want more easily with a publish function more like this:
Meteor.publish("master", function (filter, sort, skip, limit, id) {
return [
collection.find({name: new RegExp("^" + filter + "|\\s" + filter, "i")}, {limit: limit, skip: skip, sort: options, fields: {data: 0}})
, collection.find( id , {fields: {data: 1}} )
];
});
Then subscribe on client:
Deps.autorun(function () {
Meteor.subscribe("master",
Session.get("search"),
Session.get("sort"),
Session.get("skip"),
Session.get("limit"),
Session.get("selection")
);
});
Then even though all your data is in one collection you can have a reactive cursor to your selected id with the data included. Query from the client like this:
collection.find( Session.get("selection") );
I need to inform clients about changes on server side. In my case I am using different Collections on server and on client (more about it in this question: how would you build pinterest like page with meteor.js).
On the server I am getting new Products from external API. I would like to publish the number of new items to all clients that they could update their local variables needed for layout to work well.
How to do it?
It would be nice if I could publish/subscribe other kinds of data than Meteor.Collection. I found Meteor.deps, but what I understand it works only on client side.
To accomplish what you want you do need another collection - on the client. On the server, in a publish function, build a document from scratch assigning the current count of Products to an attribute. Using observe() and set, modify count when documents are added or removed from Products. Subscribe to the count "record set" on the client.
// Server
Meteor.publish('count', function () {
// Build a document from scratch
var self = this;
var uuid = Meteor.uuid();
var count = Products.find().count();
// Assign initial Products count to document attribute
self.set('count', uuid, {count: count});
// Observe Products for additions and removals
var handle = Products.find().observe({
added: function (doc, idx) {
count++;
self.set('counts', uuid, {count: count});
self.flush();
},
removed: function (doc, idx) {
count--;
self.set('counts', uuid, {count: count});
self.flush();
}
});
self.complete();
self.flush();
self.onStop(function () {
handle.stop();
});
});
// Client
Counts = new Meteor.Collection('count');
Meteor.subscribe('count');
console.log('Count: ' + Counts.findOne().count);
I must say the above solution showed me one way, but still, what if I need to publish to client data that are not connected with observe()? Or with any collection?
In my case I have i.e. 1000 products. To engage visitors I am "refreshig" the collection by updating the timestamp of random number of products, and displaying collection sorted by timestamp. Thank to this visitors have impression that something is happening.
My refresh method returns number of products (it is random). I need to pass that number to all clients. I did it, but using (I think) ugly workaround.
My refresh method sets Session.set('lastRandomNo', random). BTW: I didn't know that Session works on server side. refresh updates Products collection.
Then accoriding to above answer:
Meteor.publish 'refreshedProducts', ->
self = this
uuid = Meteor.uuid()
# create a new collection to pass ProductsMeta data
self.set('products_meta', uuid, { refreshedNo: 0 })
handle = Products.find().observe
changed: (newDocument, atIndex, oldDocument) ->
self.set('products_meta', uuid, { refreshedNo: Session.get('lastRandomNo') })
self.flush()
self.complete()
self.flush()
self.onStop ->
handle.stop()
and on client side:
ProductsMeta = new Meteor.Collection('products_meta')
# subscribe to server 'products_meta' collection that is generated by server
Meteor.subscribe('refreshedProducts')
ProductsMeta.find({}).observe
changed: (newDocument, atIndex, oldDocument) ->
# I have access to refreshedNo by
console.log ProductsMeta.findOne().refreshedNo
What do you think?