I'm working on a simple app where a User can follow other users. Users can star posts. And a user's feed is composed of posts that have been starred by users they follow. Pretty simple actually. However, this all gets complicated in Mongo and Meteor...
There are basically two way of modeling this that I can think of:
A user has a property, following, which is an array of userIds that the user follows. Also, a post has a property, starrers, which is an array of userIds that have starred this post. The good thing about this method is that publications are relatively simple:
Meteor.publish 'feed', (limit) ->
Posts.find({starrers: {$in: Meteor.users.findOne(#userId).following}}, {sort: {date: -1}, limit:limit})
We aren't reactively listening to who the user is following, but thats not too bad for now. The main problem with this approach is that (1) the individual documents will become large and inefficient if 1000000 people star a post. Another problem is that (2) it would be pain to keep track of information like when a user started following another user or when a user starred a post.
The other way of doing this is having two more collections, Stars and Follows. If a user stars a post, then we create a document with properties userId and postId. If a user follows another user, then we create a document with properties userId and followId. This gives us the advantage of smaller document sizes for Users and Posts, but complicated things when it comes to querying, especially because Mongo doesn't handle joins!
Now, I did some research and people seem to agree that the second choice is the right way to go. Now the problem I'm having is efficiently querying and publishing. Based on the Discover Meteor chapter about Advanced Publications, I created a publication that publishes the posts that are starred by user's followers -- sorted, and limited.
# a helper to handle stopping observeChanges
observer = (sub, func) ->
handle = null
sub.onStop ->
handle?.stop?()
() ->
handle?.stop?()
handle = func()
Meteor.publish 'feed', (limit) ->
sub = this
userId = #userId
followIds = null
eventIds = null
publishFollows = observer sub, () ->
followIds = {}
Follows.find({userId:userId}).observeChanges
added: (id, doc) ->
followIds[id] = doc.followId
sub.added('follows', id, doc)
publishStars()
removed: (id) ->
delete followIds[id]
sub.removed('follows', id)
publishStars()
publishStars = observer sub, () ->
eventIds = {}
Stars.find({userId: {$in: _.keys(followIds)}).observeChanges
added: (id, doc) ->
eventIds[id] = null
sub.added('stars', id, doc)
publishEvents()
removed: (id) ->
delete eventIds[id]
sub.removed('stars', id)
publishEvents()
publishEvents = observer sub, () ->
Events.find({_id: {$in: _.keys(eventIds)}}, {sort: {name:1, date:-1}, limit:limit}).observeChanges
added: (id, doc) ->
sub.added('events', id, doc)
changed: (id, fields) ->
sub.changed('events', id, fields)
removed: (id) ->
sub.removed('events', id)
While this works, it seems very limited at scale. Particularly, we have to compile a list of every starred post by every follower. The size of this list will grow very quickly. Then we do a huge $in query against all posts.
Another annoyance is querying for the feed on the client after we subscribe:
Meteor.subscribe("feed", 20)
posts = null
Tracker.autorun ->
followers = _.pluck(Follows.find({userId: Meteor.userId()}).fetch(), "followId")
starredPostIds = _.pluck(Stars.find({userId: {$in: followers}}).fetch(), "postId")
posts = Posts.find({_id: {$in: starredPostIds}}, {sort: {date: -1}, limit: 20}).fetch()
Its like we're doing all this work twice. First we do all the work on the server to publish the feed. Then we need to go through the exact same logic again on the client to get those posts...
My question here is a matter of design over everything. How can I efficiently design this feed based on followers staring posts? What collection / collection schemas should I use? How should I create the appropriate publication? How can I query for the feed on the client?
So it turns out that Mongo and "non-relational" databases simply aren't designed for relational data. Thus, there is no solution here with Mongo. I've ended up using Neo4j, but SQL would work fine as well.
meteor add reywood:publish-composite
Meteor.publishComposite('tweets', function(username) {
return {
find: function() {
// Find the current user's following users
return Relationships.find({ follower: username });
},
children: [{
find: function(relationship) {
// Find tweets from followed users
return Tweets.find({user: relationship.following});
}
}]
}
});
Meteor.publish('ownTweets', function(username) {
return Tweets.find({user: username});
});
Related
I have a page with posts and likes for each post.
In FireStore a collection of posts and a collection of likes, and I update the total_likes and recent likes array when a user likes or unlikes a post with cloud functions.
However, I can't figure out how to show for each post if the currently logged in user liked it or not. What's an efficient way to do that for.
Any pointers?
I believe you might need to look at data aggregration. Even though this example is with Angular, I also use the same principle in a different application: https://angularfirebase.com/lessons/firestore-cloud-functions-data-aggregation/
Alternatively, you could store the post_id's that your user likes in their own 'like_array'. Knowing which posts the user currently sees, you can cross reference the shown post_id's with the (single object) 'like_array' from the user to determine if he/she has liked a particular post. In the long run, you could disambiguate like_arrays based on days or weeks, and only query the like_arrays of this and last day/week - based on what post you are showing. If you are working with categories of posts, similar applies: different like_arrays for different categories.
Hope this helps!
One solution would be to have another collection in your Firestore database where you create a document by user, in which you save (and update) an object containing all the posts this user has liked.
Like
- likers (Collection)
- UserUID (doc)
- postIds {
post1_UID: true,
post2_UID: true
}
The idea is to use the technique described in the doc, here: https://firebase.google.com/docs/firestore/solutions/arrays#solution_a_map_of_values
I don't know which language you use in the front end but in JavaScript you would do:
var postToTestId = ....; <- You set this value as you need (e.g. as a function parameter)
firebase.auth().signInWithEmailAndPassword("...", ".....")
.then(function (info) {
var postId = 'azer';
return db.collection('likers')
.where('postIds.'+ postToTestId, '==', true)
.get();
})
.then(function(querySnapshot) {
if (querySnapshot.size > 0) {
console.log("USER LIKES THIS POST!!!");
}
})
.catch(function (error) {
console.log(error);
});
I don't think there is any solution without storing somewhere all the posts each user liked...
I have a custom publication on my server (which in some way join 2 collections).
This resulting set of this publication is exactly what I need but for performances issues I would like to avoid sending it entirely to the client.
If I did not care about performances, I would only subscribe to the
publication and do something like
theCollection.find({"my":"filter"})
I am therefore trying to find a way to publish a subset of the custom publication so that the filter would be applied on the custom publication on the server side.
Is there a way to chain or filter publications (server side) ?
For the question we can assume the custom publication to look like this and cannot be modified:
Meteor.publish('customPublication', function() {
var sub = this;
var aCursor = Resources.find({type: 'someFilter'});
Mongo.Collection._publishCursor(aCursor, sub, 'customPublication');
sub.ready();
});
if i understand the question right, you are looking for https://atmospherejs.com/reywood/publish-composite
It let's you "publish a set of related documents from various collections using a reactive join. This makes it easy to publish a whole tree of documents at once. The published collections are reactive and will update when additions/changes/deletions are made."
Ok I came to the following workaround. Instead of working on the publication, I simply added a new collection I update according to the other collections. In order to do so I am using the meteor hooks package
function transformDocument(doc)
{
doc.aField = "aValue"; // do what you want here
return doc;
}
ACollection.after.insert(function(userId, doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
ACollection.after.update(function(userId, doc, fieldNames, modifier, options)
{
var transformedDocument = transformDocument(doc);
delete transformedDocument._id;
AnotherCollection.update(doc._id,{$set:transformedDocument});
});
ACollection.after.remove(function(userId, doc)
{
AnotherCollection.remove(doc._id);
});
Then I have the new collection I can publish subsets the regular way
Benefits:
You can filter whatever you want into this db, no need to worry if the field is virtual or real
Only one operation every time a db changes. This avoid having several publication merging the same data
Cave eats:
This requires one more Collection = more space
The 2 db might not be always synchronised, there is few reasons for this:
The client manually changed the data of "AnotherCollection"
You had documents in "ACollection" before you added "AnotherCollection".
The transform function or source collection schema changed at some point
To fix this:
AnotherCollection.allow({
insert: function () {
return Meteor.isServer;
},
update: function () {
return Meteor.isServer;
},
remove: function () {
return Meteor.isServer;
}
});
And to synchronise at meteor startup (i.e. build the collection from scratch). Do this only once for maintenance or after adding this new collection.
Meteor.startup(function()
{
AnotherCollection.remove({});
var documents = ACollection.find({}).fetch();
_.each(documents, function(doc)
{
var transformedDocument = transformDocument(doc);
AnotherCollection.insert(transformedDocument);
});
});
For instance, when your permissions are group-based, and your user document has a list of groups that the user belongs to. I'm publishing docs in an Items collection, and you should only be able to view items with a groupOwner field matching a group you belong to.
It would be nice if you could autorun inside a publish, but I doubt you can:
Meteor.publish 'screened-items', ->
Deps.autorun ->
user = Users.findOne #userId
return Items.find {groupOwner: {$in: user.groups}}
If you can't, this is the best I can come up with, but it's going to be slow and memory-intensive. Is this the only way to do it?
Meteor.publish 'screened-items', ->
user = Users.findOne #userId
# (hope that the db doesn't change between this line and the observeChanges)
saved_items = Items.find({groupOwner: {$in: user.groups}}).fetch()
# call #added on each item
handle = Users.findOne(#userId).observeChanges {
changed: (_, fields) =>
if fields.groups
new_items = Items.find({groupOwner: {$in: fields.groups}}).fetch()
# compare new_items to saved_items, and call #added() or #removed() for each difference
}
#ready()
#.onStop ->
handle.stop()
You can achieve this two ways:
Use the publish-with-relations package, for example:
Meteor.publish 'screend-items', ->
# select the current user
Meteor.publishWithRelations
handle: this
collection: Meteor.users
filter:
_id: #userId
options:
fields:
groups: 1
mappings: [
key: 'groupOwner' # and map to the `groupOwner` field on Items
collection: Items
]
Denormalize the relationship, providing a succinct list of users to use for publishing
Items._ensureIndex(userIds: 1) # best to index this field
# basic publications
Meteor.publish 'screend-items', ->
# don't expose `userIds` to the client
return Items.find({userIds: #userId}, {fields: userIds: false})
If you want the published docs to change when the userId changes, that is the default behaviour.
However, if the logged-in user changes, the publish function is rerun with the new value. - from docs.meteor.com.
Deps.autorun() only works on the client while Meteor.publish() only works on the server. So you can not autorun inside of publish.
If you are okay to let the client see the 'groups' they're in, the code is a bit simpler because you can start and stop the subscription when the groups change. Like this:
//on client
Deps.autorun( function() {
Meteor.subscribe( 'items', Meteor.user().groups );
});
//on server
Meteor.publish( 'items', function( groups ){
var self = this;
var user = Meteor.users.findOne( {_id: self.userId});
if ( ! (user && user.groups === groups) )
return;
return Items.find({groupOwner: {$in: groups}});
});
Otherwise you would need use two observers inside the publish function - one to watch user for changes to groups and another to manage publishing items that are in the group. See this example of doing a join of collections that way.
Is there a way to store subscriptions of the same server collection in a different minimongo collection?
If not is there any best practice to work around?
I do have a summary table having 50k datasets with a lot of details in the documents.
// Server
var collection = new Meteor.Collection("collection");
Meteor.publish("detail", function (id) {
return collection.find({_id: id});
});
// A pager that does not include the data (fields:{data:0})
Meteor.publish("master", function (filter, sort, skip, limit) {
return collection.find({name: new RegExp("^" + filter + "|\\s" + filter, "i")},
{limit: limit,
skip: skip,
sort: options,
fields: {data: 0}
});
});
// Client
var collection = new Meteor.Collection("collection");
Deps.autorun(function () {
Meteor.subscribe("master",
Session.get("search"),
Session.get("sort"),
Session.get("skip"),
Session.get("limit")
);
Meteor.subscribe("detail", Session.get("selection"));
});
Problem above: both subscriptions are feed into the same collection.
This does not work well if the results of the finds are stored in the same local collection.
Having a local collection with the name of the subscription/publish would be great.
// Client
var detail = new Meteor.Collection("detail"),
master = new Meteor.Collection("master");
Any Ideas how to encourage subscriptions to use my own collections??
I found the solution through help of Andrews Hint the Discover Meteor book that shows a lot of publishing subscription scenarios.
Anyway: After reading I discovered that the question I was heading for is also answered in the Meteor documentation Meteor.publish
The last example basically creates a virtual collection "counts" for the "messages" collection. Well done already ;-)
"Wer lesen kann ist im Vorteil!"
I need to inform clients about changes on server side. In my case I am using different Collections on server and on client (more about it in this question: how would you build pinterest like page with meteor.js).
On the server I am getting new Products from external API. I would like to publish the number of new items to all clients that they could update their local variables needed for layout to work well.
How to do it?
It would be nice if I could publish/subscribe other kinds of data than Meteor.Collection. I found Meteor.deps, but what I understand it works only on client side.
To accomplish what you want you do need another collection - on the client. On the server, in a publish function, build a document from scratch assigning the current count of Products to an attribute. Using observe() and set, modify count when documents are added or removed from Products. Subscribe to the count "record set" on the client.
// Server
Meteor.publish('count', function () {
// Build a document from scratch
var self = this;
var uuid = Meteor.uuid();
var count = Products.find().count();
// Assign initial Products count to document attribute
self.set('count', uuid, {count: count});
// Observe Products for additions and removals
var handle = Products.find().observe({
added: function (doc, idx) {
count++;
self.set('counts', uuid, {count: count});
self.flush();
},
removed: function (doc, idx) {
count--;
self.set('counts', uuid, {count: count});
self.flush();
}
});
self.complete();
self.flush();
self.onStop(function () {
handle.stop();
});
});
// Client
Counts = new Meteor.Collection('count');
Meteor.subscribe('count');
console.log('Count: ' + Counts.findOne().count);
I must say the above solution showed me one way, but still, what if I need to publish to client data that are not connected with observe()? Or with any collection?
In my case I have i.e. 1000 products. To engage visitors I am "refreshig" the collection by updating the timestamp of random number of products, and displaying collection sorted by timestamp. Thank to this visitors have impression that something is happening.
My refresh method returns number of products (it is random). I need to pass that number to all clients. I did it, but using (I think) ugly workaround.
My refresh method sets Session.set('lastRandomNo', random). BTW: I didn't know that Session works on server side. refresh updates Products collection.
Then accoriding to above answer:
Meteor.publish 'refreshedProducts', ->
self = this
uuid = Meteor.uuid()
# create a new collection to pass ProductsMeta data
self.set('products_meta', uuid, { refreshedNo: 0 })
handle = Products.find().observe
changed: (newDocument, atIndex, oldDocument) ->
self.set('products_meta', uuid, { refreshedNo: Session.get('lastRandomNo') })
self.flush()
self.complete()
self.flush()
self.onStop ->
handle.stop()
and on client side:
ProductsMeta = new Meteor.Collection('products_meta')
# subscribe to server 'products_meta' collection that is generated by server
Meteor.subscribe('refreshedProducts')
ProductsMeta.find({}).observe
changed: (newDocument, atIndex, oldDocument) ->
# I have access to refreshedNo by
console.log ProductsMeta.findOne().refreshedNo
What do you think?