Meteor collection 'Client' is parent of collection 'Projects'
foreign key in Projects is 'client_id'.
Now I want to have the Projects collection extended with the Client name. If there is no Client-parent, then make Client name 'XXX'.
(Easy to see the 'orphaned' projects: Just take those with XXX as Client name)
Meteor code is as follows:
collections.js
Client = new Mongo.Collection('clients');
Project = new Mongo.Collection('projects', {
transform: function (doc) {
var client = Client.findOne({"_id":doc.client_id},{"_id":0, "name":1});
if (!client) {
// Client name is 'XXX'
return _.extend(doc, {"client_name":"XXX"});
} else {
return _.extend(doc, {"client_name":client.name});
}
}
}
);
publications.js
Meteor.publish('clientNewYork', function () {
// This publication only New York clients, and only the name.
return Client.find({"city":"New York"}, {fields: {"name": 1}}, {sort: name: -1});
});
Meteor.publish('projects', function () {
return Project.find();
});
subscriptions.js
Meteor.subscribe('clientNewYork');
Meteor.subscribe('projects');
Application starts with first subscription: A list of all Clients in New York.
Next screen is all Projects, enriched with Client name.
Important: ALL projects, not only the ones from New York Clients.
Weird thing happening:
All projects of Clients not in New York get 'XXX' as Client name.
That means that the publication is taken into account in the transform of the collection.
My suspicion:
The non New York clients are just not in the MiniMongo in my browser.
But I expect that the collection.js goes to the database running on the server instead of the minimongo on the browser.
Workaround:
Change the client pubication to all clients instead of only New York ones.
But my feeling says this is not good.
Am I doing something wrong here?
Transforms are run on a document whenever it is fetched. Your suspicion is correct:
When the project documents are fetched on the client, they each look for a corresponding client document. Because a find on the client will hit minimongo, only published clients (in your case NYC clients) will be matched.
Without knowing more about your implementation details it's hard to suggest a precise solution. Some options include:
Denormalize the data by storing the client_name field on the project (avoid the transform). The downside is that whenever you update a client's name, you will also need to update the corresponding project(s). Have a look at collection-hooks.
Have a more flexible publish function for clients. If you could identify the current project or the current set of projects, you could pass an id or an array of ids to the publish function when subscribing. In this way, you could still join the two collections without publishing the entire set of client documents.
Related
I've read the Firebase docs on Stucturing Data. Data storage is cheap, but the user's time is not. We should optimize for get operations, and write in multiple places.
So then I might store a list node and a list-index node, with some duplicated data between the two, at very least the list name.
I'm using ES6 and promises in my javascript app to handle the async flow, mainly of fetching a ref key from firebase after the first data push.
let addIndexPromise = new Promise( (resolve, reject) => {
let newRef = ref.child('list-index').push(newItem);
resolve( newRef.key()); // ignore reject() for brevity
});
addIndexPromise.then( key => {
ref.child('list').child(key).set(newItem);
});
How do I make sure the data stays in sync in all places, knowing my app runs only on the client?
For sanity check, I set a setTimeout in my promise and shut my browser before it resolved, and indeed my database was no longer consistent, with an extra index saved without a corresponding list.
Any advice?
Great question. I know of three approaches to this, which I'll list below.
I'll take a slightly different example for this, mostly because it allows me to use more concrete terms in the explanation.
Say we have a chat application, where we store two entities: messages and users. In the screen where we show the messages, we also show the name of the user. So to minimize the number of reads, we store the name of the user with each chat message too.
users
so:209103
name: "Frank van Puffelen"
location: "San Francisco, CA"
questionCount: 12
so:3648524
name: "legolandbridge"
location: "London, Prague, Barcelona"
questionCount: 4
messages
-Jabhsay3487
message: "How to write denormalized data in Firebase"
user: so:3648524
username: "legolandbridge"
-Jabhsay3591
message: "Great question."
user: so:209103
username: "Frank van Puffelen"
-Jabhsay3595
message: "I know of three approaches, which I'll list below."
user: so:209103
username: "Frank van Puffelen"
So we store the primary copy of the user's profile in the users node. In the message we store the uid (so:209103 and so:3648524) so that we can look up the user. But we also store the user's name in the messages, so that we don't have to look this up for each user when we want to display a list of messages.
So now what happens when I go to the Profile page on the chat service and change my name from "Frank van Puffelen" to just "puf".
Transactional update
Performing a transactional update is the one that probably pops to mind of most developers initially. We always want the username in messages to match the name in the corresponding profile.
Using multipath writes (added on 20150925)
Since Firebase 2.3 (for JavaScript) and 2.4 (for Android and iOS), you can achieve atomic updates quite easily by using a single multi-path update:
function renameUser(ref, uid, name) {
var updates = {}; // all paths to be updated and their new values
updates['users/'+uid+'/name'] = name;
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
updates['messages/'+messageSnapshot.key()+'/username'] = name;
})
ref.update(updates);
});
}
This will send a single update command to Firebase that updates the user's name in their profile and in each message.
Previous atomic approach
So when the user change's the name in their profile:
var ref = new Firebase('https://mychat.firebaseio.com/');
var uid = "so:209103";
var nameInProfileRef = ref.child('users').child(uid).child('name');
nameInProfileRef.transaction(function(currentName) {
return "puf";
}, function(error, committed, snapshot) {
if (error) {
console.log('Transaction failed abnormally!', error);
} else if (!committed) {
console.log('Transaction aborted by our code.');
} else {
console.log('Name updated in profile, now update it in the messages');
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.on('child_added', function(messageSnapshot) {
messageSnapshot.ref().update({ username: "puf" });
});
}
console.log("Wilma's data: ", snapshot.val());
}, false /* don't apply the change locally */);
Pretty involved and the astute reader will notice that I cheat in the handling of the messages. First cheat is that I never call off for the listener, but I also don't use a transaction.
If we want to securely do this type of operation from the client, we'd need:
security rules that ensure the names in both places match. But the rules need to allow enough flexibility for them to temporarily be different while we're changing the name. So this turns into a pretty painful two-phase commit scheme.
change all username fields for messages by so:209103 to null (some magic value)
change the name of user so:209103 to 'puf'
change the username in every message by so:209103 that is null to puf.
that query requires an and of two conditions, which Firebase queries don't support. So we'll end up with an extra property uid_plus_name (with value so:209103_puf) that we can query on.
client-side code that handles all these transitions transactionally.
This type of approach makes my head hurt. And usually that means that I'm doing something wrong. But even if it's the right approach, with a head that hurts I'm way more likely to make coding mistakes. So I prefer to look for a simpler solution.
Eventual consistency
Update (20150925): Firebase released a feature to allow atomic writes to multiple paths. This works similar to approach below, but with a single command. See the updated section above to read how this works.
The second approach depends on splitting the user action ("I want to change my name to 'puf'") from the implications of that action ("We need to update the name in profile so:209103 and in every message that has user = so:209103).
I'd handle the rename in a script that we run on a server. The main method would be something like this:
function renameUser(ref, uid, name) {
ref.child('users').child(uid).update({ name: name });
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
messageSnapshot.update({ username: name });
})
});
}
Once again I take a few shortcuts here, such as using once('value' (which is in general a bad idea for optimal performance with Firebase). But overall the approach is simpler, at the cost of not having all data completely updated at the same time. But eventually the messages will all be updated to match the new value.
Not caring
The third approach is the simplest of all: in many cases you don't really have to update the duplicated data at all. In the example we've used here, you could say that each message recorded the name as I used it at that time. I didn't change my name until just now, so it makes sense that older messages show the name I used at that time. This applies in many cases where the secondary data is transactional in nature. It doesn't apply everywhere of course, but where it applies "not caring" is the simplest approach of all.
Summary
While the above are just broad descriptions of how you could solve this problem and they are definitely not complete, I find that each time I need to fan out duplicate data it comes back to one of these basic approaches.
To add to Franks great reply, I implemented the eventual consistency approach with a set of Firebase Cloud Functions. The functions get triggered whenever a primary value (eg. users name) gets changed, and then propagate the changes to the denormalized fields.
It is not as fast as a transaction, but for many cases it does not need to be.
I have 2 collections: Meteor.users and Projecs.
Users collection have field "projects" which contains array of user's project's ids.
"projects" : [
"jut6MHx6a7kSALPEP",
"XuJNvq7KTRheK6dSZ"
]
Also I have a publication for user's projects:
Meteor.publish('projects', function() {
var userProjects = Meteor.users.findOne(this.userId).projects;
return Projects.find({_id: {$in: userProjects}});
});
Everything works fine, but when I add new project (and update users ("projects" field) who are in this project) reactive publication doesn't works. Projects page doesn't contains recently added project. It works only when I refresh page.
Subscription made in router:
waitOn: function() {
return [
Meteor.subscribe('projects')
]
},
What should I do with this publication? Thanks a lot.
This is happening because Meteor.users is not reactive. I don't know what the reason behind but I saw many developers, specially developers who try to get famous by publish really cool articles about their awesome application, exposing the tokens.
So if some idiot publish the Meteor.users to the browser, it's a security flaw. It would be even worst if it was reactive because the token would be updated in realtime. Maybe this a block to newbie who don't really know that they're doing. Just my opinion about this decision.
This collection is design to be used for managing users and after the login, it makes no sense to use to store data, as it is designed.
Yea, this is a known "problem". Publish functions aren't reactive, so Meteor.users.findOne(this.userId).projects will only be evaluated when the client subscribes. You'll find a lot of information about this if you search for "meteor reactive joins", for example https://www.discovermeteor.com/blog/reactive-joins-in-meteor/
In your case, the clients will always have access to their array of project ids, right? Then the simplest solution would probably be to do something like this on the client:
Tracker.autorun(function(){
var user = Meteor.user()
if(user){
Meteor.subscribe(user.projects)
}
})
So, when the client notices that the array of project ids has changed, it renews the subscription (I'm unsure if passing user.projects to the subscribe call is necessary, but I'm a bit afraid that the subscription isn't is renewed if it's called with the same arguments as before).
Using the reactive-publish package (I am one of authors) you can do:
Meteor.publish('projects', function () {
this.autorun(function (computation) {
var userProjects = Meteor.users.findOne(this.userId, {fields: {projects: 1}}).projects;
return Projects.find({_id: {$in: userProjects}});
});
});
Just be careful to limit the first query only to projects so that autorun is not rerun for changes in other fields.
I have a Project collection and a Task collection.
Each project has a user_id field, this holds the owner of the project.
Each task has a project_id field. So the structure is something like this:
User 1
Project 1
Task 1
Task 2
Project 2
Task 3
User 2
Project 3
Task 4
Task 5
For security purposes I only want to publish the projects belonging to a certain logged in user. For the project itself that's quite easy:
Meteor.publish('projects', function(){
return Projects.find({user_id: this.userId});
});
But how do I do this in a clean way for the Task collection? And why does the Collection.Allow doesn't have a 'view' option?
Something like:
Tasks.allow({
view: function (userId, doc) {
return Projects.findOne(doc.project_id).user_id == userId;
}
});
would be nice, is there a reason it's not there?
First, some recommended reading:
Reactive joins in meteor
A similar question on SO
Joins in meteor are currently tricky. It's easy to just join the collections in a publish function, but it isn't always straightforward to make them reactive (run again when things change).
Non-Reactive Options
You could publish both collections at the same time with:
Meteor.publish('projectsAndTasks', function() {
var projectsCursor = Projects.find({user_id: this.userId});
var projectIds = projectsCursor.map(function(p) { return p._id });
return [
projectsCursor,
Tasks.find({project_id: {$in: projectIds}});
];
});
The potential problem is that if tasks were added to a new project, they would not be published (see "The Naive Approach" from the first article above). Depending on how your application starts and stops its subscriptions, this may not matter. If you find that it does, keep reading.
Reactive Options
A simple option is just to denormalize the data. If you also added user_id to your tasks, then no joins are necessary, and the publish function looks like:
Meteor.publish('projectsAndTasks', function() {
var projectsCursor = Projects.find({user_id: this.userId});
var tasksCursor = Tasks.find({user_id: this.userId});
return [projectsCursor, tasksCursor];
});
If that doesn't appeal to you and you are using iron-router, you can do a client-side join in your routes (see "Joining On The Client" from the first article above). It's a bit slower because you need a second round trip but it's clean in that no data needs to be modified and no external packages need to be added.
Finally, you can do a reactive join on the server, either manually using observeChanges (not recommended), or by using a package. I have used publish-with-relations in the past, but it has some issues as pointed out in the articles). For a more complete list of package options, you can see this thread.
Not being a core developer on meteor, I don't have a precise answer for why allow/deny doesn't have a "read" option, but I'll take an educated guess. Depending on how the allow/deny function was written, the publisher would potentially have to run an expensive callback for every single document or partial update. The allow/deny callbacks are easy to tolerate when a single document is being modified, but if you suddenly need to publish several hundred documents and each one needs to be separately evaluated before being transmitted, I don't think that would be practical. I'm pretty sure that's why publishers can act alone as the arbiter of document read authorization.
You can do this for the tasks:
Meteor.publish('tasks', function(){
var projects = Projects.find({user_id: this.userId}, {fields: {_id: 1}});
var projectIdList = projects.map(function(project) { return project._id;});
return Tasks.find({project_id: {$in: projectIdList}});
});
First we get all the projects belonging to the user. We will only need the _id field so we filter the other fields
Then we map the _id's of the projects to a new array.
Then we publish a tasks.find that includes all the project ids in the mapped array.
The allow construction you mentionend is by my knowledge only ment to be used with updates and inserts
I'm trying to get some information via the collection-helpers package for non-logged in users and I'm obviously missing something fundamental here as I'm getting nowhere.
I have a relationship set up what is happily returning the profile.name element for the owner of a document, as long as that happens to coincide with the logged in user, but, I'm getting nothing back for non-logged in users (because of the security on the client side).
I've added a new publication on both client and server as
// User Profile
Meteor.publish("userProfile", function() {
return Meteor.users.find({_id: this.userId},
{fields: {'profile': 1}});
});
and have subscribed to this publication in the js associated with the page I'm trying to display it in
// Don't need this to be reactive, so
Meteor.subscribe("userProfile");
but am still not getting access to the profile data in the document with
<h4>Posted by: {{projOwner.profile.name}}</h4>
where projOwner looks like
projectDocs.helpers({
projOwner: function() {
console.log(this.owner._id);
var owner = Meteor.users.findOne(this.owner._id);
//console.log("Owner is: " +owner);
return owner;
}
});
What am I doing wrong??
In a publish function, this.userId is always the id of the currently logged in user. The profile of the current user is automatically published so that function doesn't do anything useful.
The real problem here is you need to get the correct subset of users published to the client. Maybe that's the project owner of the document you are looking at, maybe it's a all of the users in a group, etc. Without knowing more about your problem it's hard to say.
An easy place to start with is just publishing all of the users to make sure your code works, and then try reducing the set. Remember that publish functions can take arguments, so you could pass in, for example, the id of a project and then publish the owner like so:
Meteor.publish('projectOwner', function(projectId) {
check(projectId, String);
var project = Projects.findOne(projectId);
return Meteor.users.find(project.owner, {
fields: {'profile': 1}
});
});
I have a server side mongo collection called Profiles.
I need to publish and subscribe to the entire collection of Profiles if user: adminId.
That way the administrator can edit, updated, etc... each Profile collection item.
But I want users to be able to see their Profile record.
So I tried this...
CLIENT SIDE
MyProfile = new Meteor.Collection("myprofile");
Meteor.subscribe('profiles');
Meteor.subscribe('myprofile');
COMMON - CLIENT AND SERVER SIDE
Profiles = new Meteor.Collection("profiles");
SERVER SIDE - The publishing and subscribing of profiles works fine.
// this returns all profiles for this User
// if they belong to an ACL Group that has acl_group_fetch rights
Meteor.publish("profiles", function() {
var user_groups = Groups.find({users: this.userId()});
var user_groups_selector = [];
user_groups.forEach(function (group) {
user_groups_selector.push(group._id);
});
return Profiles.find( {
acl_group_fetch: {
$in: user_groups_selector
}
});
});
Here is where the problem seems to begin. The Profiles.find is returning collection items because I can output them to the console server side. But for some reason the publish and subscribe is not working. The client receives nothing.
// return just the users profile as myprofile
Meteor.publish("myprofile", function() {
return Profiles.find({user: this.userId()});
});
Any ideas what I am doing wrong. I want to be able to publish collections of records that User A can insert, fetch, update, delete but User B (C, D and E) can only see their record.
I think your issue is more on the MongoDB side than with meteor. Given your case I'd do two collections (Group and Profile).
Each document in the Group collection would feature an array containing DBRefs to documents in the Profile collection (actually users so I would think about renaming the Profile collection to User as imo that's more intuitive).
Same for the Profile collection and its documents; each document in the profile collection (representing a user) would have an array field containing DBrefs to groups the user belongs to (documents inside the Group collection).
I'm not entirely sure of how you're checking for the error or not, but I think you may be running into a gotcha I ran into. When you publish data with the Profiles collection, even though the pub/sub calls use the 'myprofile' name, the data always is always available in the collection that you're returning a cursor for... in this case, the data you publish in the 'myprofile' publication will show up in the 'profiles' collection on the client. The publish call doesn't create a 'myprofile' collection on the client. So if you're trying to find() on the 'myprofile' collection you won't see any data. (Meteor/MongoDB won't complain that the collection doesn't exist because they always will lazily create it when you reference it.)
I think the problem here is that you only need the one collection: Profiles.
So if you just remove the offending line
MyProfile = new Meteor.Collection("myprofile");
Everything should work fine (you'll have both datasets within the Profiles collection).