Meteor minimongo dynamic cursor - meteor

In my client UI I have a form with differents search criterias, and I'd like to reactively update the results list. The search query is transformed into a classical minimongo selector, saved in a Session variable, and then I have observers to do things with the results:
// Think of a AirBnb-like application
// The session variable `search-query` is updated via a form
// example: Session.set('search-query', {price: {$lt: 100}});
Offers = new Meteor.Collection('offers');
Session.setDefault('search-query', {});
resultsCursor = Offers.find(Session.get('search-query'));
// I want to add and remove pins on a map
resultCursor.observe({
added: Map.addPin,
removed: Map.removePin
});
Deps.autorun(function() {
// I want to modify the cursor selector and keep the observers
// so that I would only have the diff between the old search and
// the new one
// This `modifySelector` method doesn't exist
resultsCursor.modifySelector(Session.get('search-query'));
});
How could I implement this modifySelector method on the cursor object?
Basically I think this method needs to update the compiled version of the cursor, ie the selector_f attribute, and then rerun observers (without losing the cache of the previous results). Or is there any better solution?
Edit: Some of you have misunderstood what I'm trying to do. Let me provide a complete example:
Offers = new Meteor.Collection('offers');
if (Meteor.isServer && Offers.find().count() === 0) {
for (var i = 1; i < 4; i++) {
// Inserting documents {price: 1}, {price: 2} and {price: 3}
Offers.insert({price:i})
}
}
if (Meteor.isClient) {
Session.setDefault('search-query', {price:1});
resultsCursor = Offers.find(Session.get('search-query'));
resultsCursor.observe({
added: function (doc) {
// First, this added observer is fired once with the document
// matching the default query {price: 1}
console.log('added:', doc);
}
});
setTimeout(function() {
console.log('new search query');
// Then one second later, I'd like to have my "added observer" fired
// twice with docs {price: 2} and {price: 3}.
Session.set('search-query', {});
}, 1000);
}

This doesn't solve the problem in the way you seem to be wanting to, but I think the result is still the same. If this is a solution you explicitly don't want, let me know and I can remove the answer. I just didn't want to put code in a comment.
Offers = new Meteor.Collection('offers');
Session.setDefault('search-query', {});
Template.map.pins = function() {
return Offers.find(Session.get('search-query'));
}
Template.map.placepins = function(pins) {
// use d3 or whatever to clear the map and then place all pins on the map
}
Assuming your template is something like this:
<template name="map">
{{placepins pins}}
</template>

One solution is to manually diff the old and the new cursors:
# Every time the query change, do a diff to add, move and remove pins on the screen
# Assuming that the pins order are always the same, this use a single loop of complexity
# o(n) rather than the naive loop in loop of complexity o(n^2)
Deps.autorun =>
old_pins = #pins
new_pins = []
position = 0
old_pin = undefined # This variable needs to be in the Deps.autorun scope
# This is a simple algo to implement a kind of "reactive cursor"
# Sorting is done on the server, it's important to keep the order
collection.find(Session.get('search-query'), sort: [['mark', 'desc']]).forEach (product) =>
if not old_pin?
old_pin = old_pins.shift()
while old_pin?.mark > product.mark
#removePin(old_pin)
old_pin = old_pins.shift()
if old_pin?._id == product._id
#movePin(old_pin, position++)
new_pins.push(old_pin)
old_pin = old_pins.shift()
else
newPin = #render(product, position++)
new_pins.push(newPin)
# Finish the job
if old_pin?
#removePin(old_pin)
for old_pin in old_pins
#removePin(old_pin)
#pins = new_pins
But it's a bit hacky and not so efficient. Moreover the diff logic is already implemented in minimongo so it's better to reuse it.

Perhaps an acceptable solution would be to keep track of old pins in a local collection? Something like this:
Session.setDefault('search-query', {});
var Offers = new Meteor.Collection('offers');
var OldOffers = new Meteor.Collection(null);
var addNewPin = function(offer) {
// Add a pin only if it's a new offer, and then mark it as an old offer
if (!OldOffers.findOne({_id: offer._id})) {
Map.addPin(offer);
OldOffers.insert(offer);
}
};
var removePinsExcept = function(ids) {
// Clean out the pins that no longer exist in the updated query,
// and remove them from the OldOffers collection
OldOffers.find({_id: {$nin: ids}}).forEach(function(offer) {
Map.removePin(offer);
OldOffers.remove({_id: offer._id});
});
};
Deps.autorun(function() {
var offers = Offers.find(Session.get('search-query'));
removePinsExcept(offers.map(function(offer) {
return offer._id;
}));
offers.observe({
added: addNewPin,
removed: Map.removePin
});
});
I'm not sure how much faster this is than your array answer, though I think it's much more readable. The thing you need to consider is whether diffing the results as the query changes is really much faster than removing all the pins and redrawing them each time. I would suspect that this might be a case of premature optimization. How often do you expect a user to change the search query, such that there will be a significant amount of overlap between the results of the old and new queries?

I have the same problem in my own hobby Meteor project.
There is filter session var where selector is storing. Triggering any checkbox or button changes filter and all UI rerender.
That solution have some cons and the main - you can't share app state with other users.
So i realized that better way is storing app state in URL.
May be it is also better in your case?
Clicking button now change URL and UI rendering based on it. I realize it with FlowRouter.
Helpful reading: Keeping App State on the URL

Related

Meteor GroundDB granularity for offline/online syncing

Let's say that two users do changes to the same document while offline, but in different sections of the document. If user 2 goes back online after user 1, will the changes made by user 1 be lost?
In my database, each row contains a JS object, and one property of this object is an array. This array is bound to a series of check-boxes on the interface. What I would like is that if two users do changes to those check-boxes, the latest change is kept for each check-box individually, based on the time the when the change was made, not the time when the syncing occurred. Is GroundDB the appropriate tool to achieve this? Is there any mean to add an event handler in which I can add some logic that would be triggered when syncing occurs, and that would take care of the merging ?
The short answer is "yes" none of the ground db versions have conflict resolution since the logic is custom depending on the behaviour of conflict resolution eg. if you want to automate or involve the user.
The old Ground DB simply relied on Meteor's conflict resolution (latest data to the server wins) I'm guessing you can see some issues with that depending on the order of when which client comes online.
Ground db II doesn't have method resume it's more or less just a way to cache data offline. It's observing on an observable source.
I guess you could create a middleware observer for GDB II - one that checks the local data before doing the update and update the client or/and call the server to update the server data. This way you would have a way to handle conflicts.
I think to remember writing some code that supported "deletedAt"/"updatedAt" for some types of conflict handling, but again a conflict handler should be custom for the most part. (opening the door for reusable conflict handlers might be useful)
Especially knowing when data is removed can be tricky if you don't "soft" delete via something like using a "deletedAt" entity.
The "rc" branch is currently grounddb-caching-2016 version "2.0.0-rc.4",
I was thinking about something like:
(mind it's not tested, written directly in SO)
// Create the grounded collection
foo = new Ground.Collection('test');
// Make it observe a source (it's aware of createdAt/updatedAt and
// removedAt entities)
foo.observeSource(bar.find());
bar.find() returns a cursor with a function observe our middleware should do the same. Let's create a createMiddleWare helper for it:
function createMiddleWare(source, middleware) {
const cursor = (typeof (source||{}).observe === 'function') ? source : source.find();
return {
observe: function(observerHandle) {
const sourceObserverHandle = cursor.observe({
added: doc => {
middleware.added.call(observerHandle, doc);
},
updated: (doc, oldDoc) => {
middleware.updated.call(observerHandle, doc, oldDoc);
},
removed: doc => {
middleware.removed.call(observerHandle, doc);
},
});
// Return stop handle
return sourceObserverHandle;
}
};
}
Usage:
foo = new Ground.Collection('test');
foo.observeSource(createMiddleware(bar.find(), {
added: function(doc) {
// just pass it through
this.added(doc);
},
updated: function(doc, oldDoc) {
const fooDoc = foo.findOne(doc._id);
// Example of a simple conflict handler:
if (fooDoc && doc.updatedAt < fooDoc.updatedAt) {
// Seems like the foo doc is newer? lets update the server...
// (we'll just use the regular bar, since thats the meteor
// collection and foo is the grounded data
bar.update(doc._id, fooDoc);
} else {
// pass through
this.updated(doc, oldDoc);
}
},
removed: function(doc) {
// again just pass through for now
this.removed(doc);
}
}));

How to 'transform' data returned via a Meteor.publish?

Meteor Collections have a transform ability that allows behavior to be attached to the objects returned from mongo.
We want to have autopublish turned off so the client does not have access to the database collections, but we still want the transform functionality.
We are sending data to the client with a more explicit Meteor.publish/Meteor.subscribe or the RPC mechanism ( Meteor.call()/Meteor.methods() )
How can we have the Meteor client automatically apply a transform like it will when retrieving data directly with the Meteor.Collection methods?
While you can't directly use transforms, there is a way to transform the result of a database query before publishing it. This is what the "publish the current size of a collection" example describes here.
It took me a while to figure out a really simple application of that, so maybe my code will help you, too:
Meteor.publish("publicationsWithHTML", function (data) {
var self = this;
Publications
.find()
.forEach(function(entry) {
addSomeHTML(entry); // this function changes the content of entry
self.added("publications", entry._id, entry);
});
self.ready();
});
On the client you subscribe to this:
Meteor.subscribe("publicationsWithHTML");
But your model still need to create a collection (on both sides) that is called 'publications':
Publications = new Meteor.Collection('publications');
Mind you, this is not a very good example, as it doesn't maintain the reactivity. But I found the count example a bit confusing at first, so maybe you'll find it helpful.
(Meteor 0.7.0.1) - meteor does allow behavior to be attached to the objects returned via the pub/sub.
This is from a pull request I submitted to the meteor project.
Todos = new Meteor.Collection('todos', {
// transform allows behavior to be attached to the objects returned via the pub/sub communication.
transform : function(todo) {
todo.update = function(change) {
Meteor.call('Todos_update', this._id, change);
},
todo.remove = function() {
Meteor.call('Todos_remove', this._id);
}
return todo;
}
});
todosHandle = Meteor.subscribe('todos');
Any objects returned via the 'todos' topic will have the update() and the remove() function - which is exactly what I want: I now attach behavior to the returned data.
Try:
let transformTodo = (fields) => {
fields._pubType = 'todos';
return fields;
};
Meteor.publish('todos', function() {
let subHandle = Todos
.find()
.observeChanges({
added: (id, fields) => {
fields = transformTodo(fields);
this.added('todos', id, fields);
},
changed: (id, fields) => {
fields = transformTodo(fields);
this.changed('todos', id, fields);
},
removed: (id) => {
this.removed('todos', id);
}
});
this.ready();
this.onStop(() => {
subHandle.stop();
});
});
Currently, you can't apply transforms on the server to published collections. See this question for more details. That leaves you with either transforming the data on the client, or using a meteor method. In a method, you can have the server do whatever you want to the data.
In one of my projects, we perform our most expensive query (it joins several collections, denormalizes the documents, and trims unnecessary fields) via a method call. It isn't reactive, but it greatly simplifies our code because all of the transformation happens on the server.
To extend #Christian Fritz answer, with Reactive Solution using peerlibrary:reactive-publish
Meteor.publish("todos", function() {
const self = this;
return this.autorun(function(computation) {
// Loop over each document in collection
todo.find().forEach(function(entry) {
// Add function to transform / modify each document here
self.added("todos", entry._id, entry);
});
});
});

how to discard initial data in a Firebase DB

I'm making a simple app that informs a client that other clients clicked a button. I'm storing the clicks in a Firebase (db) using:
db.push({msg:data});
All clients get notified of other user's clicks with an on, such as
db.on('child_added',function(snapshot) {
var msg = snapshot.val().msg;
});
However, when the page first loads I want to discard any existing data on the stack. My strategy is to call db.once() before I define the db.on('child_added',...) in order to get the initial number of children, and then use that to discard that number of calls to db.on('child_added',...).
Unfortunately, though, all of the calls to db.on('child_added',...) are happening before I'm able to get the initial count, so it fails.
How can I effectively and simply discard the initial data?
For larger data sets, Firebase now offers (as of 2.0) some query methods that can make this simpler.
If we add a timestamp field on each record, we can construct a query that only looks at new values. Consider this contrived data:
{
"messages": {
"$messageid": {
"sender": "kato",
"message": "hello world"
"created": 123456 // Firebase.ServerValue.TIMESTAMP
}
}
}
We could find messages only after "now" using something like this:
var ref = new Firebase('https://<your instance>.firebaseio.com/messages');
var queryRef = ref.orderBy('created').startAt(Firebase.ServerValue.TIMESTAMP);
queryRef.on('child_added', function(snap) {
console.log(snap.val());
});
If I understand your question correctly, it sounds like you only want data that has been added since the user visited the page. In Firebase, the behavior you describe is by design, as the data is always changing and there isn't a notion of "old" data vs "new" data.
However, if you only want to display data added after the page has loaded, try ignoring all events prior until the complete set of children has loaded at least once. For example:
var ignoreItems = true;
var ref = new Firebase('https://<your-Firebase>.firebaseio.com');
ref.on('child_added', function(snapshot) {
if (!ignoreItems) {
var msg = snapshot.val().msg;
// do something here
}
});
ref.once('value', function(snapshot) {
ignoreItems = false;
});
The alternative to this approach would be to write your new items with a priority as well, where the priority is Firebase.ServerValue.TIMESTAMP (the current server time), and then use a .startAt(...) query using the current timestamp. However, this is more complex than the approach described above.

Meteor duplicate insert conflict resolution

Is there a design pattern in meteor application to handle multiple clients inserting the same logical record 'simultaneously'?
Specifically I have a scoring type application, and multiple clients could create the initial, basically blank, Score record for an Entrant when the entrant is ready to start. The appearance of the record is then used to make it available on the page for editing by the officials, incrementing penalty counts and such.
Stages = new Meteor.Collection("contests");
Entrants = new Meteor.Collection("entrants");
Scores = new Meteor.Collection("scores");
// official picks the next entrant
Scores.insert( stage_id:xxxx, entrant_id:yyyy)
I am happy with the implications of the conflict resolutions of edits to the Score record once it is in the Collection. I am not sure how to deal with multiple clients trying to insert the Score for the stage_id/entrant_id pair.
In a synchronous app I would tend to use some form of interlocking, or a relational DB key constraint.
Well, according to this answer Meteor $upsert flag is still in enhancement list and seems to be added in stable branch after 1.0 release.
So the first way is how it was said to add an unique index:
All the implementation ways are listed here. I would recommend you to use native mongo indexes, not a code implementation.
The optimistic concurrency way is much more complicated according to no transactions in MongoDB.
Here comes my implementation of it(be careful, might be buggy))):
var result_callback = function(_id) {
// callback to call on successfull insert made
}
var $set = {stage_id: xxxx, entrant_id: xxxx};
var created_at = Date.now().toFixed();
var $insert = _.extend({}, $set, {created_at: created_at});
Scores.insert($insert, function(error, _id) {
if (error) {
//handle it
return;
}
var entries = Scores.find($set, {sort: {created_at: -1}}).fetch()
if (entries.length > 1) {
var duplicates = entries.splice(0, entries.length - 1);
var duplicate_ids = _.map(duplicates, function(entry) {
return entry._id;
});
Scores.remove({_id: {$in: duplicate_ids}})
Scores.update(entries[0]._id, $set, function(error) {
if (error) {
// handle it
} else {
result_callback(entries[0]._id)
}
})
} else {
result_callback(_id);
}
});
Hope this will give you some good ideas)
Sorry, previous version of my answer was completely incorrect.

Deactivating events with .off

Most of my .on events use closures:
ref.on( 'child_added',
function( snapshot ) {
userCallback( snapshot.val() );
} );
which means, it is not possible to deactivate these monitors with .off(), since off needs the original callback pointer as well as the eventType. How can I do this sort of thing?
My app (jQuery Mobile) is page-based. When the user hits a certain page, I want to activate monitors. When the user leaves that page, I want to de-activate, so when they re-enter, I don't end up with multiple monitors. I think I want to call .off() with only eventType and have all callbacks removed.
Right now the only option is to store a reference to the callback. We actually make this a little easier by having .on() return it back to you. So you can do:
var childCallback = ref.on('child_added', function(snapshot) { /* whatever */ });
// later...
ref.off('child_added', childCallback);
But we've received several pieces of feedback like yours, saying it is sometimes cumbersome to keep track of your callback references. So we have a planned API change so that you can call .off() with only an event type or with no arguments at all, and we'll just remove whatever callbacks are registered. But we're focused on other features right now, so this change is probably 1+ months out.
Thanks for the feedback!
In the interim, you can reproduce that sort of behavior with a simple manager pattern. For instance:
function ObserverManager( firebaseRef, page ) {
this.firebaseRef = firebaseRef;
this.listeners = {child_added: [], child_removed: [], value: [], child_updated: [], child_changed: []};
this.page = page;
}
FirebaseObservable.prototype.on(event, callback) {
this.listeners[event].push(
this.firebaseRef.on(event, function(snapshot) {
callback(snapshot.val());
})
);
}
FirebaseObservable.prototype.off(event) {
var list = this.listeners[event], i = list.length;
while(i--) { // 50% more efficient than for(i..; list.length; ...) in IE due to scoping
firebaseRef.off(event, list[i]);
}
}

Resources