Meteor duplicate insert conflict resolution - meteor

Is there a design pattern in meteor application to handle multiple clients inserting the same logical record 'simultaneously'?
Specifically I have a scoring type application, and multiple clients could create the initial, basically blank, Score record for an Entrant when the entrant is ready to start. The appearance of the record is then used to make it available on the page for editing by the officials, incrementing penalty counts and such.
Stages = new Meteor.Collection("contests");
Entrants = new Meteor.Collection("entrants");
Scores = new Meteor.Collection("scores");
// official picks the next entrant
Scores.insert( stage_id:xxxx, entrant_id:yyyy)
I am happy with the implications of the conflict resolutions of edits to the Score record once it is in the Collection. I am not sure how to deal with multiple clients trying to insert the Score for the stage_id/entrant_id pair.
In a synchronous app I would tend to use some form of interlocking, or a relational DB key constraint.

Well, according to this answer Meteor $upsert flag is still in enhancement list and seems to be added in stable branch after 1.0 release.
So the first way is how it was said to add an unique index:
All the implementation ways are listed here. I would recommend you to use native mongo indexes, not a code implementation.
The optimistic concurrency way is much more complicated according to no transactions in MongoDB.
Here comes my implementation of it(be careful, might be buggy))):
var result_callback = function(_id) {
// callback to call on successfull insert made
}
var $set = {stage_id: xxxx, entrant_id: xxxx};
var created_at = Date.now().toFixed();
var $insert = _.extend({}, $set, {created_at: created_at});
Scores.insert($insert, function(error, _id) {
if (error) {
//handle it
return;
}
var entries = Scores.find($set, {sort: {created_at: -1}}).fetch()
if (entries.length > 1) {
var duplicates = entries.splice(0, entries.length - 1);
var duplicate_ids = _.map(duplicates, function(entry) {
return entry._id;
});
Scores.remove({_id: {$in: duplicate_ids}})
Scores.update(entries[0]._id, $set, function(error) {
if (error) {
// handle it
} else {
result_callback(entries[0]._id)
}
})
} else {
result_callback(_id);
}
});
Hope this will give you some good ideas)
Sorry, previous version of my answer was completely incorrect.

Related

How do i query a Firebase database for a nested property?

Hi i have a noSql db in firebase.
I want to get the object where userId is 288
i'v tried many combinations but i cant figure out how its done.
This is my code so far :
var refTest= database.ref('conversation')
var query = refTest
.orderByChild('messages');
query.on('value', function(data) {
var a = data.val();
console.log(a.messages.userId);
console.log(data.val());
});
This is a image of my "schema"
I'm obviously a noob when it comes to NoSQL. I do understand SQL
All help is appreciated
You can order/filter on a nested value like this:
var refTest= database.ref('conversation')
var query = refTest.orderByChild('messages/userId').equalTo("288");
query.on('value', function(snapshot) {
snapshot.forEach(function(child) {
console.log(child.key);
console.log(child.val());
});
});
The forEach is needed, since there may be multiple child nodes with messages/userId equal to 288.
The key named "messages" doesn't make sense in your schema. Because if you want to have another message under that conversation, then you wouldn't be able to add it with the same key name and you also couldn't add it under "messages" because it would overwrite the other one. My suggestion is to use the push() method for adding a new message. This way you uniquely identify each message.
Regarding your question, an easy to understand way of parsing your schema is this: you loop through each message of each conversation for finding the messages with userID.
refTest.on('value', function(data) {
var conversations = data.val();
for (conversation in conversations){
for (message in conversation) {
if (message.userId == 288) {
// do whatever you need
// and eventually return something to break the loops
}
}
}
}
Of course, you can adapt it based on your needs

angularFire startAt querying and binding deletes new data

The application shows work-shifts for certain time-period. firebaseConn.getShifts is the API-function to get the shiftData for the given time period.
versions:
firebase: 2.0.6
angularFire: 0.9.0 (confirmed with 0.8.2 also)
This is my firebase schema:
And this is the code:
.factory('watchers', function(bunch-of-dependencies) {
var unbindShifts = function() {};
var inited = false;
var shifts = {};
... some irrelevant code in between ...
function initShifts() {
unbindShifts();
shifts.object = firebaseConn.getShifts( false, from, to, $scope );
$scope.shifts = shifts.object;
shifts.object.$bindTo($scope, "shifts").then(function(unbind) {
unbindShifts = unbind;
});
}
The firebase-queries (that have worked fine before adding the unbind / bind and possibly time-based querying might cause issues too):
firebaseConn.getShifts = function(asArray, from, to, scope) {
return cacheRequest(FBURL + "shifts", asArray, [from, to]);
};
function cacheRequest(url, asArray, limits) {
var type = asArray ? "array" : "object";
var startAt = limits ? limits[0] : undefined;
var endAt = limits ? limits[1] : undefined;
var retObj, FBRef;
cached[url] = cached[url] || {};
/* If there are limits-parameters we don't cache at all atm. Since those queries should be checked differently than static urls */
if(!limits && cached[url][type]) {
FBRef = cached[url][type];
} else {
FBRef = cached[url][type] = createFBRef(url, startAt, endAt);
}
if(asArray) {
retObj = FBRef.$asArray();
} else {
retObj = FBRef.$asObject();
}
return retObj;
}
function createFBRef(resourceURL, startAt, endAt) {
var modifiedObject = $firebase( createRef( resourceURL ).orderByKey().startAt(startAt).endAt(endAt) );
return modifiedObject;
}
function createRef(resourceURL) {
return new Firebase( resourceURL );
}
Now I have located the problem to be with the query limiting. If the from and to Dates are undefined, this works without problems. But I need to be able to limit the amount of data, since loading many years of workshift-data, to show a weeks time, won't be good :).
The actual problem is not displaying and fetching the data, everything works fine, it's related to the times and re-binding.
If I do any changes to e.g. "20150115"-table. For example I add another "groups"-child there. When i unbind and rebind, the whole "20150115"-table gets deleted and this holds true only to the latest changes. If I add multiple child to different dates e.g. "20150113", "20150114", "20150115" and the latest change is in "20150115" and then I unbind + re-bind another time from firebase, all the other root-paths will stay as they are, but the latest change in "20150115" will make the whole tree deleted.
I hope I make myself clear, so for safety I try to explain it again in simpler way.
- Changes to 1. "20150113", 2. "20150114", 3. "20150115" through the app.
- Changing timeline from UI causes: unbind + re-bind
- As a side-effect the whole "20150114" tree gets deleted.
The problem is somehow related to advanced querying with orderByKey().startAt(startAt).endAt(endAt) and binding.
Also for additional info. The data which is added through the UI gets added to the firebase database, but when the re-binding happens, the data is deleted from the database. Specifically on rebind, unbinding causes no issues, if I delay rebinding with timeout.
EDIT:
I have found the source of the actual issue. After the new binding is in place and everything seems to be in order, there is an angular watch event that kicks in. The event tries to save the last change user made before re-binding.
So if I have and active timeline for december (20141201 - 20141230) and I change "20141225"-data. Then change the timeline to 20150101 - 20150130, causing unbind and rebind (or manually fetching new data). There will be an event, after the binding has been done and everything seems to be in order, trying to save 20141225 data to either the new timeline (20150101 - 20150130) or the old one, not sure which one. This causes the firebase to actually delete the whole 20141225-tree, instead of saving the data.
The new data makes it into your Firebase fine, which you can see by either checking your Firebase dashboard or by running a quick snippet like this in your browser's dev console:
new Firebase("https://firebaseurl").once('value', function(s) { console.log(s.val()); })
The data even makes it back into your application. The only problem is that Angular doesn't know that new data has arrived, so it doesn't update the view with the new data.
Normally AngularFire's $asObject and $asArray methods take care of notifying AngularJS when new data arrives from Firebase. But since you are constantly creating new queries, you'll have to take care of that yourself.
There are a few ways to signal the new data to AngularJS and I'm definitely not an expert on which one is best. But if you add $scope.$apply(); to your setDays function it works:
function setDays(ref) {
var FBRange = setFBRange(ref, from, to);
var days;
unbindDays();
days = $firebase(FBRange).$asObject();
$scope.days = days;
days.$bindTo($scope, "days").then(function(unbind) {
unbindDays = unbind;
// As a result of the new binding entry gets mysteriously deleted from firebase
});
$scope.$apply(); // Tell AngularJS about the new data, so that it updates the view
function setFBRange(ref, from, to) {
return ref.orderByKey().startAt(""+from).endAt(from + to + "");
}
}
Updated Plunkr with this change (and some others to help in debugging): http://plnkr.co/edit/YZtkzUNtjQUCcw4xb2mj?p=preview

How to 'transform' data returned via a Meteor.publish?

Meteor Collections have a transform ability that allows behavior to be attached to the objects returned from mongo.
We want to have autopublish turned off so the client does not have access to the database collections, but we still want the transform functionality.
We are sending data to the client with a more explicit Meteor.publish/Meteor.subscribe or the RPC mechanism ( Meteor.call()/Meteor.methods() )
How can we have the Meteor client automatically apply a transform like it will when retrieving data directly with the Meteor.Collection methods?
While you can't directly use transforms, there is a way to transform the result of a database query before publishing it. This is what the "publish the current size of a collection" example describes here.
It took me a while to figure out a really simple application of that, so maybe my code will help you, too:
Meteor.publish("publicationsWithHTML", function (data) {
var self = this;
Publications
.find()
.forEach(function(entry) {
addSomeHTML(entry); // this function changes the content of entry
self.added("publications", entry._id, entry);
});
self.ready();
});
On the client you subscribe to this:
Meteor.subscribe("publicationsWithHTML");
But your model still need to create a collection (on both sides) that is called 'publications':
Publications = new Meteor.Collection('publications');
Mind you, this is not a very good example, as it doesn't maintain the reactivity. But I found the count example a bit confusing at first, so maybe you'll find it helpful.
(Meteor 0.7.0.1) - meteor does allow behavior to be attached to the objects returned via the pub/sub.
This is from a pull request I submitted to the meteor project.
Todos = new Meteor.Collection('todos', {
// transform allows behavior to be attached to the objects returned via the pub/sub communication.
transform : function(todo) {
todo.update = function(change) {
Meteor.call('Todos_update', this._id, change);
},
todo.remove = function() {
Meteor.call('Todos_remove', this._id);
}
return todo;
}
});
todosHandle = Meteor.subscribe('todos');
Any objects returned via the 'todos' topic will have the update() and the remove() function - which is exactly what I want: I now attach behavior to the returned data.
Try:
let transformTodo = (fields) => {
fields._pubType = 'todos';
return fields;
};
Meteor.publish('todos', function() {
let subHandle = Todos
.find()
.observeChanges({
added: (id, fields) => {
fields = transformTodo(fields);
this.added('todos', id, fields);
},
changed: (id, fields) => {
fields = transformTodo(fields);
this.changed('todos', id, fields);
},
removed: (id) => {
this.removed('todos', id);
}
});
this.ready();
this.onStop(() => {
subHandle.stop();
});
});
Currently, you can't apply transforms on the server to published collections. See this question for more details. That leaves you with either transforming the data on the client, or using a meteor method. In a method, you can have the server do whatever you want to the data.
In one of my projects, we perform our most expensive query (it joins several collections, denormalizes the documents, and trims unnecessary fields) via a method call. It isn't reactive, but it greatly simplifies our code because all of the transformation happens on the server.
To extend #Christian Fritz answer, with Reactive Solution using peerlibrary:reactive-publish
Meteor.publish("todos", function() {
const self = this;
return this.autorun(function(computation) {
// Loop over each document in collection
todo.find().forEach(function(entry) {
// Add function to transform / modify each document here
self.added("todos", entry._id, entry);
});
});
});

Meteor minimongo dynamic cursor

In my client UI I have a form with differents search criterias, and I'd like to reactively update the results list. The search query is transformed into a classical minimongo selector, saved in a Session variable, and then I have observers to do things with the results:
// Think of a AirBnb-like application
// The session variable `search-query` is updated via a form
// example: Session.set('search-query', {price: {$lt: 100}});
Offers = new Meteor.Collection('offers');
Session.setDefault('search-query', {});
resultsCursor = Offers.find(Session.get('search-query'));
// I want to add and remove pins on a map
resultCursor.observe({
added: Map.addPin,
removed: Map.removePin
});
Deps.autorun(function() {
// I want to modify the cursor selector and keep the observers
// so that I would only have the diff between the old search and
// the new one
// This `modifySelector` method doesn't exist
resultsCursor.modifySelector(Session.get('search-query'));
});
How could I implement this modifySelector method on the cursor object?
Basically I think this method needs to update the compiled version of the cursor, ie the selector_f attribute, and then rerun observers (without losing the cache of the previous results). Or is there any better solution?
Edit: Some of you have misunderstood what I'm trying to do. Let me provide a complete example:
Offers = new Meteor.Collection('offers');
if (Meteor.isServer && Offers.find().count() === 0) {
for (var i = 1; i < 4; i++) {
// Inserting documents {price: 1}, {price: 2} and {price: 3}
Offers.insert({price:i})
}
}
if (Meteor.isClient) {
Session.setDefault('search-query', {price:1});
resultsCursor = Offers.find(Session.get('search-query'));
resultsCursor.observe({
added: function (doc) {
// First, this added observer is fired once with the document
// matching the default query {price: 1}
console.log('added:', doc);
}
});
setTimeout(function() {
console.log('new search query');
// Then one second later, I'd like to have my "added observer" fired
// twice with docs {price: 2} and {price: 3}.
Session.set('search-query', {});
}, 1000);
}
This doesn't solve the problem in the way you seem to be wanting to, but I think the result is still the same. If this is a solution you explicitly don't want, let me know and I can remove the answer. I just didn't want to put code in a comment.
Offers = new Meteor.Collection('offers');
Session.setDefault('search-query', {});
Template.map.pins = function() {
return Offers.find(Session.get('search-query'));
}
Template.map.placepins = function(pins) {
// use d3 or whatever to clear the map and then place all pins on the map
}
Assuming your template is something like this:
<template name="map">
{{placepins pins}}
</template>
One solution is to manually diff the old and the new cursors:
# Every time the query change, do a diff to add, move and remove pins on the screen
# Assuming that the pins order are always the same, this use a single loop of complexity
# o(n) rather than the naive loop in loop of complexity o(n^2)
Deps.autorun =>
old_pins = #pins
new_pins = []
position = 0
old_pin = undefined # This variable needs to be in the Deps.autorun scope
# This is a simple algo to implement a kind of "reactive cursor"
# Sorting is done on the server, it's important to keep the order
collection.find(Session.get('search-query'), sort: [['mark', 'desc']]).forEach (product) =>
if not old_pin?
old_pin = old_pins.shift()
while old_pin?.mark > product.mark
#removePin(old_pin)
old_pin = old_pins.shift()
if old_pin?._id == product._id
#movePin(old_pin, position++)
new_pins.push(old_pin)
old_pin = old_pins.shift()
else
newPin = #render(product, position++)
new_pins.push(newPin)
# Finish the job
if old_pin?
#removePin(old_pin)
for old_pin in old_pins
#removePin(old_pin)
#pins = new_pins
But it's a bit hacky and not so efficient. Moreover the diff logic is already implemented in minimongo so it's better to reuse it.
Perhaps an acceptable solution would be to keep track of old pins in a local collection? Something like this:
Session.setDefault('search-query', {});
var Offers = new Meteor.Collection('offers');
var OldOffers = new Meteor.Collection(null);
var addNewPin = function(offer) {
// Add a pin only if it's a new offer, and then mark it as an old offer
if (!OldOffers.findOne({_id: offer._id})) {
Map.addPin(offer);
OldOffers.insert(offer);
}
};
var removePinsExcept = function(ids) {
// Clean out the pins that no longer exist in the updated query,
// and remove them from the OldOffers collection
OldOffers.find({_id: {$nin: ids}}).forEach(function(offer) {
Map.removePin(offer);
OldOffers.remove({_id: offer._id});
});
};
Deps.autorun(function() {
var offers = Offers.find(Session.get('search-query'));
removePinsExcept(offers.map(function(offer) {
return offer._id;
}));
offers.observe({
added: addNewPin,
removed: Map.removePin
});
});
I'm not sure how much faster this is than your array answer, though I think it's much more readable. The thing you need to consider is whether diffing the results as the query changes is really much faster than removing all the pins and redrawing them each time. I would suspect that this might be a case of premature optimization. How often do you expect a user to change the search query, such that there will be a significant amount of overlap between the results of the old and new queries?
I have the same problem in my own hobby Meteor project.
There is filter session var where selector is storing. Triggering any checkbox or button changes filter and all UI rerender.
That solution have some cons and the main - you can't share app state with other users.
So i realized that better way is storing app state in URL.
May be it is also better in your case?
Clicking button now change URL and UI rendering based on it. I realize it with FlowRouter.
Helpful reading: Keeping App State on the URL

how to discard initial data in a Firebase DB

I'm making a simple app that informs a client that other clients clicked a button. I'm storing the clicks in a Firebase (db) using:
db.push({msg:data});
All clients get notified of other user's clicks with an on, such as
db.on('child_added',function(snapshot) {
var msg = snapshot.val().msg;
});
However, when the page first loads I want to discard any existing data on the stack. My strategy is to call db.once() before I define the db.on('child_added',...) in order to get the initial number of children, and then use that to discard that number of calls to db.on('child_added',...).
Unfortunately, though, all of the calls to db.on('child_added',...) are happening before I'm able to get the initial count, so it fails.
How can I effectively and simply discard the initial data?
For larger data sets, Firebase now offers (as of 2.0) some query methods that can make this simpler.
If we add a timestamp field on each record, we can construct a query that only looks at new values. Consider this contrived data:
{
"messages": {
"$messageid": {
"sender": "kato",
"message": "hello world"
"created": 123456 // Firebase.ServerValue.TIMESTAMP
}
}
}
We could find messages only after "now" using something like this:
var ref = new Firebase('https://<your instance>.firebaseio.com/messages');
var queryRef = ref.orderBy('created').startAt(Firebase.ServerValue.TIMESTAMP);
queryRef.on('child_added', function(snap) {
console.log(snap.val());
});
If I understand your question correctly, it sounds like you only want data that has been added since the user visited the page. In Firebase, the behavior you describe is by design, as the data is always changing and there isn't a notion of "old" data vs "new" data.
However, if you only want to display data added after the page has loaded, try ignoring all events prior until the complete set of children has loaded at least once. For example:
var ignoreItems = true;
var ref = new Firebase('https://<your-Firebase>.firebaseio.com');
ref.on('child_added', function(snapshot) {
if (!ignoreItems) {
var msg = snapshot.val().msg;
// do something here
}
});
ref.once('value', function(snapshot) {
ignoreItems = false;
});
The alternative to this approach would be to write your new items with a priority as well, where the priority is Firebase.ServerValue.TIMESTAMP (the current server time), and then use a .startAt(...) query using the current timestamp. However, this is more complex than the approach described above.

Resources