I'm dealing with a problem where a user can update a document within a specified time limit, and if he doesn't, the server will.
The update involves incrementing a value and adding an object to an array of a document. I need to ensure that only one of the user/server updates the document. Not both.
To ensure this happens, some checks are run to see if the document has already been updated, but there are times where the user and server run at exactly the same time and both pass the checks and then the document is updated twice.
I've been trying many different ways of fixing this, but I haven't been able to. I tried implement a lock similar to this: http://en.wikipedia.org/wiki/Peterson%27s_algorithm to ensure that only one update will happen and the second update will fail, but I haven't been successful. Any ideas?
To ensure this happens, some checks are run to see if the document has already been updated, but there are times where the user and server run at exactly the same time and both pass the checks and then the document is updated twice.
You can achieve this by using a MongoDB update query that simultaneously checks if the value has been updated and updates it. Like this:
var post = Posts.findOne("ID");
// ... do some stuff with the post ...
Posts.update({counter: post.counter}, {$push: {items: newItem}, $inc: {counter: 1}});
As you can see, in one query we both check the counter and increment it - so if two of these queries run one right after another only one will actually update the document (since the counter won't match anymore).
Related
When writing event based cloud functions for firebase firestore it's common to update fields in the affected document, for example:
When a document of users collection is updated a function will trigger, let's say we want to determine the user info state and we have a completeInfo: boolean property, the function will have to perform another update so that the trigger will fire again, if we don't use a flag like needsUpdate: boolean to determine if excecuting the function we will have an infinite loop.
Is there any other way to approach this behavior? Or the situation is a consequence of how the database is designed? How could we avoid ending up in such scenario?
I have a few common approaches to Cloud Functions that transform the data:
Write the transformed data to a different document than the one that triggers the Cloud Function. This is by far the easier approach, since there is no additional code needed - and thus I can't make any mistakes in it. It also means there is no additional trigger, so you're not paying for that extra invocation.
Use granular triggers to ensure my Cloud Function only gets called when it needs to actually do some work. For example, many of my functions only need to run when the document gets created, so by using an onCreate trigger I ensure my code only gets run once, even if it then ends up updating the newly created document.
Write the transformed data into the existing document. In that case I make sure to have the checks for whether the transformation is needed in place before I write the actual code for the transformation. I prefer to not add flag fields, but use the existing data for this check.
A recent example is where I update an amount in a document, which then needs to be fanned out to all users:
exports.fanoutAmount = functions.firestore.document('users/{uid}').onWrite((change, context) => {
let old_amount = change.before && change.before.data() && change.before.data().amount ? change.before.data().amount : 0;
let new_amount = change.after.data().amount;
if (old_amount !== new_amount) {
// TODO: fan out to all documents in the collection
}
});
You need to take care to avoid writing a function that triggers itself infinitely. This is not something that Cloud Functions can do for you. Typically you do this by checking within your function if the work was previously done for the document that was modified in a previous invocation. There are several ways to do this, and you will have to implement something that meets your specific use case.
I would take this approach from an execution time perspective, this means that the function for each document will be run twice. Each time when the document is triggered, a field lastUpdate would be there with a timestamp and the function only updates the document if the time is older than my time - eg 10 seconds.
I have a node on firebase that lists all the players in the game. This list will update as and when new players join. And when the current user ( me ) disconnects, I would like to remove myself from the list.
As the list will change over time, at the moment I disconnect, I would like to update this list and update firebase.
This is the way I am thinking of doing it, but it doesn't work as .update doesnt accept a function. Only the object. But if I create the object beforehand, when .onDisconnect calls, it will not be the latest object... How should I go about doing this?
payload.onDisconnect().update( () => {
const withoutMe = state.roomObj
const index = withoutMe.players.indexOf( state.userObj.name )
if ( index > -1 ) {
withoutMe.players.splice( index, 1 )
}
return withoutMe
})
The onDisconnect handler was made for this use-case. But it requires that the data of the write operation is known at the time that you set the onDisconnect. If you think about it, this should make sense: since the onDisconnect happens after your client is disconnected, the data of the data of that write operation must be known before the disconnect.
It sounds like you're building a so-called presence system: a list that contains a node for each user that is currently online. The Firebase documentation has an example of such a presence system. The key difference from your approach is that it in the documentation each user only modifies their own node.
So: when the user comes online, they write a node for themselves. And then when they get disconnected, that node gets removed. Since all users write their node under the same parent, that parent will reflect the users that are online.
The actual implementation is a bit more involved since it deals with some edge cases too. So I recommend you check out the code in the documentation I linked, and use that as the basis for your own similar system.
After lots of reading, I'm starting to get a better handle on Meteor's publish/subscribe model. I've removed the autopublish training wheels from my first app and while I have most everything working, I am seeing one issue.
When the app first loads, my publish and subscribe hooks work great. I have a block of code that runs in a Tracker.autorun() block which makes the subscribe calls, I am able to sequentially wait for data from the server using ready() on my subscribe handles, etc.
One feature of my app is that it allows the user to insert new documents into a collection. More specifically, when the user performs a certain action, this triggers an insert. At that point, the client-side JS runs and the insert into MiniMongo completes. The reactive autorun block runs and the client can see the inserted documented. The client updates the DOM with the new inserted data and all is well.
Furthermore, when I peek into the server-side MongoDB, I see the inserted document which means the server-side JS is running fine as well.
Here's where it gets weird. The client-side autorun block runs a second time (I'm not sure why) and this time, the client no longer has the inserted item. When the DOM renders, the newly inserted item is now gone. If I reload the page, all is well again.
Has anyone seen this behavior before? I'm also noticing that the server-side publish call runs once on page load but then it doesn't run again after the insert. This seems wrong because how else will the client get the reconciled data from the server after the insertion (i.e. after Meteor's client-side latency compensation)?
The important functions (ComponentInstances is the collection that is bugging out):
Publish block:
Meteor.publish('allComponentInstances', function (documentId, screenIndex) {
console.log(`documentId: ${documentId} screenIndex: ${screenIndex}`)
const screens = Screens.find({ownerDocumentId: documentId})
const selectedScreen = screens.fetch()[screenIndex]
return ComponentInstances.find({_id: {$in: selectedScreen.allComponentInstanceIds}})
})
Subscription block in autorun:
// ... a bunch of irrelevant code above
const allComponentInstancesHandle = Meteor.subscribe('allComponentInstances', document._id, 0)
if (allComponentInstancesHandle.ready()) {
isReady = true
screens = Screens.find({ownerDocumentId: document._id}).fetch()
const componentInstanceObjects = ComponentInstances.find().fetch()
allComponentInstances = {}
componentInstanceObjects.map((componentInstance) => {
allComponentInstances[componentInstance._id] = componentInstance
})
}
This is most probably you're inserting documents from client side. And you have not set up your permission rules properly. When you remove autopublish and insecure from your app, you are not allowed to insert/update/remove documents into collection unless you have allow/deny rules set up in the server side.
Meteor has a great feature called latency compensation which tries emulate your db operations before it gets the actual write operation in the db. And when the server tries to write in the db, it looks for allow/deny rules.If the permission rules doesn't allow the db operation or Whatever the reason( either allow/deny or authentication) for not actually written in the db, then the server data gets synchronized with your client side db.
This is why i assume you are seeing your document being inserted for the first time and gets disappeared within a second.
check this section of meteor docs.
http://docs.meteor.com/#/full/allow
I ended up solving this a different way. The core issue, I believe, has nothing to do with accept/deny rules. In fact, their role is still hazy to me.
I realize now what I've been reading all along in the Meteor docs: the publish functions return cursors. If the cursor itself doesn't change (e.g. if you're passing specific keys you want to fetch), then it won't really work as a reactive data source in the sense that new documents in a collection will not make the data publish again. You are, after all, still requesting the same keys.
The way forward is to come up with a publish cursor that accurately reflects the reactive data you want to retrieve. This sounds abstract but in practice, it means make sure the cursor is general, not specific to the specific keys you are retrieving.
I have a SysOperation Framework process that creates a ReliableAsynchronous batch to post packing slips and several get created at a time.
Depending on how quickly I click to create them, I get:
Cannot edit a record in LastValue (SysLastValue).
An update conflict occurred due to another user process deleting the record or changing one or more fields in the record.
And
Cannot create a record in LastValue (SysLastValue). User ID: t edit a, Class.
The record already exists.
On a couple of them in the BatchHistory. I have this.parmLoadFromSysLastValue(false); set. I'm not sure how to prevent writing to SysLastValue table.
Any idea what could be going on?
I get this exception a lot too, so I've created the habit of catching DuplicateKeyException in my service operation. When it is thrown, catch it and retry (for a default of 5x).
The error occurs when a lot of processes run simultaneously, like you are doing now.
DupplicateKeyException can be caught inside a transaction so you could improve by putting a try/catch around the code that does the insert in the SysLastValue table if you can find the code.
As far as I can see these are the only to occurrences where a record is inserted in this table (except maybe in kernel):
InventUnusedDimCleanUp.serialize()
SysAutoSemaphore.autoSemaphore()
Put a breakpoint there and see if that code is executed. If so you can add a try/catch with retry and see if that "fixes" it.
You could also use the tracing cockpit and the trace parser to figure out where that record is inserted if it's not one of those two.
My theory about LoadFromSysLastValue: I believe setting this.parmLoadFromSysLastValue(false) does not work since it is only taken into account when the dialog is started, not when your operation is executed. When in batch, no SysLastValue will be used to initialize your data contract as you want it to use the exact parameters you have supplied in your data contract .
It's because of the code calling SysOperationController.savelast() while in batch, my solution is to set loadFromSysLastValue to false in SysOperationController.loadFromSysLastValue() as part of the in batch check:
if (!this.isInBatch())
{
.....
}
//Begin
else
{
loadFromSysLastValue = false;
}
//End
On my meteor project users can post events and they have to choose (via an autocomplete) in which city it will take place. I have a full list of french cities and it will never be updated.
I want to use a collection and publish-subscribes based on the input of the autocomplete because I don't want the client to download the full database (5MB). Is there a way, for performance, to tell meteor that this collection is "static"? Or does it make no difference?
Could anyone suggest a different approach?
When you "want to tell the server that a collection is static", I am aware of two potential optimizations:
Don't observe the database using a live query because the data will never change
Don't store the results of this query in the merge box because it doesn't need to be tracked and compared with other data (saving memory and CPU)
(1) is something you can do rather easily by constructing your own publish cursor. However, if any client is observing the same query, I believe Meteor will (at least in the future) optimize for that so it's still just one live query for any number of clients. As for (2), I am not aware of any straightforward way to do this because it could potentially mess up the data merging over multiple publications and subscriptions.
To avoid using a live query, you can manually add data to the publish function instead of returning a cursor, which causes the .observe() function to be called to hook up data to the subscription. Here's a simple example:
Meteor.publish(function() {
var sub = this;
var args = {}; // what you're find()ing
Foo.find(args).forEach(function(document) {
sub.added("client_collection_name", document._id, document);
});
sub.ready();
});
This will cause the data to be added to client_collection_name on the client side, which could have the same name as the collection referenced by Foo, or something different. Be aware that you can do many other things with publications (also, see the link above.)
UPDATE: To resolve issues from (2), which can be potentially very problematic depending on the size of the collection, it's necessary to bypass Meteor altogether. See https://stackoverflow.com/a/21835534/586086 for one way to do it. Another way is to just return the collection fetch()ed as a method call, although this doesn't have the benefits of compression.
From Meteor doc :
"Any change to the collection that changes the documents in a cursor will trigger a recomputation. To disable this behavior, pass {reactive: false} as an option to find."
I think this simple option is the best answer
You don't need to publish your whole collection.
1.Show autocomplete options only after user has inputted first 3 letters - this will narrow your search significantly.
2.Provide no more than 5-10 cities as options - this will keep your recordset really small - thus no need to push 5mb of data to each user.
Your publication should look like this:
Meteor.publish('pub-name', function(userInput){
var firstLetters = new RegExp('^' + userInput);
return Cities.find({name:firstLetters},{limit:10,sort:{name:1}});
});