Is there any way in meteor to do bulk changes in published collections in server side?... like updating/inserting hundreds or thousands records without each individual record being sent to all subscribers one by one?
I am pulling in periodically third party data and just want to gather all updates or inserts in one pull as one batch update so all clients will receive it as one change package not thousands of mini-updates. Doing it one by one creates a big bottleneck in my app atm.
If there is no support in meteor for this atm then should i just do the updates directly to mongo and let meteor to pick it up on the next mongo poll?
// imagine myChanges array with 1000 items
myChanges.forEach(function(change){
// this will trigger the sync with clients immediately... 1000 times
// currently this will practically hang my server
// i want to gather the changes here instead
MyCollection.update({_id: change.docId}, change);
});
// and trigger the sync here instead
Thanks,
Reio
Depending on how complex your application is, or how many publish calls you have, this could be acheived by rolling your own Publish functions.
i.e. Instead of
Meteor.publish("myCollection", function() {
return myCollection.find(); } );
Create your own Publish / Cursor and add hooks to it.
Meteor.publish("myCollection", function() {
globalObserver = myCollection.find().observe({
added: function(item) {
publication.added(item);
}
// And So on
});
Your batch updates would then need to interact with this publish function in some way, causing it to stop and re-intialise the observer once your updates are complete.
Related
After lots of reading, I'm starting to get a better handle on Meteor's publish/subscribe model. I've removed the autopublish training wheels from my first app and while I have most everything working, I am seeing one issue.
When the app first loads, my publish and subscribe hooks work great. I have a block of code that runs in a Tracker.autorun() block which makes the subscribe calls, I am able to sequentially wait for data from the server using ready() on my subscribe handles, etc.
One feature of my app is that it allows the user to insert new documents into a collection. More specifically, when the user performs a certain action, this triggers an insert. At that point, the client-side JS runs and the insert into MiniMongo completes. The reactive autorun block runs and the client can see the inserted documented. The client updates the DOM with the new inserted data and all is well.
Furthermore, when I peek into the server-side MongoDB, I see the inserted document which means the server-side JS is running fine as well.
Here's where it gets weird. The client-side autorun block runs a second time (I'm not sure why) and this time, the client no longer has the inserted item. When the DOM renders, the newly inserted item is now gone. If I reload the page, all is well again.
Has anyone seen this behavior before? I'm also noticing that the server-side publish call runs once on page load but then it doesn't run again after the insert. This seems wrong because how else will the client get the reconciled data from the server after the insertion (i.e. after Meteor's client-side latency compensation)?
The important functions (ComponentInstances is the collection that is bugging out):
Publish block:
Meteor.publish('allComponentInstances', function (documentId, screenIndex) {
console.log(`documentId: ${documentId} screenIndex: ${screenIndex}`)
const screens = Screens.find({ownerDocumentId: documentId})
const selectedScreen = screens.fetch()[screenIndex]
return ComponentInstances.find({_id: {$in: selectedScreen.allComponentInstanceIds}})
})
Subscription block in autorun:
// ... a bunch of irrelevant code above
const allComponentInstancesHandle = Meteor.subscribe('allComponentInstances', document._id, 0)
if (allComponentInstancesHandle.ready()) {
isReady = true
screens = Screens.find({ownerDocumentId: document._id}).fetch()
const componentInstanceObjects = ComponentInstances.find().fetch()
allComponentInstances = {}
componentInstanceObjects.map((componentInstance) => {
allComponentInstances[componentInstance._id] = componentInstance
})
}
This is most probably you're inserting documents from client side. And you have not set up your permission rules properly. When you remove autopublish and insecure from your app, you are not allowed to insert/update/remove documents into collection unless you have allow/deny rules set up in the server side.
Meteor has a great feature called latency compensation which tries emulate your db operations before it gets the actual write operation in the db. And when the server tries to write in the db, it looks for allow/deny rules.If the permission rules doesn't allow the db operation or Whatever the reason( either allow/deny or authentication) for not actually written in the db, then the server data gets synchronized with your client side db.
This is why i assume you are seeing your document being inserted for the first time and gets disappeared within a second.
check this section of meteor docs.
http://docs.meteor.com/#/full/allow
I ended up solving this a different way. The core issue, I believe, has nothing to do with accept/deny rules. In fact, their role is still hazy to me.
I realize now what I've been reading all along in the Meteor docs: the publish functions return cursors. If the cursor itself doesn't change (e.g. if you're passing specific keys you want to fetch), then it won't really work as a reactive data source in the sense that new documents in a collection will not make the data publish again. You are, after all, still requesting the same keys.
The way forward is to come up with a publish cursor that accurately reflects the reactive data you want to retrieve. This sounds abstract but in practice, it means make sure the cursor is general, not specific to the specific keys you are retrieving.
I have several function calls in a row that run and wait to return, then the next one runs. After these are run I have one function I want to run, but then I don't want to wait for it to be done before I run my return.
Here is an example of what I mean.
get_card, create_order, create_association and debit_order all need to wait for the previous function to complete before they can run. When I get to Queue.start_account_creation_task I want it to start running, but then let the return on the line below run right away too.
Meteor.methods({
singleDonation: function (data) {
logger.info("Started singleDonation");
//Get the card data from balanced and store it
var card = Utils.get_card(customerData._id, data.paymentInformation.href);
//Create a new order
var orders = Utils.create_order(data._id, customerData.href);
//Associate the card with the balanced customer
var associate = Utils.create_association(customerData._id, card.href, customerData.href);
//Debit the order
var debitOrder = Utils.debit_order(data.paymentInformation.total_amount, data._id, customerData._id, orders.href, card.href);
Queue.start_account_creation_task(customerData._id, data._id, debitOrder._id);
return {c: customerData._id, don: data._id, deb: debitOrder._id};
}
});
Sounds like you need parallel and serial control for tasks. The (as in, 400,000 downloads a day) Node.js module for that is called async, and a Meteor wrapper for it is peerlibrary:async.
Sooner or later you'll need a dedicated background task management package. If async is insufficient, have a look at my evaluation of packages to control background tasks in Meteor.
The thing that seemed to work the best for what I was trying to do was to just use a Meteor.setTimeout({}). It might seem like an odd choice, but it does everything I needed, including setting the Meteor Environment so that I didn't have to do any BindEnrironment call. It also breaks out of the current thread of calls, which means it then returns the result to the client and a second later finishes the rest of the calls (which are to external APIs that I didn't need my users sitting there waiting for).
I'm building an online store in meteor where customers can customize products in the store. I have setup a client-only collection called Inventory which stores all the product data and is updated accordingly in response to user input. Once the user is ready to checkout, I dump the product data into a client & server side collection called ShoppingCart. I want to allow users to go back and revise their edits on the product in Inventory so I setup my router to $set data from the ShoppingCart into Inventory if it finds a match:
Router.route '/:_type/:_id', ->
Session.set "inCart", false
#render #params._type,
data: =>
storedItem = ShoppingCart.findOne {
userId: Meteor.userId(),
image: #params._id
}
if storedItem?
delete storedItem._id
Inventory.update {image: #params._id}, {
$set: storedItem
}
Inventory.findOne image: #params._id
EDIT: This seems to cause my router method to get stuck in an infinite loop whenever data in Inventory changes. Is there any way to avoid this issue? Is there a better way of handling this kind of data altogether that I should consider?
MAJOR CAVEAT - I don't do CoffeeScript, so this is what I can gather having put your code through a compiler.
I think the problem is that the data function is reactive, and you're updating and returning an item from the Inventory collection within it. Every time the route runs, unless there is no storedItem, it's going to invalidate a computation on which it itself depends and thus rerun again immediately (and subsequently do the same again, etc...).
As a general rule, I think it's a very bad idea indeed to be updating a collection from within a data function - if you have to do this within the route function, consider the onRun, or onBeforeAction hooks for the update.
Final thing, just because I don't understand: why do you need to dump the item from the ShoppingCart back into Inventory? Shouldn't it already be there, unless the user has started a new session?
I have a situation in which I need to subscribe to the same collection twice. The two publish methods in my server-side code are as follows:
Meteor.publish("selected_full_mycollection", function (important_id_list) {
check(important_id_list, Match.Any); // should do better check
// this will return the full doc, including a very long array it contains
return MyCollection.find({
important_id: {$in: important_id_list}
});
});
Meteor.publish("all_brief_mycollection", function() {
// this will return all documents, but only the id and first item in the array
return MyCollection.find({}, {fields: {
important_id: 1,
very_long_array: {$slice: 1}
}});
});
My problem is that I am not seeing the full documents on the client end after I subscribe to them. I think this is because they are being over-written by the method that publishes only the brief versions.
I don't want to clog up my client memory with long arrays when I don't need them, but I do want them available when I do need them.
The brief version is subscribed to on startup. The full version is subscribed to when the user visits a template that drills down for more insight.
How can I properly manage this situation?
TL/DR - skip to the third paragraph.
I'd speculate that this is because the publish function thinks that the very_long_array field has already been sent to the client, so it doesn't send it again. You'd have to fiddle around a bit to confirm this, but sending different data on the same field is bound to cause some problems.
In terms of subscribing on two collections, you're not supposed to be able to do this as the unique mongo collection name needs to be provided to the client and server-side collections object. In practice, you might be able to do something really hacky by making one client subscription a fake remote subscription via DDP and having it populate a totally separate Javascript object. However, this cannot be the best option.
This situation would be resolved by publishing your summary on something other than the same field. Unfortunately, you can't use transforms when returning cursors from a publish function (which would be the easiest way), but you have two options:
Use the low-level publications API as detailed in this answer.
Use collection hooks to populate another field (like very_long_array_summary) with the first item in the array whenever very_long_array changes and publish just the summary field in the former publication.
A third option might be publishing the long version to a different collection that exists for this purpose on the client only. You might want to check the "Advanced Pub/Sub" Chapter of Discover Meteor (last sub chapter).
I'm having trouble with realizing a reactive publication with a moving date. I have a calendar app with events/meetings. Im displaying these events on a special view page but i only want to display events of today and only with a starting time of currenttime - 30 mins.
The code im having and trying always works fine on initial load, after refreshing and when I add/delete an event from an admin page (collection gets refreshed?). But when I leave the page open the events which have passed just stay on the page. Whenever I add/remove a event via the admin page the publication is updated just fine. I assume this is because the publication isn't refreshing the date in the query or something?
I have tried:
normal publications and just subscribing via either iron-router
before hook or via Deps.autorun
publish all events and filtering on the client side
a publication with observeChanges
keep the vars in a deps.autorun function and passing them via the subscription as parameters
but I just keep getting the same results.
publication with observeChanges:
Meteor.publish('currentEventsObserve', function(calId) {
var self = this;
var nowMin30mins = moment().subtract('minutes',30).valueOf();
var endOfToday = moment(moment().format('DD-MM-YYYY 23:59'), 'DD-MM-YYYY HH:mm').valueOf();
var handle = Events.find({
calId : calId, //CalendarId
eventDate: { $gt: nowMin30mins, $lt: endOfToday }
},
{
sort: { eventDate: 1 },
limit: 5
}).observeChanges({
added: function(id,event){
//console.log("added: ",id,event);
self.added("events", id, event);
},
removed: function (id,event) {
//console.log("removed: ",id,event);
self.removed("events", id);
}
});
self.ready();
self.onStop(function () {
handle.stop();
});
});
As said before: the above works fine on initial load and refreshes, but after leaving the page open for a while the events are staying on there and not being removed from the publication.
Also, whenever I check the collection ones certain events already should have been removed via Events.find().fetch() for example, the events are still there, so it's not the template which isn't updating correctly.
I hope it's clear what I mean. I have read many other questions about this sort of reactivity but I just can't figure it out.
The problem is that nowMin30mins is only computed once when the publication is activated, and won't update afterwards, even though you have an observeChanges on Events. You basically have two constant values (nowMin30mins and endOfToday) bracketing the documents that will be shown and only adding or removing items will cause them to disappear.
I can think of two ways for how you'd want to do this reactively.
You can put all the reactivity on the client. Just send the entire day's events over in a publish, which is very simple, and have the client filter out what's between 30 mins from now and the end of the day using Date.now() in a computation that updates every minute or so, or when new events are added/deleted. It doesn't seem to incur a lot of overhead to store those events anyway and you can offload the computational cost of observes, which can build up if you have a lot of clients
If you want to do it on the server, you'll have to do things a little more carefully than what you have now. Because the live query won't account for the change in time, you'll need to watch all of today's events and then add or remove them from the subscription if they are created, or deleted or go out of range, respectively. To reiterate, you can't do this purely with an observeChanges because the current time is always changing.