We have a problem with our meteor server. When we publish 300 or so items with Meteor.publish/Meteor.subscribe the server increases its memory and eventually becomes unresponsive.
We thought of:
1) monitor the number of reactive subscribtions / memory taken by an active subscription
2) make something like ,,one time publish" - ignore changes in server side collection
Any thoughts on how any of the above can be accomplished ?
Or any other tips to debug /improve meteor app performance ?
Thanks
zorlak's answer is good.
Some other things:
You can do a one-time publish by writing your own custom publisher via the this.set API, based on the code in _publishCursor. You'd do something like:
Meteor.publish("oneTimeQuery", function () {
MyCollection.find().forEach(function (doc) {
sub.added("collectionName", doc._id, doc);
});
sub.ready();
});
This does a query, sends its results down, and then never updates it again.
That said, we hope that Meteor's performance will be such that this is unnecessary!
I'd also like to add an easy way to get stats (like number of observed cursors) from an app to Meteor (exposed as an authenticated subscription) but haven't had the time yet.
As of Meteor 0.5.1, one thing you can do is to remove dependencies on the userId from the publish function. If a publish function does not depend on which user is subscribing, then Meteor will cache the db query so it doesn't get slower as more users subscribe.
See this Meteor blog post: http://meteor.com/blog/2012/11/20/meteor-051-database-scaling
Related
I'm sure these are common scenarios, but after researching some hours, I couldn't really find what the common practice is. Maybe someone with more experience in Firebase can point me to the right direction.
I have two particular scenarios:
1. Code that runs once
Example 1: adding new data to all users in firestore, which is needed for a new feature
Example 2: start duplicating data into existing documents
I currently write the code in a cloud function, and run it on a firestore event (onUpdate of a "hidden" document) and then I immediately delete the function if everything goes well.
I also increase the timeout and memory for this function, as the idea is to potentially update millions of documents.
2. Manually trigger a function from the firebase console (or command line)
Example: Give a user admin privileges in the app (function that sets custom claims and firestore data). We don't have time to implement a back-office, so doing this from the firebase web portal/console would be ideal, specifying the user id.
My current solution is to use a https function, and run it from the GCP portal (on the function's "Testing" tab, being able to pass a json). BUT the function can be triggered publicly, which I don't really like...
What are the common practices for these scenarios?
To expand on my comment: if you want to create a node script to run one-off code, you just write your JS code like for any cloud function but simply run it immediately. Something like this.
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();
db.collection('users')
.where('myField', '==', true)
.get()
.then(querySnapshot => {
querySnapshot.docs.forEach(doc => {
// update doc
});
});
If you save this as script.js and execute it with node script.js you’ll be pointed towards downloading a JSON file with credentials. Follow the instructions and you can then run the script again and now you’re running your own code on Firestore, from the command line.
For administrative type operations, you're better off just running them on your desktop or some other server you control. Cloud Functions is not well suited for long running operations, or things that must just happen once on demand.
Case 1 really should be managed by a standalone program or script that you can monitor by running it on your desktop.
Case 2 can be done a number of ways, such as building your own admin web site. But you might find it easiest to mirror the contents of a document to custom claims using a Firestore trigger. Read this: https://medium.com/firebase-developers/patterns-for-security-with-firebase-supercharged-custom-claims-with-firestore-and-cloud-functions-bb8f46b24e11
Currently developing a hybrid mobile app using ionic. When the app starts up, and a user writes to the Realtime Database for the first time, it's always delayed by around 10 or more seconds. But any subsequent writes are almost instantaneous (less than 1 second).
My calculation of delay is based on watching the database in the Firebase console.
Is this a known issue, or maybe I am doing something wrong. Please share your views.
EDIT:
The write is happening via Firebase Cloud Function.
This is the call to the Firebase Cloud function
this.http.post(url+"/favouritesAndNotes", obj, this.httpOptions)
.subscribe((data) => {
console.log(data);
},(error)=>{
console.log(error);
});
This is the actual function
app.post('/favouritesAndNotes', (request, response) => {
var db = admin.database().ref("users/" + request.body.uid);
var favourites = request.body.favourites;
var notes = request.body.notes;
if(favourites!==undefined){
db.child("favourites/").set(favourites);
}
if(notes!==undefined){
db.child("notes/").set(notes);
}
console.log("Write successfull");
response.status(200).end();
});
The first time you interact with the Firebase Database in a client instance, the client/SDK has to do quite some things:
If you're using authentication, it needs to check if the token that it has is still valid, and if not refresh it.
It needs to find the server that the database is currently hosted on.
It needs to establish a web socket connection.
Each of these may take multiple round trips, so even if you're a few hundred ms from the servers, it adds up.
Subsequent operations from the same client don't have to perform these steps, so are going to be much faster.
If you want to see what's actually happening, I recommend checking the Network tab of your browser. For the realtime database specifically, I recommend checking the WS/Web Socket panel of the Network tab, where you can see the actual data frames.
I'm writing a Meteor app which allows clients to execute terminal commands on the server at the click of a button.
I know how to do this with a single command:
//server
Meteor.methods({ exec : cmd => { ... } })
//client
Meteor.call('exec', cmd, (err, result) => {
console.log(result)
})
But now I'm trying to implement a more complex protocol and don't quite know what the best way is. I want the client to kick off a series of commands, have the server run them and tell me, step by step, whether they succeeded or failed.
Obviously I could implement this with the above code by writing client-side code that runs exec with the first command, checks the result from the server, runs exec with the next command and so on.
The crux is that in my case the series of commands is always the same, so it would make much more sense to only do one Meteor.call on the client -- the server would know what commands to run. However I would also like to have the results of the individual commands available on the client as they come in -- and this is what I can't do, because Meteor.call only returns once, of course.
What I'm looking for is a sort of stream or iterator through which I can send a number of messages to the client until everything is done. I've seen some outdated packages called meteor-streams and similar that might be able to do something like that, but I'm thinking there must be a smart way in Meteor itself to solve this. Ideas?
A common solution is a Notifications collection. Create the collection with a schema: for: ${userid}, msg: ${msg string}, type: ${err success etc}. Create a Notifications publication, which publishes docs with the users userid.
You can then subscribe to the Notifications collection in some main template page on the client. Use observeChanges to look for changes to the collection and either console.log them, use JavaScript to display them on the page or simply install a package like sAlerts to handle them.
Inside the observe changes callback, a seenNotification method should be called which removes the notification from the db, so it is not shown again.
I'll post code snippets a bit later.
Have a look at this: https://github.com/RocketChat/meteor-streamer
I think it will solve your problem easily.
A very simply scenario: in a local meteor (v 1.2.1) development environment (WebStorm), with autopublish, and insecure enabled, I have a single MongodbDB (v 3.0.4) collection 'Letters'. I wish to respond immediately to any documents being added, removed, or modified in this collection.
For this purpose, I have the following autorun function:
Template.diagram.rendered = function(){
Tracker.autorun(function () {
Letters.find({}).observe({
added: function(document) {
console.log('a new document has been added');
},
changed: function(newDocument) {
console.log('a document has been changed');
},
removed: function(document) {
console.log('a document has been removed');
}
});
})
}
When a new document is added from within the same application, I can see the console messages right away (meteor latency compensation). However, when I connect to the same MongoDB database using an external tool (Robomongo), and add, change, or remove a document within the 'Letters' collection - it takes about 6-10 seconds before this change is detected, and the corresponding console message appears in the browser.
Why does it take this long, instead of being almost instantaneous?
Once I have this question posted on meteor forums, I was pointed to a meteor blog post from 2014, which describes the oplog tailing feature, and the fact, that it is only on by default in the dev instance. This made me realize, that, by using MONGO_URL env variable with my dev application, I was forcing my meteor app to work with the MongoDB instance, which was running on my mac all the time, independently from my meteor development, and, as such, was considered as "production" by my meteor app. Once I have switched the app to work with ad-hock mongo connection / db, the oplog tailing went into effect, and I started to see immediate event propagation to the browser.
Thanks, #dburles from the meteor forums!
I am trying to implement some methods for a DDP API, for use with a C# remote client.
Now, I want to be able to track the connection to implement some type of persistent session, to this end, id hope to be able to use the session id given by DDP on connection, example:
{
"msg": "connected",
"session": "CmnXKZ34aqSnEqscR"
}
After reading the documentation, I see that inside meteor methods, I can access the current connection using "this.connection", however, I always get an undefined "this.connection".
Was it removed? If so, how can i access it now?
PS: I dont want to login as a user and access this.userId, since the app I want to create should not login, but actually just get a document id and do work associated with that, including changes to other collections, but all, regarding ONLY this id, and I dont want to have to include this id every time I call a function, since, this could possibly lead security problems if anyone can just send any id. The app would ideally do a simple login, then associate token details with his "session".
Changing from:
() => { this.connection; }
to:
function() { this.connection; }
solves the problem from me. Based on a comment in the accepted answer.
The C# client on github has a few bugs with it as it doesn't follow the DDP spec exactly. When you send commands to it to connect and run a call, it usually sends the '.call' too soon.
The method does work if you do it this way with this.connection on the server side Meteor method.
You need to make sure you send the method calls after you know that you are actually connected. This is what works at least with Meteor 0.8.2
I was using a file named ".next.js" to force meteor to use the newest unsupported javascript spec using a package.
Somehow this messed it up. Changed back to default javascript and it now works.
Thank you :)
init.coffee
Meteor.startup ->
# client init
if Meteor.isClient
Meteor.call "init"
methods.coffee
Meteor.methods
init: ->
console.log #connection.httpHeaders.host
it's that easy...