I just want to check because I don't see it in the docs. Maybe I'm just missing it.
As far as I can tell if I subscribe with skip then I don't skip on the client. Correct?
I'm using iron router. I have code like this
Router.route('/docs/:_page', {
template: 'doclist',
subscriptions: function() {
var page = parseInt(this.params._page) - 1;
var skip = page * 10;
var limit = 10;
return Meteor.subscribe("pages", skip, limit);
},
});
The corresponding publish is this
Meteor.publish("pages", function (skip, limit) {
return Docs.find({}, {skip: skip, limit: limit});
});
But now in the template helper I don't use the skip AFAICT because there's only limit results in the MiniMongo
Template.doclist.helpers({
docs: function () {
var route = Router.current();
var pageId = parseInt(route.params._page) || 1;
var page = pageId - 1;
var skip = page * 10;
return Docs.find({}, {
// skip: skip
limit: limit,
});
},
});
It seems to work. If I comment in the skip line then I get no results on page 2.
Is that correct or am I doing something wrong?
You are correct - the client does not require a skip in this case. Let's say you have 100 documents in the DB and you skip the first 20 with a limit of 10. Then only 10 documents will exist on the client. Whenever you find on the client (in your templates), you are querying the local database (in this case 10 documents), so a skip would be inappropriate.
I'll caution that all of this is predicated on the notion that you have only one subscription for Docs. To extend the example above, if you had another 15 documents in the same collection on the client from another subscription, then you may need to do some additional filtering in order to show only the ones you are are interested in.
Related
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)
We have use publication and subscription with skip and limit in meteor and it fetch all records at client side.
Meteor.publish('jobOrders', function(skip, limit) {
return (jobOrders.find({}, {
skip: skip,
limit: limit
}));
});
Template.templateName.onCreated(function() {
this.subscribe("jobOrders", 0, 2);
});
Template.templateName.helpers({
methodName: function() {
var totalNoOfRecords = jobOrders.find({}).count();
console.og("records are", totalNoOfRecords);
return totalNoOfRecords;
}
});
The reason could be because you have other pub-subs in other templates which are fetching the data from jobOrders collection.
Please comment your subscription in your current template and see whether you are still able to fetch the records from this collection. If you are seeing all the records even after commenting the subscription, it should be because of some other subscription.
To overcome this problem, you have to either do one of the following:
Change the other subscription which is fetching all the records such that the data is available only with in that template by using
Template.instance().subscribe()
instead of using Meteor.subscribe()which will make the data available throughout client side across all templates.
Filter you records on the client side as well.
Template.templateName.helpers({
methodName: function() {
var totalNoOfRecords = jobOrders.find({skip:0, limit: 2}).count(); //Filter on the client side
console.og("records are", totalNoOfRecords);
return totalNoOfRecords;
}
});
Note: Another possibility could be that you did not remove the insecure package which will automatically publish all the records in all the collection.
I am trying to make a fairly big meteor app, and I noticed that it has gotten slower over the past few days and I read on the meteor forum that publications can cause slow loading times. After I refresh the page when making a change in the application itself (code change), it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
if(Meteor.isServer){
Meteor.publish('notes', function () {
return Notes.find()
});
Meteor.publish('users', function () {
return Meteor.users.find()
});
Meteor.publish("user", function(){
return Meteor.user()
})
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
}
document example:
let noteInfo = { title, subject, description, imageURL, userId, userEmail, createdAt }
let title = this.refs.title.value;
let subject = this.refs.subject.value;
let description = this.refs.description.value;
let allUrls = [this.refs.imageURL.value].concat(this.state.urls);
let imageURL = allUrls.filter(function(entry) { return entry.trim() != ''; });
let userId = Meteor.userId();
let userEmail = Meteor.user().emails[0].address;
let createdAt = Date.parse(new Date());
This is a very broad performance tuning question. You haven't told us how many documents are in your collections or how large these documents are. Some possible issues:
You are over-publishing either by publishing too many documents or because your documents are too big. For example when you do:
Meteor.publish('notes', function () {
return Notes.find()
});
If there are 100,000 Notes documents at 100 bytes each then that's 10 MB that needs to go over the network to the client. This is also the case if there are 1,000 notes documents at 10KB each.
Solution: Limit the number of documents with limit and/or reduce the number of fields transmitted with fields:
Meteor.publish('notes', function () {
return Notes.find({},{ limit: 100, fields: { key1: 1, key2: 1 }});
});
Your collection is missing one or more indexes. When you do:
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
if there are 1M notes documents but there is no index on the createdAt key then this will be terribly slow.
Solution: Adding an index on the createdAt key will make such a publication much faster.
You have an invalid publication.
Meteor.publish("user", function(){
return Meteor.user();
})
Is invalid because Meteor.user() is not a cursor and a publication must return either a cursor or an array of cursors. It is also redundant because Meteor.user() is automatically available on the client, albeit it doesn't include all the keys.
Solution: Remove this unnecessary publication altogether. If you want to publish some of the keys that are not available on the client for the current user you can do so as follows:
Meteor.publish("user", function(){
return Meteor.users.find(this.userId,{fields: {services: 1, emails: 1, profile: 1}})
});
After I refresh the page when making a change, it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
What does this mean? Do you mean when you change the code of your website, it takes 1-2 minutes for your changes to appear in a newly loaded page? That's to be expected: it takes time for Meteor to rebuild your application. Watch the terminal for the progress.
Using lots of packages and external code will slow down building. The speed of compilation is unrelated to your end user's experience.
A couple things I do with my meteor projects is that I send specific fields "profile.name" etc. I've also used collection.createIndex({}) which creates an index and inside it and you could put your sort in which is created on server startup and works well with your collection.find.sort. I think limiting your notes would be quite essential as you have a limit on one and not the other.
It happened to me too. At that time I checked with kadira (formerly still opensource but now it is not). Source code at kadira on github, but service is gone kadira.io
The problem is in pubication and subscribtion. When client request subscribtions, he opens up some kind of "connection" - i say that, and it applies to others who call the same publication. imagine how many connections are formed.
where the same data accessed by many people must server side must request data to mongodb, that same to all of number of such connections. This is what makes it slow.
Finally I added redis.io.
meteor npm install --save redis
meteor npm install --save hiredis
On server side :
import { Meteor } from "meteor/meteor";
var redis = require("redis");
var clientRedis = redis.createClient({
host: "YOURREDIS_IP",
port: "YOURREDIS_PORT"
});
clientRedis.setSync = Meteor.wrapAsync(clientRedis.set);
clientRedis.getSync = Meteor.wrapAsync(clientRedis.get);
setRedis_Object = function (keyREDIS, timeRefresh, valueREDIS) {
return clientRedis.setSync(keyREDIS + moment(new Date()).format(timeRefresh), JSON.stringify(valueREDIS));
};
getRedis_Object = function (keyREDIS, timeRefresh) {
return JSON.parse(clientRedis.getSync(keyREDIS + moment(new Date()).format(timeRefresh)));
};
Meteor.methods({
getRedis_YOURCOLLECTIONS: function(timeRefresh) {
var data = getRedis_Object(getRedis_YOURCOLLECTIONS, timeRefresh);
if (data != null) {
return data;
} else {
var data = YOURCOLLECTIONS.find().fetch();
setRedis_Object(key, timeRefresh, data);
return data
}
},
});
On Client Side
Meteor.call("getRedis_YOURCOLLECTIONS", "YYYYMMDD", function (error, result) {
if (error) {
console.log(error);
} else {
Session.set("getRedis_YOURCOLLECTIONS", result);
}
});
After i am add this REDIS consumtion Memory and CPU on server low, and app more Fast. I hope its work for you. Thanks
I have angular-meteor app that needs Material md-autocomplete from a collection with 53,296 documents with angularUtils.directives.dirPagination but this amount of data make my browser hang.
I'm publishing the collection with:
Meteor.publish('city', function (options, searchString) {
var where = {
'city_name': {
'$regex': '.*' + (searchString || '') + '.*' ,
'$options': 'i'
}
};
return City.find(where, options);
});
I subscribe with:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
and have pagination on controller :
$scope.currentPage = 1;
$scope.pageSize = 100;
$scope.sort = {city_name_sort : 1};
$scope.orderProperty = '1';
$scope.helpers({
city: function(){
return City.find({});
}
});
but it takes a long time to load and its make chrome stop working.
You already have most of the server-side searching done because your search is running inside a subscription. You should make sure that the city_name field is indexed in mongo! You should only return that field to minimize data transfer. You can also simplify your regex.
Meteor.publish('city', function (searchString) {
const re = new RegExp(searchString,'i');
const where = { city_name: { $regex: re }};
return City.find(where, {sort: {city_name: 1}, fields: {city_name: 1}});
});
What I've found helps with server-side auto-complete is:
Don't start searching until the user has typed 3 or 4 characters. This drastically narrows down the search results.
Throttle the search to only run every 500ms so that you're not sending every character to the server because then it has to keep re-executing the search. If the person is typing fast the search might only run every 2 or 3 characters.
Run the same .find() on the client that you're running on the server (instead of just querying for {}). That's just good practice since the client-side collection is the union of all subscriptions on that collection, there might be documents there that you don't want to list.
Lastly I don't know why you're subscribing twice here:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
only one of those Meteor.subscribe('city') calls is necessary.
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)