I would like to run a query in meteor and limit the number of field returned to only 5. Here is my code :
var courses = Courses.find(
{ day_of_week : {$in: day_selector},
price : {$gt : price_min, $lt : price_max},
starts : {$gt : schedule_min},
ends : {$lt : schedule_max}},
{limit : 10});
console.log(courses);
return courses;
However when I do this, I get all the courses that fit the selector in the console log and not only 10 of those. In the template everything is fine and only 10 courses are displayed.
I looked at this question :
Limit number of results in Meteor on the server side?
but it did not help as I am not using specifics _id fields for my courses, I am using specific _id fields but for other collections though.
Currently, the server is sending over your entire collection of courses and you are just filtering them to 10 on the client side. You can actually create a reactive subscription/publication to set the limit dynamically, or you could just limit the amount of records sent on the server
Meteor.publish('courses', function(limit) {
//default limit if none set
var dl = limit || 10;
return Posts.find({}, {limit: dl});
});
Meteor.subscribe('courses', Session.get('limit'));
And then set the limit dynamically by using an event that calls:
Session.set('limit', 5);
Related
I am writing Azure Cosmos DB stored procedure in azure portal , I am checking the what is max size response it can give initially it gave me 4.8 MB response later on any size more that it was showing error of message size is too large . Later on after 5 min executed same procedure and same parameters it was showing error of too large page size.
// SAMPLE STORED PROCEDURE
function sample(prefix) {
var collection = getContext().getCollection();
// Query documents and take 1st item.
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT r.data.truckSerialNo FROM root r',{pageSize : -1},
function (err, feed, options) {
if (err) throw err;
// Check the feed and if empty, set the body to 'no docs found',
// else take 1st element from feed
if (!feed || !feed.length) {
var response = getContext().getResponse();
response.setBody('no docs found');
}
else {
console.log(feed.length);
var response = getContext().getResponse();
var body = { prefix: prefix, feed: feed };
var str = {};
str.x1 = body ;
str.x2 = body ;
response.setBody(JSON.stringify(str));
}
});
if (!isAccepted) throw new Error('The query was not accepted by the server.');
}
and error
Failed to execute stored procedure testProcedure for collection
iotcollection:
{"code":400,"body":"{\"code\":\"BadRequest\",\"message\":\"Message:
{\\"Errors\\":[\\"Encountered exception while executing function.
Exception = Error: Resulting message would be too large because of
\\\\"Body\\\\". Return from script with current message and use
continuation token to call the script again or modify your
script.\\r\\nStack trace: Error: Resulting message would be too
large because of \\\\"Body\\\\". Return from script with current
message and use continuation token to call the script again or modify
your script.\\n at validateSize (bulkUpsert_v_1_0.js:177:25)\\n
at setValueInternal (bulkUpsert\\"]}\r\nActivityId:
d61226ab-4d5e-4a3d-aa09-83351649ce4c, Request URI:
/apps/59d3b9ef-17ca-4bbf-8a11-39d0199a8d29/services/1b26e00f-1f51-4d34-88ec-4090b8e7db00/partitions/45a313b7-2cf2-419e-9885-48bf9cfe6277/replicas/131862936473830809p/,
RequestStats: \r\nRequestStartTime: 2018-11-10T06:41:26.3320532Z,
Number of regions attempted: 1\r\n, SDK:
Microsoft.Azure.Documents.Common/2.1.0.0\"}","activityId":"d61226ab-4d5e-4a3d-aa09-83351649ce4c","substatus":413}
Per my experience, {pageSize : -1} setting is not recommanded strongly when your data is too huge in cosmos db query operation. Continuation token(namely x-ms-continuation-token in cosmos db rest api) need to be considered into your business. Please refer to my previous case:CosmosDB stored procedure not returning all documents.
As you said, later on after 5 min executed same procedure and same parameters it was showing error of too large page size. The size limit is total response which including other Response Headers,not only the body part. In addition, the query operation returns maximum results based on current performance if you set {pageSize : -1}. It's not fixed.
I have a Google drive table data source which stores list of open positions. Now in the data source I've set "Query per size" field to 10 so that I can get 10 records per page. I've added a Pager as well to show pagination.
My query is I want to display like "Page 1 of X" to my end users and this X will vary based on certain search filters. What will the best way to achieve this in Appmaker?
I've tried counting total records in a data source as per below code but every time updating that with the search criteria and recounting it is not a proper solution.
//Server side
var newQuery = app.models.Company.newQuery();
var records = newQuery.run();
var totalCount =0;
for(var i=0;i<records.length;i++)
{
totalCount=totalCount+1;
}
return totalCount;
In case you don't have any filters in your table your server code can be as simple as
// Server script
function getPagesCount(pageSize) {
var recordsCount = app.models.MyModel.newQuery().run().length;
var pagesCount = Math.ceil(recordsCount / pageSize);
return pagesCount;
}
As an alternative you can consider creating Calculated Model with a single field PagesCount.
In case you have some filters associated with the table then you'll need to run the query for the pages number with exact same filters.
Most likely the entire setup will not work effectively with Drive Tables since there is no way to query records number without querying records themselves. With Cloud SQL data backend one can create Calculated SQL Model with lightweight native SQL query (here :PageSize is query parameter which should be equal to the query.limit of the actual datasource):
SELECT
Ceil(COUNT(1) / :PageSize) AS RecordsNumber
FROM
TableName
WHERE
...
I've achieved this using Calculated Model as suggested by Pavel.
Steps :
Create a calculated data source with one field count.
In that data source add one parameter searchQuery. This will contain users filter going forward. Currently I have only one search query in which user can search many things. So I've added one parameter only.
In this data source add following server script.
Code:
// Server script
function getTotalRecords(query) {
var receivedQuery = query.parameters.searchQuery;
// console.log('Received query:' + query.parameters.searchQuery);
var records = app.models.Company.newQuery();
records.parameters.SearchText = query.parameters.searchQuery;
if(receivedQuery !== null) {
records.where = '(Name contains? :SearchText or InternalId contains? ' +
':SearchText or LocationList contains? :SearchText )';
}
var recordsCount = records.run().length;
var calculatedModelRecords = [];
var draftRecord = app.models.RecordCount.newRecord();
draftRecord.count = ''+recordsCount;
calculatedModelRecords.push(draftRecord);
return calculatedModelRecords;
}
.
On the Appmaker page bind a label with this data source.
On search query/your filter applied event add following code which Reload this data source and assign value to Parameter.
// Client script
function updateRecordCount(newValue) {
var ds = app.datasources.RecordCount;
ds.query.parameters.searchQuery = newValue;
ds.unload();
ds.load();
}
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)
I am implementing a website with a dynamodb + nodejs backend. I use Vogels.js in server side to query dynamodb and show results on a webpage. Because my query returns a lot of results, I would like to return only N (such as 5) results back to a user initially, and return the next N results when the user asks for more.
Is there a way I can run two vogels queries with the second query starts from the place where the first query left off ? Thanks.
Yes, vogels fully supports pagination on both query and scan operations.
For example:
var Tweet = vogels.define('tweet', {
hashKey : 'UserId',
rangeKey : 'PublishedDateTime',
schema : {
UserId : Joi.string(),
PublishedDateTime : Joi.date().default(Date.now),
content : Joi.string()
}
});
// Fetch the 5 most recent tweets from user with id 555:
Tweet.query(555).limit(5).descending().exec(function (err, data) {
var paginationKey = data.LastEvaluatedKey;
// Fetch the next page of 5 tweets
Tweet.query(555).limit(5).descending().startKey(paginationKey).exec()
});
Yes it is possible, DynamoDB has some thing called "LastEvaluatedKey" which will server your purpose.
Step 1) Query your table with option "Limit" = number of records
refer: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
Step 2) If your query has more records than the "Limit value", DynamoDB will return a "LastEvaluatedKey" which you can pass in your next query as "ExclusiveStartKey" to get next set of records until there are no records left
refer: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScan.Query
Note: Be aware that to get previous set of records you might have to store all the "LastEvaluatedKeys" and implement this at application level