I am implementing a website with a dynamodb + nodejs backend. I use Vogels.js in server side to query dynamodb and show results on a webpage. Because my query returns a lot of results, I would like to return only N (such as 5) results back to a user initially, and return the next N results when the user asks for more.
Is there a way I can run two vogels queries with the second query starts from the place where the first query left off ? Thanks.
Yes, vogels fully supports pagination on both query and scan operations.
For example:
var Tweet = vogels.define('tweet', {
hashKey : 'UserId',
rangeKey : 'PublishedDateTime',
schema : {
UserId : Joi.string(),
PublishedDateTime : Joi.date().default(Date.now),
content : Joi.string()
}
});
// Fetch the 5 most recent tweets from user with id 555:
Tweet.query(555).limit(5).descending().exec(function (err, data) {
var paginationKey = data.LastEvaluatedKey;
// Fetch the next page of 5 tweets
Tweet.query(555).limit(5).descending().startKey(paginationKey).exec()
});
Yes it is possible, DynamoDB has some thing called "LastEvaluatedKey" which will server your purpose.
Step 1) Query your table with option "Limit" = number of records
refer: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
Step 2) If your query has more records than the "Limit value", DynamoDB will return a "LastEvaluatedKey" which you can pass in your next query as "ExclusiveStartKey" to get next set of records until there are no records left
refer: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScan.Query
Note: Be aware that to get previous set of records you might have to store all the "LastEvaluatedKeys" and implement this at application level
Related
Post table
{
...otherPostFields,
tags: string[]
}
User table
{
...otherUserFields,
tags: string[]
}
I am trying to make a feed
I am first fetching User to get the tags
I don't want to use scan since its very expensive as it goes through all the records in the table.
Any better approach?
Once I have the user tags I use scan operation on Post table
const { tags } = Items[0] as IUser & Pick<CUser, 'tags'>;
const ExpressionAttributeValues = tags.reduce<Record<string, string>>((acc, tag, index) => {
acc[`:tags${index}`] = tag;
return acc;
}, {});
const FilterExpression = tags.reduce<string>((acc, _, index) => {
if (index === 0) return `contains(tags, :tags${index})`;
return `${acc} OR contains(tags, :tags${index})`;
}, '');
// expensive operation
const { Items: posts } = await client
.scan({
TableName: PostsTable.get(),
FilterExpression,
Limit: 10,
ExpressionAttributeValues,
})
.promise();
You didn't state the schema of your DynamoDB table nor which information you have before you make a read so it's difficult to help you.
However, to answer your question in short, you are not doing an expensive read as you are setting Limit=10 which will consume 5 RCU per request. If requests are infrequent (less than 5 times per second) you still stay within DynamoDBs free tier of 25 RCU.
Update
I am trying to make a feed I am first fetching User to get the tags
Why not use a Query as you are trying to get a single users tag it seems.
One thing that I noticed is the above query does not return any document when the table has over 100k items. Why is that happening?
This is because DynamoDB only returns up to 1MB per API call, if you require more than 1MB the you must paginate.
A single Query operation will read up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression. If LastEvaluatedKey is present in the response, you will need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide.
Imagine the following function which is querying a GlobalSecondaryIndex and associated Range Key in order to find a limited number of results:
#Override
public List<Statement> getAllStatementsOlderThan(String userId, String startingDate, int limit) {
if(StringUtils.isNullOrEmpty(startingDate)) {
startingDate = UTC.now().toString();
}
LOG.info("Attempting to find all Statements older than ({})", startingDate);
Map<String, AttributeValue> eav = Maps.newHashMap();
eav.put(":userId", new AttributeValue().withS(userId));
eav.put(":receivedDate", new AttributeValue().withS(startingDate));
DynamoDBQueryExpression<Statement> queryExpression = new DynamoDBQueryExpression<Statement>()
.withKeyConditionExpression("userId = :userId and receivedDate < :receivedDate").withExpressionAttributeValues(eav)
.withIndexName("userId-index")
.withConsistentRead(false);
if(limit > 0) {
queryExpression.setLimit(limit);
}
List<Statement> statementResults = mapper.query(Statement.class, queryExpression);
LOG.info("Successfully retrieved ({}) values", statementResults.size());
return statementResults;
}
List<Statement> results = statementRepository.getAllStatementsOlderThan(userId, UTC.now().toString(), 5);
assertThat(results.size()).isEqualTo(5); // NEVER passes
The limit isn't respected whenever I query against the database. I always get back all results that match my search criteria so if I set the startingDate to now then I get every item in the database since they're all older than now.
You should use queryPage function instead of query.
From DynamoDBQueryExpression.setLimit documentation:
Sets the maximum number of items to retrieve in each service request
to DynamoDB.
Note that when calling DynamoDBMapper.query, multiple
requests are made to DynamoDB if needed to retrieve the entire result
set. Setting this will limit the number of items retrieved by each
request, NOT the total number of results that will be retrieved. Use
DynamoDBMapper.queryPage to retrieve a single page of items from
DynamoDB.
As they've rightly answered the setLimit or withLimit functions limit the number of records fetched only in each particular request and internally multiple requests take place to fetch the results.
If you want to limit the number of records fetched in all the requests then you might want to use "Scan".
Example for the same can be found here
I would like to run a query in meteor and limit the number of field returned to only 5. Here is my code :
var courses = Courses.find(
{ day_of_week : {$in: day_selector},
price : {$gt : price_min, $lt : price_max},
starts : {$gt : schedule_min},
ends : {$lt : schedule_max}},
{limit : 10});
console.log(courses);
return courses;
However when I do this, I get all the courses that fit the selector in the console log and not only 10 of those. In the template everything is fine and only 10 courses are displayed.
I looked at this question :
Limit number of results in Meteor on the server side?
but it did not help as I am not using specifics _id fields for my courses, I am using specific _id fields but for other collections though.
Currently, the server is sending over your entire collection of courses and you are just filtering them to 10 on the client side. You can actually create a reactive subscription/publication to set the limit dynamically, or you could just limit the amount of records sent on the server
Meteor.publish('courses', function(limit) {
//default limit if none set
var dl = limit || 10;
return Posts.find({}, {limit: dl});
});
Meteor.subscribe('courses', Session.get('limit'));
And then set the limit dynamically by using an event that calls:
Session.set('limit', 5);
I have a requirement for to show the search result on the jsp with maxcount of 10 and it should have a pagination to traverse back and forward as pagination functionality.
Dynamodb has a lastevaluatedkey, but it doesn't help to go back to the previous page, though I can move to the next result set by the lastevaluatedKey.
Can anybody please help on this.
I am using Java SPRING and DynamoDB as the stack.
Thanks
Satya
To enable forward/backward, all you need is to keep
the first key, which is hash key + sort key of the first record of the previously returned page (null if you are about to query the first page).
and
the last key of the retrieved page, which is hash key + sort key of the last record of the previously returned page
Then to navigate forward or backward, you need to pass in below parameters in the query request:
Forward: last key as the ExclusiveStartKey, order = ascend
Backward: first key as the ExclusiveStartKey, order = descend
I have achieved this in a project in 2016. DynamoDB might provide some similar convenient APIs now, but I'm not sure as I haven't checked DynamoDB for a long time.
Building on Ray's answer, here's what I did. sortId is the sort key.
// query a page of items and create prev and next cursor
// cursor idea from this article: https://hackernoon.com/guys-were-doing-pagination-wrong-f6c18a91b232
async function queryCursor( cursor) {
const cursor1 = JSON.parse(JSON.stringify(cursor));
const pageResult = await queryPage( cursor1.params, cursor1.pageItems);
const result = {
Items: pageResult.Items,
Count: pageResult.Count
};
if ( cursor.params.ScanIndexForward) {
if (pageResult.LastEvaluatedKey) {
result.nextCursor = JSON.parse(JSON.stringify(cursor));
result.nextCursor.params.ExclusiveStartKey = pageResult.LastEvaluatedKey;
}
if ( cursor.params.ExclusiveStartKey) {
result.prevCursor = JSON.parse(JSON.stringify(cursor));
result.prevCursor.params.ScanIndexForward = !cursor.params.ScanIndexForward;
result.prevCursor.params.ExclusiveStartKey.sortId = pageResult.Items[0].sortId;
}
} else {
if (pageResult.LastEvaluatedKey) {
result.prevCursor = JSON.parse(JSON.stringify(cursor));
result.prevCursor.params.ExclusiveStartKey = pageResult.LastEvaluatedKey;
}
if ( cursor.params.ExclusiveStartKey) {
result.nextCursor = JSON.parse(JSON.stringify(cursor));
result.nextCursor.params.ScanIndexForward = !cursor.params.ScanIndexForward;
result.nextCursor.params.ExclusiveStartKey.sortId = pageResult.Items[0].sortId;
}
}
return result;
}
You will have to keep a record of the previous key in a session var, query string, or something similar you can access later, then execute the query using that key when you want to go backwards. Dynamo does not keep track of that for you.
For a simple stateless forward and reverse navigation with dynamodb check out my answer to a similar question: https://stackoverflow.com/a/64179187/93451.
In summary it returns the reverse navigation history in each response, allowing the user to explicitly move forward or back until either end.
GET /accounts -> first page
GET /accounts?next=A3r0ijKJ8 -> next page
GET /accounts?prev=R4tY69kUI -> previous page
I am working on an application which lists and searches document from alfresco. The issue is the alfresco can return upto 5000 records per query. but I don't want my application to list down all documents instead if I can some how implement pagination in alfresco, so that alfresco only return X result per page. I am using Alfresco 4 enterprise edition.
Any help or suggestion please.
UPDATE (Example)
I have written a web script which executes the query and returns all the documents satisfies the condition. Lets say, there are 5000 entries found. I want to modify my web script in a way that the web script returns 100 documents for 1st page, next 100 for second page and so on...
It'll be something like usage of Limit BY and OFFSET keywords. something like this
There are two ways to query on the SearchService (excluding the selectNodes/selectProperties calls). One way is to specify all your arguments directly to the query method. This has the advantage of being concise, but the disadvantage is that you don't get all the options.
Alternately, you can query with a SearchParameters object. This lets you do everything the simple query does, and more. Included in that more are setLimit, setSkipCount and setMaxItems, which will allow you to do your paging.
If your query used to be something like:
searchService.query(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE, "lucene", myQuery);
You'd instead do something like:
SearchParameters sp = new SearchParameters();
sp.addStore(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
sp.setLanguage("lucene");
sp.setQuery(myQuery);
sp.setMaxItems(100);
sp.setSkipCount(900);
searchService.query(sp);
Assuming you have written your webscript in Javascript you can use the search.query() function and add the page property to the search definition as shown below:
var sort1 = {
column: "#{http://www.alfresco.org/model/content/1.0}modified",
ascending: false
};
var sort2 = {
column: "#{http://www.alfresco.org/model/content/1.0}created",
ascending: false
};
var paging = {
maxItems: 100,
skipCount: 0
};
var def = {
query: "cm:name:test*",
store: "workspace://SpacesStore",
language: "fts-alfresco",
sort: [sort1, sort2],
page: paging
};
var results = search.query(def);
You can find more information here: http://wiki.alfresco.com/wiki/4.0_JavaScript_API#Search_API