When querying a DynamoDB table, you may get back a LastEvaluatedKey parameter, indicating that the result is paginated and that LastEvaluatedKey is the primary key of the last item that was processed in this page. To fetch the next page, ExclusiveStartKey should be supplied with the value of the LastEvaluatedKey from the previous page.
However, if querying through all items and inside that loop deleting some items, I'm curious as to how DynamoDB handles scenarious like ExclusiveStartKey referencing an item which was just deleted. Will fetching the next page then fail, or does DynamoDB maintain deleted keys for the purpose of concurrent queries and deletes?
Example:
Items: A, B, C, D, E, F, G, H
Query({ExclusiveStartKey: undefined}) => A, B, C, D and LastEvaluatedKey: D
Query({ExclusiveStartKey: D}) => E, F, G, H and LastEvaluatedKey: undefined
But what happens if I (or another client in a concurrent scenario) delete item D between these two statements? Will the next page be returned as E, F, G, H or will it return no items since it cannot find where it left off due to D not existing anymore?
I don't simply want to perform a quick test this to determine how it works, since there might be factors such as the key being valid for some time after being deleted, or concurrency/replication/transactional issues at play, so I'm looking for a more definitive answer. I cannot find anything in the documentation covering this case.
If I understand correctly, your concern is does DynamoDB pagination still work after you delete an item which is associated with pagination tokens (the lastEvaluatedKeys you kept). If that is the question, then pagination still works even after item is deleted. You can still use 'stale' lastEvaluatedKeys to the next pagination call, and DynamoDB is able to find out next batch of items.
Related
I have about 780K(count) items stored in DDB.
I'm calling DynamoDBMapper.query(...) method to get all of them.
The result is good, bcs I can get all of the items. But it cost me 3min to get them.
From the log, I see the DynamoDBMapper.query(...) method is trying to get items page by page, each page will request an individual query call to DDB which will cost about 0.7s for each page.
I counted that all items returned with 292 pages, so the total duration is about 0.7*292=200s which is unacceptable.
My code is basically like below:
// setup query condition, after filter the items count would be about 780K
DynamoDBQueryExpression<VendorAsinItem> expression = buildFilterExpression(filters, expression);
List<VendorAsinItem> results = new ArrayList<>();
try {
log.info("yrena:Start query");
DynamoDBMapperConfig config = getTableNameConfig();
results = getDynamoDBMapper().query( // get DynamoDBMapper instance and call query method
VendorAsinItem.class,
expression,
config);
} catch (Exception e) {
log.error("yrena:Error ", e);
}
log.info("yrena:End query. Size:" + results.size());
So how can I get all items at once without pagination.
My final goal is to reduce the query duration.
EDIT Just re-read the title of the question and realized that perhaps I didn't address the question head on: there is no way to retrieve 780,000 items without some pagination because of a hard limit of 1MB per page
Long form answer
780,000 items retrieved, in 3 minutes, using 292 pages: that's about 1.62 pages per second.
Take a moment and let that sync in..
Dynamo can return 1MB of data per page, so you're presumably transferring 1.5MB of data per second (that will saturate a 10 Mbit pipe).
Without further details about (a) the actual size of the items retrieved; (b) the bandwidth of your internet connection; (c) the number of items that might get filtered out of query results and (d) the provisioned read capacity on the table I would start looking at:
what is the network bandwidth between your client and Dynamo/AWS -- if you are not maxing that out, then move on to next;
how much read capacity is provisioned on the table (if you see any throttling on the requests, you may be able to increase RCU on the table to get a speed improvement at a monetary expense)
the efficiency of your query:
if you are applying filters, know that those are applied after query results are generated and so the query is consuming RCU for stuff that gets filtered out and that also means the query is inefficient
think about whether there are ways you can optimize your queries to access less data
Finally 780,000 items is A LOT for a query -- what percentage of items in the database is that?
Could you create a secondary index that would essentially contain most, or all of that data that you could then simply scan instead of querying?
Unlike a query, a scan can be parallelized so if your network bandwidth, memory and local compute are large enough, and you're willing to provision enough capacity on the database you could read 780,000 items significantly faster than a query.
I have a graph database storing different types of entities and I am building an API to fetch entities from the graph. It is however a bit more complicated since for each type of entity there is a set of rules that are used for fetching related entities as well as the original.
To do this I used an aggregate step to aggregate all related entities that I am fetching into a collection.
An added requirement is to fetch a batch of entities (and related entities). I was going to do this by changing the has step that is fetching entities to use P.within and map the aggregation to each of the found entities.
This works if I continue fetching a single entity, but if I want to fetch two then my result set will be correct for the first one but the result set for the second contains the results of the first one as well as its own results.
I think this is because second one will simply add to the aggregated collection from the first one since the aggregation key is the same.
I haven't found any way to clear the collection between the first and the second, nor any way to have a dynamic aggregation side effect key.
Code:
return graph.traversal().V()
.hasLabel(ENTITY_LABEL)
.has("entity_ref", P.within(entityRefs)) // entityRefs is a list of entities I am looking for
.flatMap(
__.aggregate("match")
.sideEffect(
// The logic that applies the rules lives here. It will add items to "match" collection.
)
.select("match")
.fold()
)
.toStream()
...
The result should be a list of lists of entities where the first list of entities in the outer list contains results for the first entity in entityRefs, and the second list of entities contains results for the second entity in entityRefs.
Example:
I want to fetch the vertices for entity refs A and B and their related entities.
Let's say I expect the results to then be [[A, C], [B, D, E]], but I get the results [[A, C], [A, C, B, D, E]] (The second results contain the results from the first one).
Questions:
Is there a way to clear the "match" collection after the selection?
Is there a way to have dynamic side effect keys such that I create a collection for each entityRef?
Is there perhaps a different way I can do this?
Have I misidentified the problem?
EDIT:
This is an example that is a miniature version of the problem. The graph is setup like so:
g.addV('entity').property('id',1).property('type', 'customer').as('1').
addV('entity').property('id',2).property('type', 'email').as('2').
addV('entity').property('id',6).property('type', 'customer').as('6').
addV('entity').property('id',3).property('type', 'product').as('3').
addV('entity').property('id',4).property('type', 'subLocation').as('4').
addV('entity').property('id',7).property('type', 'location').as('7').
addV('entity').property('id',5).property('type', 'productGroup').as('5').
addE('AKA').from('1').to('2').
addE('AKA').from('2').to('6').
addE('HOSTED_AT').from('3').to('4').
addE('LOCATED_AT').from('4').to('7').
addE('PART_OF').from('3').to('5').iterate()
I want to fetch a batch of entities, given their ids and fetch related entities. Which related entities should be returned is a function of the type of the original entity.
My current query is like this (slightly modified for this example):
g.V().
hasLabel('entity').
has('id', P.within(1,3)).
flatMap(
aggregate('match').
sideEffect(
choose(values('type')).
option('customer',
both('AKA').
has('type', P.within('email', 'phone')).
sideEffect(
has('type', 'email').
aggregate('match')).
both('AKA').
has('type', 'customer').
aggregate('match')).
option('product',
bothE('HOSTED_AT', 'PART_OF').
choose(label()).
option('PART_OF',
bothV().
has('type', P.eq('productGroup')).
aggregate('match')).
option('HOSTED_AT',
bothV().
has('type', P.eq('subLocation')).
aggregate('match').
both('LOCATED_AT').
has('type', P.eq('location')).
aggregate('match')))
).
select('match').
unfold().
dedup().
values('id').
fold()
).
toList()
If I only fetch for one entity i get correct results. For id: 1 I get [1,2,6] and for id: 3 I get [3,5,4,7]. However when i fetch for both I get:
==>[3,5,4,7]
==>[3,5,4,7,1,2,6]
The first result is correct, but the second contains the results for both ids.
You can leverage the (not too well documented to be honest but seemingly powerful traversal step) group().by(key).by(value).
That way you can drop the aggregate() side effect step that is causing you trouble. As an alternative to collect multiple vertices matching some traversal into a list I used union().
An example that uses the graph you posted(I only included the Customer option for brevity):
g.V().
hasLabel('entity').
has('id', P.within(1,3)).
<String, List<Entity>>group()
.by("id")
.by(choose(values("type"))
.option('customer', union(
identity(),
both('AKA').has('type', 'email'),
both('AKA').has('type', within('email', 'phone')).both('AKA').has('type', 'customer'))
.map((traversal) -> new Entity(traversal.get())) //Or whatever business class you have
.fold() //This is important to collect all 3 paths in the union together
.option('product', union()))
.next()
This traversal has the obvious drawback of the code being a bit more verbose. It declares it will step over the 'AKA' from a Customer twice. Your traversal only declared it once.
It does however keep the by(value) part of the group() step separate between different keys. Which is what we wanted.
At first sight, it's clear what the continuation token does in Cosmos DB: attaching it to the next query gives you the next set of results. But what does "next set of results" mean exactly?
Does it mean:
the next set of results as if the original query had been executed completely without paging at the time of the very first query (skipping the appropriate number of documents)?
the next set of results as if the original query had been executed now (skipping the appropriate number of documents)?
Something completely different?
Answer 1. would seem preferable but unlikely given that the server would need to store unlimited amounts of state. But Answer 2. is also problematic as it may result in inconsistencies, e.g. the same document may be served multiple times across pages, if the underlying data has changed between the page queries.
Cosmos DB query executions are stateless at the server side. The continuation token is used to recreate the state of the index and track progress of the execution.
"Next set of results" means, the query is executed again on from a "bookmark" from the previous execution. This bookmark is provided by the continuation token.
Documents created during continuations
They may or may not be returned depending on the position of insert and query being executed.
Example:
SELECT * FROM c ORDER BY c.someValue ASC
Let us assume the bookmark had someValue = 10, the query engine resumes processing using a continuation token where someValue = 10.
If you were to insert a new document with someValue = 5 in between query executions, it will not show up in the next set of results.
If the new document is inserted in a "page" that is > the bookmark, it will show up in next set of results
Documents updated during continuations
Same logic as above applies to updates as well
(See #4)
Documents deleted during continuations
They will not show up in the next set of results.
Chances of duplicates
In case of the below query,
SELECT * FROM c ORDER BY c.remainingInventory ASC
If the remainingInventory was updated after the first set of results and it now satisfies the ORDER BY criteria for the second page, the document will show up again.
Cosmos DB doesn’t provide snapshot isolation across query pages.
However, as per the product team this is an incredibly uncommon scenario because queries over continuations are very quick and in most cases all query results are returned on the first page.
Based on preliminary experiments, the answer seems to be option #2, or more precisely:
Documents created after serving the first page are observable on subsequent pages
Documents updated after serving the first page are observable on subsequent pages
Documents deleted after serving the first page are omitted on subsequent pages
Documents are never served twice
The first statement above contradicts information from MSFT (cf. Kalyan's answer). It would be great to get a more qualified answer from the Cosmos DB Team specifying precisely the semantics of retrieving pages. This may not be very important for displaying data in the UI, but may be essential for data processing in the backend, given that there doesn't seem to be any way of disabling paging when performing a query (cf. Are transactional queries possible in Cosmos DB?).
Experimental method
I used Sacha Bruttin's Cosmos DB Explorer to query a collection with 5 documents, because this tool allows playing around with the page size and other request options.
The page size was set to 1, and Cross Partition Queries were enabled. Different queries were tried, e.g. SELECT * FROM c or SELECT * FROM c ORDER BY c.name.
After retrieving page 1, new documents were inserted, and some existing documents (including documents that should appear on subsequent pages) were updated and deleted. Then all subsequent pages were retrieved in sequence.
(A quick look at the source code of the tool confirmed that ResponseContinuationTokenLimitInKb is not set.)
I'm working on a website that mostly displays items created by registered users. So I'd say 95% of API calls are to read a single item and 5% are to store a single item. System is designed with AWS API Gateway that calls AWS Lambda function which manipulates data in DynamoDB.
My next step is to implement voting system (upvote/downvote) with basic fetaures:
Each registered user can vote only once per item, and later is only allowed to change that vote.
number of votes needs to be displayed to all users next to every item.
items have only single-item views, and are (almost) never displayed in a list view.
only list view I need is "top 100 items by votes" but it is ok to calculate this once per day and serve cached version
My goal is to design a database/lambda to minimize costs of AWS. It's easy to make the logic work but I'm not sure if my solution is the optimal one:
My items table currently has hashkey slug and sortkey version
I created items-votes table with hashkey slug and sortkey user and also voted field (containing -1 or 1)
I added field votes to items table
API call to upvote/downvote inserts to item-votes table but before checks constraints that user has not already voted that way. Then in second query updates items table with updated votes count. (so 1 API call and 2 db queries)
old API call to show an item stays the same but grabs new votes count too (1 API call and 1 db query)
I was wondering if this can be done even better with avoiding new items-votes table and storing user votes inside items table? It looks like it is possible to save one query that way, and half the lambda execution time but I'm worried it might make that table too big/complex. Each user field is a 10 chars user ID so if item gets thousands of votes I'm not sure how Lambda/DynamoDB will behave compared to original solution.
I don't expect thousands of votes any time soon, but it is not impossible to happen to a few items and I'd like to avoid situation where I need to migrate to different solution in the near future.
I would suggest to have a SET DynamoDB (i.e. SS) attribute to maintain the list of users who voted against the item. Something like below:-
upvotes : ['user1', 'user2']
downvotes : ['user1', 'user2']
When you update the votes using UpdateExpression, you can use ADD operator which adds users to SET only if it doesn't exists.
ADD - Adds the specified value to the item, if the attribute does not
already exist. If the attribute does exist, then the behavior of ADD
depends on the data type of the attribute:
If the existing data type is a set and if Value is also a set, then
Value is added to the existing set. For example, if the attribute
value is the set [1,2], and the ADD action specified [3], then the
final attribute value is [1,2,3]. An error occurs if an ADD action is
specified for a set attribute and the attribute type specified does
not match the existing set type. Both sets must have the same
primitive data type. For example, if the existing data type is a set
of strings, the Value must also be a set of strings.
This way you don't need to check whether the user already upvote or downvote for the item or not.
Only thing you may need to ensure is that the same user shouldn't be present on upvote and downvote set. Probably, you can use REMOVE or ConditionExpression to achieve this.
From the Transactions doc, second paragraph:
The intention here is for the client to increment the total number of
chat messages sent (ignore for a moment that there are better ways of
implementing this).
What are some standard "better ways" of implementing this?
Specifically, I'm looking at trying to do things like retrieve the most recent 50 records. This requires that I start from the end of the list, so I need a way to determine what the last record is.
The options as I see them:
use a transaction to update a counter each time a record is added, use the counter value with setPriority() for ordering
forEach() the parent and read all records, do my own sorting/filtering at client
write server code to analyze Firebase tables and create indexed lists like "mostRecent Messages" and "totalNumberOfMessages"
Am I missing obvious choices?
To view the last 50 records in a list, simply call "limit()" as shown:
var data = new Firebase(...);
data.limit(50).on(...);
Firebase elements are ordering first by priority, and if priorities match (or none is set), lexigraphically by name. The push() command automatically creates elements that are ordered chronologically, so if you're using push(), then no additional work is needed to use limit().
To count the elements in a list, I would suggest adding a "value" callback and then iterating through the snapshot (or doing the transaction approach we mention). The note in the documentation actually refers to some upcoming features we haven't released yet which will allow you to count elements without loading them first.