I'm creating an API based on Cosmos DB and ASP.NET Core 3.0. Using the Cosmos DB 4.0 preview 1 .NET Core SDK. I implemented paging using the OFFSET and LIMIT clause. I'm seeing the RU charge increase significantly the higher in the page count you go. Example for a page size of 100 items:
Page 1: 9.78 RU
Page 10: 37.28 RU
Page 100: 312.22 RU
Page 500: 358.68 RU
The queries are simply:
SELECT * from c OFFSET [page*size] LIMIT [size]
Am I doing something wrong, or is this expected? Does OFFSET require scanning the entire logical partition? I'm querying against a single partition key with about 10000 items in the partition. It seems like the more items in the partition, the worse the performance gets. (See also comment by "Russ" in the uservoice for this feature).
Is there a better way to implement efficient paging through the entire partition?
Edit 1: Also, I notice doing queries in the Cosmos Emulator also slow waaayyy down when doing OFFSET/LIMIT in a partition with 10,000 items.
Edit 2: Here is my repository code for the query. Essentially, it is wrapping the Container.GetItemQueryStreamIterator() method and pulling out the RU while processing IAsyncEnumerable. The query itself is the SQL string above, no LINQ or other mystery there.
public async Task<RepositoryPageResult<T>> GetPageAsync(int? page, int? pageSize, EntityFilters filters){
// Enforce default page and size if null
int validatedPage = GetValidatedPageNumber(page);
int validatedPageSize = GetValidatedPageSize(pageSize);
IAsyncEnumerable<Response> responseSet = cosmosService.Container.GetItemQueryStreamIterator(
BuildQuery(validatedPage, validatedPageSize, filters),
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(ResolvePartitionKey())
});
var pageResult = new RepositoryPageResult<T>(validatedPage, validatedPageSize);
await foreach (Response response in responseSet)
{
LogResponse(response, COSMOS_REQUEST_TYPE_QUERY_ITEMS); // Read RU charge
if (response.Status == STATUS_OK && response.ContentStream != null)
{
CosmosItemStreamQueryResultSet<T> responseContent = await response.ContentStream.FromJsonStreamAsync<CosmosItemStreamQueryResultSet<T>>();
pageResult.Entities.AddRange(responseContent.Documents);
foreach (var item in responseContent.Documents)
{
cache.Set(item.Id, item); // Add each item to cache
}
}
else
{
// Unexpected status. Abort processing.
return new RepositoryPageResult<T>(false, response.Status, message: "Unexpected response received while processing query response.");
}
}
pageResult.Succeeded = true;
pageResult.StatusCode = STATUS_OK;
return pageResult;
}
Edit 3:
Running the same raw SQL from cosmos.azure.com, I noticed in query stats:
OFFSET 0 LIMIT 100: Output document count = 100, Output document size = 44 KB
OFFSET 9900 LIMIT 100: Output document count = 10000, Output document size = 4.4 MB
And indeed, inspecting the network tab in browser reveals 100 separate HTTP queries, each retrieving 100 documents! So OFFSET appears to be currently not at the database, but at the client, which retrieves EVERYTHING before throwing away the first 99 queries worth of data. This can't be the intended design? Isn't the query supposed to tell the database to return only 100 items total, in 1 response, not all 10000 so the client can throw away 9900?
Based on the code it would mean that the client is skipping the documents and thus the increase of RUs.
I tested the same scenario on the browser (cosmos.azure.com, uses the JS SDK) and the behavior is the same, as offset moves, the RU increases.
It is documented here in the official documentation, under remarks https://learn.microsoft.com/en-us/azure/cosmos-db/sql-query-offset-limit
The RU charge of a query with OFFSET LIMIT will increase as the number of terms being offset increases. For queries that have multiple pages of results, we typically recommend using continuation tokens. Continuation tokens are a "bookmark" for the place where the query can later resume. If you use OFFSET LIMIT, there is no "bookmark". If you wanted to return the query's next page, you would have to start from the beginning.
Related
Post table
{
...otherPostFields,
tags: string[]
}
User table
{
...otherUserFields,
tags: string[]
}
I am trying to make a feed
I am first fetching User to get the tags
I don't want to use scan since its very expensive as it goes through all the records in the table.
Any better approach?
Once I have the user tags I use scan operation on Post table
const { tags } = Items[0] as IUser & Pick<CUser, 'tags'>;
const ExpressionAttributeValues = tags.reduce<Record<string, string>>((acc, tag, index) => {
acc[`:tags${index}`] = tag;
return acc;
}, {});
const FilterExpression = tags.reduce<string>((acc, _, index) => {
if (index === 0) return `contains(tags, :tags${index})`;
return `${acc} OR contains(tags, :tags${index})`;
}, '');
// expensive operation
const { Items: posts } = await client
.scan({
TableName: PostsTable.get(),
FilterExpression,
Limit: 10,
ExpressionAttributeValues,
})
.promise();
You didn't state the schema of your DynamoDB table nor which information you have before you make a read so it's difficult to help you.
However, to answer your question in short, you are not doing an expensive read as you are setting Limit=10 which will consume 5 RCU per request. If requests are infrequent (less than 5 times per second) you still stay within DynamoDBs free tier of 25 RCU.
Update
I am trying to make a feed I am first fetching User to get the tags
Why not use a Query as you are trying to get a single users tag it seems.
One thing that I noticed is the above query does not return any document when the table has over 100k items. Why is that happening?
This is because DynamoDB only returns up to 1MB per API call, if you require more than 1MB the you must paginate.
A single Query operation will read up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression. If LastEvaluatedKey is present in the response, you will need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide.
I read the article here about IAsyncEnumerable, more specifically towards a Cosmos Db-datasource
public async IAsyncEnumerable<T> Get<T>(string containerName, string sqlQuery)
{
var container = GetContainer(containerName);
using FeedIterator<T> iterator = container.GetItemQueryIterator<T>(sqlQuery);
while (iterator.HasMoreResults)
{
foreach (var item in await iterator.ReadNextAsync())
{
yield return item;
}
}
}
I am wondering how the CosmosDB is handling this, compared to paging, lets say 100 documents at the time. We have had some "429 - Request rate too large"-errors in the past and I dont wish to create new ones.
So, how will this affect server load/performance.
I dont see a big difference from the servers perspective, between when client is streaming (and doing some quick checks), and old way, get all document and while (iterator.HasMoreResults) and collect the items in a list.
The SDK will retrieve batches of documents that can be adjusted in size using the QueryRequestOptions and changing the MaxItemCount (which defaults to 100 if not set). It has no option though to throttle the RU usage apart from it running into the 429 error and using the retry mechanism the SDK offers to retry a while later. Depending on how generous you set the retry mechanism it'll retry oft & long enough to get a proper response.
If you have a situation where you want to limit the RU usage for e.g. there's multiple processes using your cosmos and you don't want those to result in 429 errors you would have to write the logic yourself.
An example of how something like that could look:
var qry = container
.GetItemLinqQueryable<Item>(requestOptions: new() { MaxItemCount = 2000 })
.ToFeedIterator();
var results = new List<Item>();
var stopwatch = new Stopwatch();
var targetRuMsRate = 200d / 1000; //target 200RU/s
var previousElapsed = 0L;
var delay = 0;
stopwatch.Start();
var totalCharge = 0d;
while (qry.HasMoreResults)
{
if (delay > 0)
{
await Task.Delay(delay);
}
previousElapsed = stopwatch.ElapsedMilliseconds;
var response = await qry.ReadNextAsync();
var charge = response.RequestCharge;
var elapsed = stopwatch.ElapsedMilliseconds;
var delta = elapsed - previousElapsed;
delay = (int) ((charge - targetRuMsRate * delta) / targetRuMsRate);
foreach (var item in response)
{
results.Add(item);
}
}
Edit:
Internally the SDK will call the underlying Cosmos REST API. Once your code reaches the iterator.ReadNextSync() it will call the query documents method in the background. If you would dig into the source code or intercept the message send to HttpClient you can observe the resulting message which lacks the x-ms-max-item-count header that determines the number of the documents it'll try to retrieve (unless you have specified a MaxItemCount yourself). According to the Microsoft Docs it'll default to 100 if not set:
Query requests support pagination through the x-ms-max-item-count and x-ms-continuation request headers. The x-ms-max-item-count header specifies the maximum number of values that can be returned by the query execution. This can be between 1 and 1000, and is configured with a default of 100.
We are using Azure cosmos DB and we are accessing it from Java using the mongo API
Some simple queries that usually use about 4 RUs sometime take about 1500 RU's and we get the message "rate is too large"
If we do the same query from the azure portal UI we always get the lower RU's consumption
Example Document:
{
"_id": {
"$oid": "5c40a6e3f6fe4d1fec5f092e"
},
"attempts": 0,
"status": {
"confirmed": false,
"nSent": true
}
...
rest of non relevant fields
}
if i translate the query to SQL is will look something like this:
SELECT * FROM c WHERE c.status.confirmed = true and (c.status.nSent=null or c.status.nSent=false) and (c.attempts=null or c.attempts<10)
2 important notes:
we do not use the shard key in this query. I don't know if it is relevant here or not
note that we check that c.status.nSent does not exists (=null) or that it is equal to false. If we remove the part that check that c.status.nSent is null then the query goes fine. This is the strange part...
Example of the query as azure sees it in the their log:
(((r1["p1"]["p2"] = true) AND (((r1["p1"]["p3"] ?? null) = null) OR (r1["p1"]["p3"] = false))) AND ((((r1["p4"] ?? null) = null) OR (r1["p4"] <= 10)) AND (r1["p5"] >= 67)))","parameters":[]}"}}
I have also tried to replace this part of the query
(c.status.nSent=null or c.status.nSent=false)
with
(c.status.nSent!=true)
hoping that it will also apply for document that do not have this property but the results are the same
Our average RU/s usage is 40 RU/s and our limit is 4000 RU/s so there should be a lot of spare RU/s for this query
On the exception we get we see this:
RequestCharge: 0.38
Imagine the following function which is querying a GlobalSecondaryIndex and associated Range Key in order to find a limited number of results:
#Override
public List<Statement> getAllStatementsOlderThan(String userId, String startingDate, int limit) {
if(StringUtils.isNullOrEmpty(startingDate)) {
startingDate = UTC.now().toString();
}
LOG.info("Attempting to find all Statements older than ({})", startingDate);
Map<String, AttributeValue> eav = Maps.newHashMap();
eav.put(":userId", new AttributeValue().withS(userId));
eav.put(":receivedDate", new AttributeValue().withS(startingDate));
DynamoDBQueryExpression<Statement> queryExpression = new DynamoDBQueryExpression<Statement>()
.withKeyConditionExpression("userId = :userId and receivedDate < :receivedDate").withExpressionAttributeValues(eav)
.withIndexName("userId-index")
.withConsistentRead(false);
if(limit > 0) {
queryExpression.setLimit(limit);
}
List<Statement> statementResults = mapper.query(Statement.class, queryExpression);
LOG.info("Successfully retrieved ({}) values", statementResults.size());
return statementResults;
}
List<Statement> results = statementRepository.getAllStatementsOlderThan(userId, UTC.now().toString(), 5);
assertThat(results.size()).isEqualTo(5); // NEVER passes
The limit isn't respected whenever I query against the database. I always get back all results that match my search criteria so if I set the startingDate to now then I get every item in the database since they're all older than now.
You should use queryPage function instead of query.
From DynamoDBQueryExpression.setLimit documentation:
Sets the maximum number of items to retrieve in each service request
to DynamoDB.
Note that when calling DynamoDBMapper.query, multiple
requests are made to DynamoDB if needed to retrieve the entire result
set. Setting this will limit the number of items retrieved by each
request, NOT the total number of results that will be retrieved. Use
DynamoDBMapper.queryPage to retrieve a single page of items from
DynamoDB.
As they've rightly answered the setLimit or withLimit functions limit the number of records fetched only in each particular request and internally multiple requests take place to fetch the results.
If you want to limit the number of records fetched in all the requests then you might want to use "Scan".
Example for the same can be found here
I am implementing a website with a dynamodb + nodejs backend. I use Vogels.js in server side to query dynamodb and show results on a webpage. Because my query returns a lot of results, I would like to return only N (such as 5) results back to a user initially, and return the next N results when the user asks for more.
Is there a way I can run two vogels queries with the second query starts from the place where the first query left off ? Thanks.
Yes, vogels fully supports pagination on both query and scan operations.
For example:
var Tweet = vogels.define('tweet', {
hashKey : 'UserId',
rangeKey : 'PublishedDateTime',
schema : {
UserId : Joi.string(),
PublishedDateTime : Joi.date().default(Date.now),
content : Joi.string()
}
});
// Fetch the 5 most recent tweets from user with id 555:
Tweet.query(555).limit(5).descending().exec(function (err, data) {
var paginationKey = data.LastEvaluatedKey;
// Fetch the next page of 5 tweets
Tweet.query(555).limit(5).descending().startKey(paginationKey).exec()
});
Yes it is possible, DynamoDB has some thing called "LastEvaluatedKey" which will server your purpose.
Step 1) Query your table with option "Limit" = number of records
refer: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
Step 2) If your query has more records than the "Limit value", DynamoDB will return a "LastEvaluatedKey" which you can pass in your next query as "ExclusiveStartKey" to get next set of records until there are no records left
refer: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScan.Query
Note: Be aware that to get previous set of records you might have to store all the "LastEvaluatedKeys" and implement this at application level