AWS Scan ignores withLimit() - amazon-dynamodb

Question

I am trying to fetch the items from a DynamoDB table to put them in a csv file. Following is the code:
ArrayList<String> ids = new ArrayList<String>();
ScanResult result = null;
do{
ScanRequest req = new ScanRequest();
req.setTableName("table");
req.withLimit(10);
if(result != null){
req.setExclusiveStartKey(result.getLastEvaluatedKey());
}
AmazonDynamoDBClient client = new AmazonDynamoDBClient(awsCreds);
result = client.scan(req);
List<Map<String, AttributeValue>> rows = result.getItems();
for(Map<String, AttributeValue> map : rows){
try{
AttributeValue v = map.get("prod_number");
String id = v.getS();
ids.add(id);
} catch (NumberFormatException e){
System.out.println(e.getMessage());
}
}
} while(result.getLastEvaluatedKey() != null);
System.out.println("Result size: " + ids.size());
I want to know why 'req.withLimit(10)' has no impact on the number of results. The query still tries to fetch all the records.

Answer 1

The limit property of ScanRequest means:
The maximum number of items to evaluate (not necessarily the number of matching items). If DynamoDB processes the number of items up to the limit while processing the results, it stops the operation and returns the matching values up to that point, and a key in LastEvaluatedKey to apply in a subsequent operation, so that you can pick up where you left off. Also, if the processed dataset size exceeds 1 MB before DynamoDB reaches this limit, it stops the operation and returns the matching values up to the limit, and a key in LastEvaluatedKey to apply in a subsequent operation to continue the operation. For more information, see Working with Queries in the Amazon DynamoDB Developer Guide.
So, it limits only the size of a portion of data returned by a single request, but not the whole scan operation. And I see you're doing multiple requests, so you'll get more data.