I've written a API gateway to scan a dynamodb table and get values based on the condition and my code is as below.
var params = {
TableName: 'CarsData',
FilterExpression: '#market_category = :market_category and #vehicle_size = :vehicle_size and #transmission_type = :transmission_type and #price_range = :price_range and #doors = :doors',
ExpressionAttributeNames: {
"#market_category": "market_category",
"#vehicle_size": "vehicle_size",
"#transmission_type": "transmission_type",
"#price_range": "price_range",
"#doors": "doors"
},
ExpressionAttributeValues: {
":market_category": body.market_category,
":vehicle_size": body.vehicle_size,
":transmission_type": body.transmission_type,
":price_range": body.price_range,
":doors": body.doors
}
}
dynamodb.scan(params).promise().then(function (data) {
var uw = data.Items;
console.log(data + "\n" + JSON.stringify(data) + "\n" + JSON.stringify(data.Items));
var res = {
"statusCode": 200,
"headers": {},
"body": JSON.stringify(uw)
};
ctx.succeed(res);
}).catch(function (err) {
console.log(err);
var res = {
"statusCode": 404,
"headers": {},
"body": JSON.stringify({ "status": "error" })
};
ctx.succeed(res);
});
when I run this code, I get the result as expected. But when I was going through some online forums, I came to know that scanning is expensive compared to querying. But I'm unable to know on how can I change my query from scan to query. Here my primary key is ID. please let me know on how can I do this.
Thanks
Scan operation is more expensive comparing to query operation, in terms of performance as well as costing. Dynamodb calculates cost based on the number of read capacity units consumed for processing not on number of records returned.
Query operation finds value based on primary key (Hash) or composite primary key (Hash key and Sort Key).
Your schema should be redesigned with composite primary key(Hash key and Sort Key).
Its not neccessary to have column Id as primary Key like old school RDBMS. If you are not using Id effectively remove that column from your schema and redefine it with some other attributes. For an example am using Market Category (market_category ) as Hash Key & Price Range (price_range) as Range Key.
var params = {
"TableName": 'CarsData',
"ConsistentRead": true,
//Composite Primary Key in Key Condition Expression
"KeyConditionExpression": "#market_category = :market_category AND #price_range = :price_range",
//Remaining column in filter expression
"FilterExpression": '#vehicle_size = :vehicle_size and #transmission_type = :transmission_type and #doors = :doors',
"ExpressionAttributeNames": {
"#market_category": "market_category",
"#vehicle_size": "vehicle_size",
"#transmission_type": "transmission_type",
"#price_range": "price_range",
"#doors": "doors"
},
"ExpressionAttributeValues": {
":market_category": body.market_category,
":vehicle_size": body.vehicle_size,
":transmission_type": body.transmission_type,
":price_range": body.price_range,
":doors": body.doors
}
}
dynamodb.query(params).promise()
.then(function (data) {
console.log(data);
}).catch(function (err) {
console.log(err);
});
Hope this example will give you insights about using composite primary key,
Based on your usage choose the widely used columns for Hash & Range key.
Related
I have a dynamodb table with a partition key for client_id and no sort key. The json stored in the table just contains the client_id and compliance_level (which is a string title). I need to query the table for a list of client_id's because there is an application that displays client information in a tabular display. I am trying to use ExpressionFilter but get an "Error ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request." exception. I don't however have a keycondition to query on. Any assistance would be appreciated. I can't use batchgetitem as there will be more than 100 items to get the status for.
var AWS = require('aws-sdk');
AWS.config.update({region: 'ap-southeast-2'});
var ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
var params = {
TableName : "Client-Compliance",
ProjectionExpression: "username, version",
FilterExpression : "client_id IN (:client1, :client2)",
ExpressionAttributeValues : {
":client1" : { "S": "c1234567" },
":client2" : { "S": "c88888888" }
}
};
ddb.query(params, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log(data);
}
});
Many thanks
I am attempting to create a leader board using dynamo db for an quiz style Alexa skill. I have set up the table and users are added to the table with their appropriate data e.g.:
Item: {
"PlatformId": 2,
"UserId": 12345,
"Score": 100,
"NickName": "scott",
"Sport": "football",
}
In my table the Primary key is their UserId, the sort key is the PlatformId (this is the same for all users). I have a secondary global index which sets the platformId as the primary key, and the score as the sort key.
In this leader board i want users to be ranked, the highest scorer being number 1, my first attempt at this was to scan the table using the secondary index, this nicely returned all the users sorted by score, however with the potential to have thousands of users on this leader board, i discovered that the time to scan a table with 10000+ users exceeds the 8 second response time that Alexa skills have. This causes the skill to error and close.
Before the response time exceeded, i was using the LastEvaluatedKey to perform an extra scan if the first one didn't cover the entire table, but on this second scan is when the response time limit was exceeded. Annoyingly it's just taking too long to scan the table.
dbHelper.prototype.scanGetUsers = (ad, newParams = null) => {
return new Promise((resolve, reject) => {
let params = {};
if (newParams != null) {
params = newParams
} else {
params = {
TableName: tableName,
IndexName: 'PlatformId-Score-index',
FilterExpression: "Score >= :s AND PlatformId = :p",
ProjectionExpression: `NickName, Sport, Score`,
// Limit: 10,
ExpressionAttributeValues: {
":p": User.PlatformId,
":s": User.Score,
},
}
}
docClient.scan(params, function (err, data) {
if (err || !data) {
console.error("Unable to read item. Error JSON:", JSON.stringify(err, null, 2));
return reject(JSON.stringify(err, null, 2))
} else {
console.log("scan users data succeeded:", JSON.stringify(data, null, 2));
if(data.LastEvaluatedKey) {
console.log("found a LastEvalutedKey, Continuing scan");
params.ExclusiveStartKey = data.LastEvaluatedKey;
data = data.concat(this.scanGetUsers(ad, params));
}
resolve(data);
}
});
});
}
Is there a way to work around these issues that i haven't explored yet? Or a way to create a leader board with dynamo db that can be structured in an easier way?
You can try Sort Key:
When you combine a partition key and sort key, they create a composite key, and that composite key is the primary key for individual items in a table. With a composite key, you gain the ability to use queries with a KeyConditionExpression against the sort key. In a query, you can use KeyConditionExpression to write conditional statements by using comparison operators that evaluate against a key and limit the items returned. In other words, you can use special operators to include, exclude, and match items by their sort key values.
The article contains all information how to setup and use it.
My DynamoDB table alexas has this item with key "abc" as seen in the DynamoDB console below:
However, the following query returns no result:
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: { ":deviceId": "abc"}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
The above code returns null for err and in data:
{ Items: [], Count: 0, ScannedCount: 0 }
I am new to the DynamoDB style of expressions. Is there anything wrong with my code which I took from here.
If instead of query, I used the scan method and just have TableName in params, I get the items in my table. This confirms that I am performing the operations on the correct table that has data.
The query returned no data because the key value does not match.
The item's deviceId is the string "abc" and not abc. Note the extra quotation marks.
The item was inserted using the DynamoDB console's Create editor and there is no need to include "" if the value is already expected to be of type string.
DynamoDB's Scan operation doesn't take a KeyConditionExpression - only the Query operation takes this parameter. Scan always scans the entire table, and has a FilterExpression to post-filter these results (however please note that you still pay for scanning the entire table).
For example, here is the official documentation of Scan: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
Check QueryAPI
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: {
":devideId":{
S: "abc", // here
}
}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
ExpressionAttributeValues needs to be passed in a different manner.
Update:
Try using Exp attribute names, (I'm not sure if this will make a difference)
var params = {
TableName: "alexas",
KeyConditionExpression: "#d = :dId",
ExpressionAttributeNames:{
"#d": "email"
},
ExpressionAttributeValues: {
":dId": "abc"
}
};
I am using following function ->
const params = {
TableName: process.env.dummyTable,
Key: {
outlet_id:event.pathParametrs.id,
id:{"S":"default"}
}
}
dynamoDb.get(params).promise()
.then(result => {
console.log('-->',result);
const response = {
statusCode: 200,
body: JSON.stringify(result),
};
callback(null, response);
})
.catch(error => {
console.error(error);
callback(new Error('Couldn\'t fetch table Data'));
return;
});
}
I want to fetch records based on outlet_id and id.Here outlet_id is primary partition key while id is primary sort key(with uuid).
How to specify id(primary sort key) with default value so that I can fetch data
It's unclear whether you are trying to fetch a single item by specifying a composite partition+sort key, or trying to query all items with the same partition key (ie. without specifying a sort key).
If you were trying to get a single item, with a particular partition key and sort key, your code looks good with one exception. The partition key should be specified with the type as well, like so:
const params = {
TableName: process.env.dummyTable,
Key: {
outlet_id:{"S": event.pathParametrs.id},
id:{"S":"default"}
}
}
However, if you are trying to query for all items with the same partition key (ie. without specifying a sort key) then you would need to modify the code to do a query rather than a get:
const params = {
TableName: process.env.dummyTable,
Key: {
outlet_id: {"S": event.pathParametrs.id}
}
}
dynamoDb.query(params).promise()
.then(
//...
)
.catch(
//...
)
I have a dynamodb table that stores users videos.
It's structured like this:
{
"userid": 324234234234234234, // Hash key
"videoid": 298374982364723648 // Range key
"user": {
"username": "mario"
}
}
I want to update username for all videos of a specific user. It's possible with a simple update or i have to scan the complete table and update one item a time?
var params = {
TableName: DDB_TABLE_SCENE,
Key: {
userid: userid,
},
UpdateExpression: "SET username = :username",
ExpressionAttributeValues: { ":username": username },
ReturnValues: "ALL_NEW",
ConditionExpression: 'attribute_exists (userid)'
};
docClient.update(params, function(err, data) {
if (err) fn(err, null);
else fn(err, data.Attributes.username);
});
I receive the following error, I suppose the range key is necessary.
ValidationException: The provided key element does not match the schema
Dynamo does not support write operations across multiple items (ie. for more than one item at a time). You will have to first scan/query the table, or otherwise generate a list of all items you'd like to update, and then update them one by one.
Dynamo does provide a batching API but that is still just a way to group updates together in batches of 25 at a time. It's not a proxy for a multi-item update like you're trying to achieve.