Query works at the console but not in code - amazon-dynamodb

My DynamoDB table alexas has this item with key "abc" as seen in the DynamoDB console below:
However, the following query returns no result:
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: { ":deviceId": "abc"}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
The above code returns null for err and in data:
{ Items: [], Count: 0, ScannedCount: 0 }
I am new to the DynamoDB style of expressions. Is there anything wrong with my code which I took from here.
If instead of query, I used the scan method and just have TableName in params, I get the items in my table. This confirms that I am performing the operations on the correct table that has data.

The query returned no data because the key value does not match.
The item's deviceId is the string "abc" and not abc. Note the extra quotation marks.
The item was inserted using the DynamoDB console's Create editor and there is no need to include "" if the value is already expected to be of type string.

DynamoDB's Scan operation doesn't take a KeyConditionExpression - only the Query operation takes this parameter. Scan always scans the entire table, and has a FilterExpression to post-filter these results (however please note that you still pay for scanning the entire table).
For example, here is the official documentation of Scan: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html

Check QueryAPI
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: {
":devideId":{
S: "abc", // here
}
}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
ExpressionAttributeValues needs to be passed in a different manner.
Update:
Try using Exp attribute names, (I'm not sure if this will make a difference)
var params = {
TableName: "alexas",
KeyConditionExpression: "#d = :dId",
ExpressionAttributeNames:{
"#d": "email"
},
ExpressionAttributeValues: {
":dId": "abc"
}
};

Related

Get status for a list of items

I have a dynamodb table with a partition key for client_id and no sort key. The json stored in the table just contains the client_id and compliance_level (which is a string title). I need to query the table for a list of client_id's because there is an application that displays client information in a tabular display. I am trying to use ExpressionFilter but get an "Error ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request." exception. I don't however have a keycondition to query on. Any assistance would be appreciated. I can't use batchgetitem as there will be more than 100 items to get the status for.
var AWS = require('aws-sdk');
AWS.config.update({region: 'ap-southeast-2'});
var ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
var params = {
TableName : "Client-Compliance",
ProjectionExpression: "username, version",
FilterExpression : "client_id IN (:client1, :client2)",
ExpressionAttributeValues : {
":client1" : { "S": "c1234567" },
":client2" : { "S": "c88888888" }
}
};
ddb.query(params, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log(data);
}
});
Many thanks

How to backfill new AppSync fields using AWS Amplify

I'm adding a sort field to one of my AppSync tables using GraphQL. The new schema looks like:
type MyTable
#model
#auth(rules: [{allow: owner}])
#key(name: "BySortOrder", fields: ["sortOrder"], queryField: "tableBySortOrder")
{
id: ID!
name: String!
sortOrder: Int
}
However, when retrieving a list using tableBySortOrder I get an empty list because the new field sortOrder is null.
My question is, how do I backfill this data in the DynamoDB table so that my existing users will not be disrupted by this new change? With a traditional database, I would run a SQL update: UPDATE MyTable SET sortOrder = #.
However, I'm new to NoSQL/AWS and couldn't find a way to do this except build a backfill script whenever a user logs into my app. That feels very hacky. What is the best practice for handling this type of scenario?
Have you already created the new field in DDB?
If yes, I think you should backfill it before making the client side change.
Write a script to iterate through and update the table. Options for this:
Java - Call updateItem to update the table if you have any integ tests running.
Bash - Use AWS CLI: aws dynamodb scan --table-name item_attributes --projection-expression "whatever" > /tmp/item_attributes_table.txt and then aws dynamodb update-item --table-name item_attributes --key. This is a dirty way.
Python - Same logic as above.
Ended up using something similar to what Sunny suggested with a nodejs script:
const AWS = require('aws-sdk')
AWS.config.update({
region: 'us-east-1'
})
// To confirm credentials are set
AWS.config.getCredentials(function (err) {
if (err) console.log(err.stack)
// credentials not loaded
else {
console.log('Access key:', AWS.config.credentials.accessKeyId)
console.log('Secret access key:', AWS.config.credentials.secretAccessKey)
}
})
const docClient = new AWS.DynamoDB.DocumentClient()
const table = 'your-table-dev'
const params = {
TableName: table
}
const itemMap = new Map()
// Using scan to retrieve all rows
docClient.scan(params, function (err, data) {
if (err) {
console.error('Unable to query. Error:', JSON.stringify(err, null, 2))
} else {
console.log('Query succeeded.')
data.Items.forEach(item => {
if (itemMap.has(item.owner)) {
itemMap.set(item.owner, [...itemMap.get(item.owner), item])
} else {
itemMap.set(item.owner, [item])
}
})
itemMap.forEach(ownerConnections => {
ownerConnections.forEach((connection, index) => {
connection.sortOrder = index
update(connection)
})
})
}
})
function update(connection) {
const params = {
TableName: table,
Key: {
'id': connection.id
},
UpdateExpression: 'set sortOrder = :s',
ExpressionAttributeValues: {
':s': connection.sortOrder,
},
ReturnValues: 'UPDATED_NEW'
};
console.log('Updating the item...');
docClient.update(params, function (err, data) {
if (err) {
console.error('Unable to update item. Error JSON:', JSON.stringify(err, null, 2));
} else {
console.log('UpdateItem succeeded:', JSON.stringify(data, null, 2));
}
});
}

dynamodb index does not return all data

I have a table storedgames, which contains 2092 items.
And it also has an index on that table, which also lists 2092 items.
when I fetch data, I use the index, to obtain the items for one specific user.
const params = {
TableName: "storedgames",
IndexName: "user-index",
KeyConditionExpression: "#usr = :usr",
ExpressionAttributeNames: { "#usr": "user" },
ExpressionAttributeValues: { ":usr": user }
};
const data = await new Promise((resolve, reject) => {
docClient.query(params, (err, data) => {
if (err) { reject(err); } else { resolve(data); }
});
}).catch((err) => {
console.error(err);
return false;
});
However, the above code does not return all items. It only finds 42. And for today's items there is only 1 hit. When I check directly on the AWS webpage, I actually find more items for today.
And even when I do this using the index, it finds more records.
When I leave out the filtering of the day, I actually find over 130 items,
while my javascript code only returns 42 items when I leave out the day filter.
So my question is, why does the data of my index seem to be incomplete when I call it programmatically ?
The records actually contain a lot of data, and there appears to be a limit in the amount of data that can be fetched per query.
A single Query operation can retrieve a maximum of 1 MB of data. This
limit applies before any FilterExpression is applied to the results.
If LastEvaluatedKey is present in the response and is non-null, you
must paginate the result set (see Paginating the Results).
So, I one possible solution, is to perform multiple fetches until you have the entire collection.
const queryAllItems = (params, callback) => {
let fullResult = { Items: [], Count: 0, ScannedCount: 0 };
const queryExecute = (callback) => {
docClient.query(params, (err, result) => {
if (err) {
callback(err);
return;
}
const { Items, LastEvaluatedKey, Count, ScannedCount } = result;
fullResult.Items = [...fullResult.Items, ...Items];
fullResult.Count += Count;
fullResult.ScannedCount += ScannedCount;
if (!LastEvaluatedKey) {
callback(null, fullResult);
return;
}
params.ExclusiveStartKey = LastEvaluatedKey;
queryExecute(callback);
});
}
queryExecute(callback);
}
Unfortunately, this isn't a complete solution. In my situation, a query for a mere 130 items (which require 4 actual fetches) takes about 15 seconds.

(DynamoDB) ConditionExpression behaves unpredictably with an '=' operator. How could I debug it better?

My objects in Dynamodb look roughly like this:
{
userId: "GEFOeE8EsaWmq4NQ3oh7tbeVkLx1",
url: 'objectURL',
object: {}
}
I have this simple piece of code for deleting an object, when the user that owns the object requests a delete. The user argument here is a parsed JWT, by the way.
export async function deleteObject(user, url) {
let params = {
TableName: OBJECTS_TABLE,
Key: {
url: url,
},
ConditionExpression: `userId = :uid`,
ExpressionAttributeValues: {
":uid": {
S: user.sub
}
}
};
let deleteResult = await dynamoDb.delete(params).promise();
return deleteResult;
}
The problem is that it doesn't work, and I've made sure that the problem stems from the ConditionExpression by changing = to <>. I simply get this:
ConditionalCheckFailedException: The conditional request failed
I'm sure solving the problem wouldn't be difficult, but I barely have any information
Questions:
Why is the condition expression failing? Everything looks alright, and it should work. Right?
How could I debug this issue better?
The await/async is not supported by AWS SDK at the moment. Please refer this similar issue.
The SDK currently relies on CLS to trace the call context. It doesn't
work with async/await functionality right now. You can see the
discussion here.
It should work if you remove the await. Example below:-
let deleteResult = dynamodb.deleteItem(params).promise();
deleteResult.then(function (data) {
console.error("Delete item result :", JSON.stringify(data,
null, 2));
}).catch(function (err) {
console.error("Delete item result error :", JSON.stringify(err,
null, 2));
});
I figured it out. ExpressionAttributeValues can be directed used, without mentioning the datatype. The Javascript SDK does that automatically.
export async function deleteObject(user, url) {
let params = {
TableName: OBJECTS_TABLE,
Key: {
url: url,
},
ConditionExpression: `userId = :uid`,
ExpressionAttributeValues: {
":uid": user.sub
}
};
let deleteResult = await dynamoDb.delete(params).promise();
return deleteResult;
}

convert a dynamodb scan to query

I've written a API gateway to scan a dynamodb table and get values based on the condition and my code is as below.
var params = {
TableName: 'CarsData',
FilterExpression: '#market_category = :market_category and #vehicle_size = :vehicle_size and #transmission_type = :transmission_type and #price_range = :price_range and #doors = :doors',
ExpressionAttributeNames: {
"#market_category": "market_category",
"#vehicle_size": "vehicle_size",
"#transmission_type": "transmission_type",
"#price_range": "price_range",
"#doors": "doors"
},
ExpressionAttributeValues: {
":market_category": body.market_category,
":vehicle_size": body.vehicle_size,
":transmission_type": body.transmission_type,
":price_range": body.price_range,
":doors": body.doors
}
}
dynamodb.scan(params).promise().then(function (data) {
var uw = data.Items;
console.log(data + "\n" + JSON.stringify(data) + "\n" + JSON.stringify(data.Items));
var res = {
"statusCode": 200,
"headers": {},
"body": JSON.stringify(uw)
};
ctx.succeed(res);
}).catch(function (err) {
console.log(err);
var res = {
"statusCode": 404,
"headers": {},
"body": JSON.stringify({ "status": "error" })
};
ctx.succeed(res);
});
when I run this code, I get the result as expected. But when I was going through some online forums, I came to know that scanning is expensive compared to querying. But I'm unable to know on how can I change my query from scan to query. Here my primary key is ID. please let me know on how can I do this.
Thanks
Scan operation is more expensive comparing to query operation, in terms of performance as well as costing. Dynamodb calculates cost based on the number of read capacity units consumed for processing not on number of records returned.
Query operation finds value based on primary key (Hash) or composite primary key (Hash key and Sort Key).
Your schema should be redesigned with composite primary key(Hash key and Sort Key).
Its not neccessary to have column Id as primary Key like old school RDBMS. If you are not using Id effectively remove that column from your schema and redefine it with some other attributes. For an example am using Market Category (market_category ) as Hash Key & Price Range (price_range) as Range Key.
var params = {
"TableName": 'CarsData',
"ConsistentRead": true,
//Composite Primary Key in Key Condition Expression
"KeyConditionExpression": "#market_category = :market_category AND #price_range = :price_range",
//Remaining column in filter expression
"FilterExpression": '#vehicle_size = :vehicle_size and #transmission_type = :transmission_type and #doors = :doors',
"ExpressionAttributeNames": {
"#market_category": "market_category",
"#vehicle_size": "vehicle_size",
"#transmission_type": "transmission_type",
"#price_range": "price_range",
"#doors": "doors"
},
"ExpressionAttributeValues": {
":market_category": body.market_category,
":vehicle_size": body.vehicle_size,
":transmission_type": body.transmission_type,
":price_range": body.price_range,
":doors": body.doors
}
}
dynamodb.query(params).promise()
.then(function (data) {
console.log(data);
}).catch(function (err) {
console.log(err);
});
Hope this example will give you insights about using composite primary key,
Based on your usage choose the widely used columns for Hash & Range key.

Resources