DynamoDB update - "ValidationException: An operand in the update expression has an incorrect data type" - amazon-dynamodb

I am trying to append to a string set (array of strings) column, which may or may not already exist, in a DynamoDB table. I referred to SO questions like this and this when writing my UpdateExpression.
My code looks like this.
const AWS = require('aws-sdk')
const dynamo = new AWS.DynamoDB.DocumentClient()
const updateParams = {
// The table definitely exists.
TableName: process.env.DYNAMO_TABLE_NAME,
Key: {
email: user.email
},
// The column may or may not exist, which is why I am combining list_append with if_not_exists.
UpdateExpression: 'SET #column = list_append(if_not_exists(#column, :empty_list), :vals)',
ExpressionAttributeNames: {
'#column': 'items'
},
ExpressionAttributeValues: {
':vals': ['test', 'test2'],
':empty_list': []
},
ReturnValues: 'UPDATED_NEW'
}
dynamo.update(updateParams).promise().catch((error) => {
console.log(`Error: ${error}`)
})
However, I am getting this error: ValidationException: An operand in the update expression has an incorrect data type. What am I doing incorrectly here?
[Update]
Thanks to Nadav Har'El's answer, I was able to make it work by amending the params to use the ADD operation instead of SET.
const updateParams = {
TableName: process.env.DYNAMO_TABLE_NAME,
Key: {
email: user.email
},
UpdateExpression: 'ADD items :vals',
ExpressionAttributeValues: {
':vals': dynamo.createSet(['test', 'test2'])
}
}

A list and a string set are not the same type - a string set can only hold strings while a list may hold any types (including nested lists and objects), element types don't need to be the same, and a list can hold also duplicate items. So if your original item is indeed as you said a string set, not a list, this explains why this operation cannot work.
To add items to a string set, use the ADD operation, not the SET operation. The parameter you will give to add should be a set (not a list, I don't know the magic js syntax to specify this, check your docs) with a bunch of elements. If the attribute already exists these elements will be added to it (dropping duplicates), and if the attribute doesn't already exit, it will be set to the set of these elements. See the documentation here: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html#DDB-UpdateItem-request-UpdateExpression

Related

ConditionExpression for PutItem not evaluating to false

I am trying to guarantee uniqueness in my DynamoDB table, across the partition key and other attributes (but not the sort key). Something is wrong with my ConditionExpression, because it is evaluating to true and the same values are getting inserted, leading to data duplication.
Here is my table design:
email: partition key (String)
id: sort key (Number)
firstName (String)
lastName (String)
Note: The id (sort key) holds randomly generated unique number. I know... this looks like a bad design, but that is the use case I have to support.
Here is the NodeJS code with PutItem:
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'})
const params = {
TableName: <table-name>,
Item: {
"email": { "S": "<email>" },
"id": { "N": "<someUniqueRandomNumber>" },
"firstName": { "S": "<firstName>" },
"lastName": { "S": "<lastName>" }
},
ConditionExpression: "attribute_not_exists(email) AND attribute_not_exists(firstName) AND attribute_not_exists(lastName)"
}
dynamodb.putItem(params, function(err, data) {
if (err) {
console.error("Put failed")
}
else {
console.log("Put succeeded")
}
})
The documentation https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html says the following:
attribute_not_exists (path)
True if the attribute specified by path does not exist in the item.
Example: Check whether an item has a Manufacturer attribute.
attribute_not_exists (Manufacturer)
it specifically says "item" not "items" or "any item", so I think it really means that it checks only the item being overwritten. As you have a random sort key, it will always create a new item and the condition will be always true.
Any implementation which would check against a column which is not an index and would test all the records would cause a scan of all items and that is something what would not perform very well.
Here is an interesting article which covers how to deal with unique attributes in dynamodb https://advancedweb.hu/how-to-properly-implement-unique-constraints-in-dynamodb/ - the single table design together with transactions would be a possible solution for you if you can allow the additional partition keys in your table. Any other solution may be challenging under your current schema. DynamoDB has its own way of doing things and it may be frustrating to try to push to do things which it is not designed for.

QML Firebase startAt returns undefined

I am working on a 'typeahead’ type function which will check my Database with the current typed text to provide search suggestions of users using Felgo.
Here is the link for Felgos Firebase documentation
As to not search every entry I am looking to use the startAt and limitTo for a lower data use.
However when applying the startAt my searches only return undefined, I have tried testing this by changing my startAt from a variable to explicit data but this still only returns undefined.
My function is below:
function searchUsers(searchString) {
db.getValue("public/nameList/", {
orderByChild: true,
startAt: searchString, //searchString is a variable with my .currentText to search.
limitToFirst: 10,
}, function(success, key, value) {
if(success) {
searchArr = []
searchArr = value
console.debug("Read user value for key", key, "from DB:", value)
}
})
}
I have also tried by passing my var searchString through JSON.stringify(searchString) and also return undefined!
Removing the startAt: query entirely returns the entire result of nameList as expected, but no matter how I try to implement my startAt it always returns undefined.
A sample of my nameList JSON is:
nameList: {
"EddieLaw245" : 530343772383,
"EddieLawrence91" : 530343772385,
"EdwardL91" : 530343772386,
"EdwardLaw" : 530343772384,
"Edwardlawrence91" : 530343772380,
"JoBrownLondon" : 530343772381,
"KatiePrescottHair" : 543592635596,
"Tracey-Sweeting" : 530343772382
}
So with the above example, When I type E it should remove the last 3 entries, and so on.
The problem is that you're specifying orderByChild: true. If we look at the documentation of that:
orderByChild: If present, the queried object will have its properties ordered by values at sub-paths defined by the value of this property. Ordering by child properties makes the filter properties startAt, endAt and equalTo filter by the child property values
It may not be immediately clear from this, but orderByChild allows you to order the results on a property value under each of those nodes. So your code tries to order the child nodes on the value of a property true, which isn't possible (and should actually generate a compile-time error in the library) as the nodes under nameList don't have any child properties of their own. They merely have a key and a value.
What you're looking for is orderByKeys, which orders the child nodes on their keys. So:
db.getValue("public/nameList/", {
orderByKeys: true,
startAt: searchString,
limitToFirst: 10,
}
You'll typically also want to specify an endAt value, to ensure your type-ahead only shows values that start with the search string. If you only allow ASCII values in the keys, the simplest way to do this is:
startAt: searchString,
endAt: searchString + "~",
The ~ here is no magic operator, but merely the last ASCII characters. If you want to allow a broader character set, you'll need to use the last character in that character set - for example \uF7FF is the last code point for Unicode.
Update from OP
Though I'm certian Franks correct with typical Firebase usage; I suspect due to the Felgo plugin I am using the full solution has a slight adjustment;
db.getValue("public/nameList/", {
"orderByKey": true,
"startAt": searchString,
"endAt": searchString+"~",
"limitToFirst": 10,
}, function(success, key, value) {....}
})
Notes on the above - my filters/queries are surrounded by quotation marks "startAt", also instead of orderByKeys, I have used orderByKey

Add to list only if string doesn't already exist in DynamoDB table

I'm trying with the following code
{
ExpressionAttributeNames: {
"#items": "items"
},
ExpressionAttributeValues: {
":item": [slug]
},
Key: {
listId: listId,
userId: userData.userId,
},
UpdateExpression: "SET #items = list_append(#items,:item)",
ConditionExpression: "NOT contains (#items, :item)",
TableName: process.env.listsTableName,
}
but the item is still added even if string already exists in the list. What am I doing wrong?
The list structure is like so:
{
Item: {
userId: userData.userId,
listId: crypto.createHash('md5').update(Date.now() + userData.userId).digest('hex'),
listName: 'Wishlist',
items: [],
},
TableName: process.env.listsTableName,
};
Later Edit: I know I should use SS as it does the condition for me but SS doesn't work in my context because SS can't be empty.
As the documentation explains, the contains() function only works on a string value (checking for a substring) or a set value (checking for membership). But in your case, you don't have a set but rather a list - with are different things in DynamoDB.
If all the items which you want to add to this list are strings, and you anyway don't want duplicates in the list, the most efficient way would be to stop using a list, and instead use the set-of-strings (a.k.a. SS) type. To add an item to the set (without duplicates), you would simply use "ADD #items :item" (no need for any additional condition - duplicates will not be added).

dynamo db FilterExpression, find in json object using value as key

It is possible to somehow filter results by key name that stored in the same object?
I have JSON object "keys", in property "default" stored key of the object that I need. Is it somehow possible to filter like that keys[keys.default].type = some_type?
var params = {
TableName: 'TABLE_NAME',
IndexName: 'TABLE_INDEX', // optional (if querying an index)
KeyConditionExpression: 'myId = :value',
FilterExpression: '#kmap[#kmap.#def].#tp = :keyval',
ExpressionAttributeNames: {names with special characters
'#kmap': 'keys',
'#tp': 'type',
'#def': 'default'
},
ExpressionAttributeValues: { // a map of substitutions for all attribute values
':value': '1',
':keyval': 'some_type'
},
Limit: 10, // optional (limit the number of items to evaluate)
ProjectionExpression: "displayName, #kmap",
ReturnConsumedCapacity: 'TOTLAL', // optional (NONE | TOTAL | INDEXES)
};
docClient.query(params, function(err, data) {
if (err) ppJson(err); // an error occurred
else ppJson(data); // successful response
});
I'm pretty sure the answer is no.
This keys[keys.default] is not even valid json, as far as I can tell.
Of course, you can do this in two steps:
First, query to get the default key
Then query to get the value
Don't forget, filters are obly applied to the result set - it still requires a libear traversal as specified by your Query or Scan operation.
So you can probably more easily run your query on the client.
And lastly, if this is a typical query ypu need to perform, as an optimization, you can lift the default key and value to be top level attributes on the item. Thrn you can actually create a GSI on that attribure and can actually do efficient lookups.

inserting a nested attributes array element in dynamodb table without retrieving the entire item

looking for an answer to this question if possible, not looking for a refactoring advice or redesign, i just need to understand what else I am missing here :
I have an existing item in dynamodb:
{
"CartId": 321,
"UserId": usr555,
"CartItems": [
{
"ProductId":59999,
"Quantity": 1
},
{
"ProductId": 58888,
"Quantity": 2
}
]
}
in my code I want to insert another nested attribute into the array CartItems in the item above. i can't find a way of doing this without retrieving the entire item and then inserting, which could amount to KBs in size.
all I want to do is insert a single array element in an existing nested attribute without having to retrieve the item.
the language I am using is nodejs and the function is DynamoDB.put.
UpdateExpression attribute supports SET action and SET action supports list_append function. Since list_append function operands must be list, enclose the new value within a list as below.
response = dynamodb_client.update_item(TableName='your_table',
Key={'CartId':'321'},
UpdateExpression='SET CartItems = list_append(CartItems, :val1)',
ExpressionAttributeValues = {':val1':[{'ProductId':12345, 'Quantity': 100}]})
Hope this helps

Resources