DynamoDB how to Update a Map if an attribute exists, else silently ignore - amazon-dynamodb

I have a table called Products, whose Key is a Range : orgzviceid + productid. It has a map attribute called "checkout" and a quantity storing attribute called "prod_stk_qty_i_i".
Say initially, for a product with Product ID 34, total available quantity is 10. As soon as a Cart checkout happens, assuming the Checkout ID is 5, and it has checked 2 quantities out a product id 34, then the product's (for productid 34) "checkout" map entry and "prod_stk_qty_i_i" in DynamoDB would be something like this:
"checkout" : { "5" : 2 },
"prod_stk_qty_i_i" : 8
If another checkout happens for the same product (say 1 quantity), and if that checkout ID is 7, then the checkout ooks like this:
"checkout" : { "5" : 2, "7" : 1 },
"prod_stk_qty_i_i" : 7
If payment is made, the checkout entry is removed, and quantity is increased.
Now, my requirement is to periodically after some timeout (30 minutes), release the Product Quantities which have been checked out, but not released. I do this by
Increasing the Quantity by "checkout."'s value
Removing the checkout. map entry
It is important that this operation not fail even if this operation is attempted multiple times, (idempotent), so its necessary that it only update if the checkout.checkoutID field exists. If not, it should simply ignore.
I tried the following:
[
"UpdateItem",
[
{
"TableName": "Products",
"Key": {
"orgzviceid": {
"N": "3000161710"
},
"productid": {
"N": "11"
}
},
"UpdateExpression": "REMOVE #checkout.#checkoutID SET #prod_stk_qty_i_i = #prod_stk_qty_i_i + #checkout.#checkoutID",
"ExpressionAttributeNames": {
"#checkout": "checkout",
"#checkoutID": "235",
"#prod_stk_qty_i_i": "prod_stk_qty_i_i"
},
"ConditionExpression": "attribute_exists(#checkout.#checkoutID)",
"ReturnValues": "ALL_NEW"
}
]
]
However, it gives me an error in case the checkout entry is not found for checkout id 235. Note that I've written ConditionExpression to do the update only if attribute "condition.235" exists.
Error Logs:
com.amazonaws.dynamodb.v20120810#ConditionalCheckFailedException","message":"The
conditional request failed ..."
So, how do I write a query such that if the map entry exist, then do the above operation, other wise not fail?
Obviously, one bad hack is to first check in a GetItem query if the checkout entry exists for the provided CheckoutID, and then only do this, however, it just does not seem right

I believe your using conditional expressions incorrectly. The point of the conditional is to fail if certain criteria is not met. WHy do you have the conditional at all? Without the conditional it would just execute the update expression and if the item does not exist I would not expect you to get an error. Like querying for an item that does not exist. You should simply get an empty set back. Not an error.

Your approach will not work because you are mixing "Attribute" and "AttributeValue" together in your conditional expression. Let me explain:
"ConditionExpression": "attribute_exists(#checkout.#checkoutID)"
In your table, checkout is an attribute in dynamo db, whereas checkoutID is in no way related to the table schema. So for dynamo DB, checkoutID is part of the attribute's value and not the attribute itself.
Therefore, to having the condition that you do will not work.
A conditional expression for your use case would be something which says attribute checkout exists and it's value is . However, in order to do that, you'd need to pass the expected map which boils down to reading the record before updating.
I do think that reading the record, updating the value and persisting it should be the way to go ahead in this case (and is not necessarily a bad idea)
Do consider using some kind of optimistic locking in this case to prevent against dirty reads and writes.

Related

Create item if not found and return old one if existing in dynamodb, in one call

In DynamoDb, is it possible to do a conditional put and return the old item if there already was a matching item?
I would like something like the following to create the row if the user does not already exist, and if it does exist, I want it to return the old item (with whatever name was there before). I.e., insert item if new, else just read existing item.
await documentClient.put({
TableName: 'table',
Item: {
userId: 'user0',
name: 'smith',
},
ConditionExpression: 'attribute_not_exists(userId)',
});
Is this possible to do in one call?
Sort of.
If you use UpdateItem it will create or update the item. It only updates the fields that have changed however, so if your fields have not changed, then no update will be made. It will also create the item if you do not have it.
Using Conditional Items for this will return an ErrorCode if the condition fails ConditionalCheckFailedException - which would require an additional operation.
(I don't use the javascript SDK that much, more in python, so I'm not sure this is the correct version/page but here is some documentation:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#updateItem-property )

DynamoBD/Amplify non-negative field and field validation on mutations

I am new to AWS in general, I am building a relatively simple application with Amplify, but I've used Google Firebase before. My question is: Is there a way to set a constrain for a field to be non-negative? I have an application that does transactions and I don't want my balance to be negative. I just need a simple error/exception. Is it possible to set a field constraint in DynamoDB that says "This field should be >= 0"?.
I also checked if it was possible to do it in the VTL amplify generated resolver of my graphql mutation, and indeed it is possible to set some constraints, But somehow it allows the operation and crashes on the next one (when the balance on the DB is already < 0, like if it checks it before the update). I tried saying something like "current_balance - transaction >= 0" but I couldn't get it to work.
So it seems that the only way is to create a custom lambda resolver that does the various checks before submitting the mutation to DynamoDB. I haven't tried it yet but I don't understand how I can do a check on the current balance (stored in the DB) without doing a query.
More in general is it even possible to validate fields (even with simple assertions like non-negative) on amplify/dynamoDB? Moving to another DB like Aurora would help?
Thanks for you help
DynamoDb supports conditional updates which allow an update to be applied when the given condition is met. You can set the condition current_balance >= cost for your update.
However, the negative balance is not the main problem. What you should address is how to prevent other requests from updating the same current_balance at the same time, or in short, race conditions on current_balance. In order to deal with that, you also need a conditional update whose condition is "current_balance = initial_balance". The initial_balance is, I guess, what you get from DynamoDB at the very beginning of the purchase process.
Sample VTL code
#set( $remaining_balance = $initial_balance - $transaction_cost )
#if( $remaining_balance < 0 )
$util.error("Insufficient balance")
#end
{
"version" : "2018-05-29",
"operation" : "UpdateItem",
"key": { <your-dynamodb-key> },
"update" : {
"expression" : "SET current_balance = :remaining_balance",
"expressionValues" : {
":remaining_balance" : $util.dynamodb.toNumberJson($remaining_balance)
}
},
"condition": {
"expression": "current_balance = :initial_balance",
"expressionValues" : {
":initial_balance" : $util.dynamodb.toNumberJson($initial_balance)
}
}
}

Can a DynamoDB Condition Expression work on just the Partition Key of a table with a Composite Key

I have a DynamoDB table, with a composite key, which looks like this:
PK
SK
Type
Email
Description
USER#A
USER#A
User
a#example.com
USER#A
BUG#1
Bug
This looks ok
USER#B
BUG#2
Bug
My user wasn't created first!
I'd like to ensure that a "User" record exists before adding a related "Bug" record - So the 3rd item here is incorrect.
When I put a bug item with the condition attribute_exists(PK), the condition is never true. When I remove the condition, I end up with a that third row; A Bug with no corresponding User.
My understanding is that attribute_exists() only looks at items with the combined composite key, and not across the whole table, regardless of which attribute you supply.
Is there a method of ensuring an item with the same Partition Key exists, while ignoring the Sort Key in this scenario?
DynamoDB condition expressions can be confusing, and the docs can compound that problem!
The DynamoDB condition expression works by 1) finding the item, 2) evaluating the condition expression, and finally 3) writing to the database if the condition evaluates to true.
I assume your put operation looks something like this:
ddbClient.put({
TableName: "YOUR TABLE",
Item: {
PK: "USER#B",
SK: "BUG#2",
Type "Bug",
Description: "My user wasn't created first!"
},
ConditionExpression: "attribute_exists(PK)"
})
In this example, DynamoDB first tries to find the item with PK: "USER#B" SK: "BUG#2", which does not exist. As you're experiencing, this item will not be written to DynamoDB because an item with that primary key does not exist.
The problem you are seeing, as you've alluded to in your question, is that a CondttionExpression applies to only a single item. However, you are trying to conditionally put an item in the database by applying the condition to another item. That is a great candidate for a DynamoDB transaction.
Transactions let you group operations together in an all-or-nothing operation. If one of the operations in your transaction fails, the entire transaction will fail and none of the operations will apply.
You can achieve what you are after by taking this approach
ddbClient.transactWriteItems({
TransactItems=[
{ "PUT":
{
TableName: "YOUR TABLE",
Item: {
PK: "USER#B",
SK: "BUG#2",
Type "Bug"
}
}
},
{ "ConditionCheck":
{
TableName: "YOUR TABLE",
Item: {
PK: "USER#B",
SK: "USER#B"
},
ConditionExpression: "attribute_exists(PK)"
}
}
]
})
In the above transaction, I'm using a ConditionCheck to confirm the existence of a user before entering the bug. If the user does not exist, the transaction will fail and the bug won't be written to DDB.
For a more thorough explanation of DynamoDB Condition Expressions, I highly recommend you check out Understanding DynamoDB Condition Expressions by Alex Debrie.

Azure Cosmos Db document partition key having duplicate, but find duplicate document with combination of other columns

I have below document JSON (pasted partial JSON, actual JSON will be complex and embedded). The JSON has Code as ParitionKey, I am trying to build No SQL database documents by migrating my sql tables, and I will have Code, Type making Unique row, as you can see below Code = 4 is duplicated with different Type and id I just generated GUID (not sure on id field so generated GUID and assigned to it).
we only have two values for Type filed, it's either RI or NRI for entire data, and Code is duplicated like below sample data Code:4, but combination of Type & Code fields make it unique.
Example JSON:
{
"id" : "88725628-2a9a-4fc7-90ed-29c5ffbd45fa"
"Code": "4",
"Type": "RI",
"Description": "MAC/CHEESE ",
},
{
"id" : "88725628-9a3b-4fc7-90ed-29c5ffbd34sk"
"Code": "8",
"Type": "RI",
"Description": "Cereals",
},
{
"id" : "88725628-6d9f-4fc7-90ed-29c4ffbd87de"
"Code": "4",
"Type": "NRI",
"Description": "Christmas Deal",
}
In NoSQL cosmos document db, I couldn't use two columns as partition key, so I have only code as Partition key, but when I am trying to insert into Cosmos Db how do I check if not exists then only insert or else I would end up creating duplicate documents:
CreateItemAsync --> I need a way to check if the document already exists if not then create
I have below code to check and if not found create Item
try
{
// Read the item to see if it exists.
ItemResponse<Item> itemResponse = await this.container.ReadItemAsync<Item>(itm.Id, new PartitionKey(itm.Code));
}
catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
{
// Create an item in the container representing the Andersen family. Note we provide the value of the partition key for this item, which is "Andersen"
ItemResponse<Item> itemResponse = await this.container.CreateItemAsync<Item>(itm, new PartitionKey(itm.Code));
}
But from above code in ReadItemAsync parameters, how do I know id parameter as it is a GUID randomly generated on every insert, is there a better way to utilize id property before insert into Cosmos DB, so it can be utilized while ReadItemAsync ?
second parameter is paritionKey, If I give code as partition key, it wouldn't work as expected as Code can be duplicated with different "Type" values and it's valid, but Code & Type together makes it unique and we shouldn't allow another document to be inserted if code and type are same.
How do I do it in Cosmos db insert ? I have below questions:
id field --> can I generate GUID and save document or id filed has any purpose which can be utilized during reads ?
Is it ok to pick a partition key which can potentially have duplicates like Code field.
How do I check document exists before insert with above qualifiers as Code filed can be duplicated but only With Type it makes it unique ?
Any suggestions ?
If code and type make a unique row then you should use the value of type for id as well rather than generating a GUID because in Cosmos DB the combination of your partition key and id must be unique.
Then when you do an insert, if the data is already there it will throw an exception which you can catch. For reads, if you know the value for code and type, you can use these to perform a point read to get a single row of data, rather than using a query. This is the most efficient way to fetch data in Cosmos DB.
It is fine to have duplicates for partition key values. You only need to make sure that you have less than 20GB of data for each partition key value.

How to update dynamodb attribute based on a condition on another attribute

For ex:
I have:
{
"status": "running",
"ts": "1590560955"
}
If current system ts >= above "ts", then:
{
"status": "completed",
"ts": "1590560955"
}
Is this even possible? Does Lambda function help here? If yes, how?
Yes, it is possible. It is called a "conditional update" - see for example this documentation. You can update one attribute based on the value of a different attribute - if those belong to the same item.
Here is an example (in Python, using Amazon's boto3 library, but you can translate it to whichever language you prefer):
table.update_item(
Key={'p': p},
UpdateExpression='SET status = :completedl',
ConditionExpression='ts > :currentts',
ExpressionAttributeValues={':completed': 'completed', ':currenttts': currentts})
This conditionally updates the item with key p: If ConditionExpression is fullfilled, i.e., the item's ts value is higher than "currentts" (which you'll supply), then the update described in UpdateExpression takes place: This item's status attribute is set to the string 'completed'.

Resources