Is it possible to enforce table level schema validation on DynamoDB
For instance, consider the following table
aws dynamodb create-table\
--table-name spaces-tabs-votes\
--attribute-definitions AttributeName=id,AttributeType=S
--key-schema AttributeName=id,KeyType=HASH
--provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
--endpoint-url http://localhost:8000
At time T, table looked like below (Notice that votes is of type N)
$ aws dynamodb scan --table-name spaces-tabs-votes
{
"Count": 2,
"Items": [
{
"votes": {
"N": "104"
},
"id": {
"S": "space"
}
},
{
"votes": {
"N": "60"
},
"id": {
"S": "tab"
}
}
],
"ScannedCount": 2,
"ConsumedCapacity": null
}
At time T+1, I was able to change the type of votes from N to S. All I had to do was do a put with votes set as String 1 instead of incrementing the already existing N value.
$ aws dynamodb scan --table-name spaces-tabs-votes
{
"Count": 2,
"Items": [
{
"votes": {
"N": "104"
},
"id": {
"S": "space"
}
},
{
"votes": {
"S": "1"
},
"id": {
"S": "tab"
}
}
],
"ScannedCount": 2,
"ConsumedCapacity": null
}
I would like schema enforcement - i.e, if the record that I am trying to post to DynamoDB doesn't conform to a schema, I want it to throw an exception. Is it possible at all?
Related
My table data looks like below one
{
"id": {
"S": "alpha-rocket"
},
"images": {
"SS": [
"apple/value:50",
"Mango/aa:284_454_51.0.0",
"Mango/bb:291",
"Mango/cc:4"
]
},
"product": {
"S": "fruit"
}
}
Below is my code to update table. The variables I am passing to function has values product_id has alpha-rocket, image_val has 284_454_53.0.0 and image has Mango/aa:284_454_53.0.0.
I am trying to update value of Mango/aa from 284_454_51.0.0 to 284_454_53.0.0 but getting an error "The document path provided in the update expression is invalid for update"
def update_player_score(product_id, image_val, image):
dynamo = boto3.resource('dynamodb')
tbl = dynamo.Table('<TableName>')
result = tbl.update_item(
expression_attribute_names: {
"#image_name" => "image_name"
},
expression_attribute_values: {
":image_val" => image_val,
},
key: {
"product" => "fruit",
"id" => product_id,
},
return_values: "ALL_NEW",
table_name: "orcus",
update_expression: "SET images.#image_val = :image_val",
}
Is there a way to update the value of Mango/aa or replace full string "Mango/aa:284_454_51.0.0" to "Mango/aa:284_454_53.0.0"
You cannot update a string in a list by matching the string. If you know the index of it you can replace the value of the string by index:
SET images[1] = : image_val
It seems like maybe what you want is not a list of strings, but another map. So instead of your data looking like it does you'd make it look like this, which would allow you to do the update you're looking for:
{
"id": {
"S": "alpha-rocket"
},
"images": {
"M": {
"apple" : {
"M": {
"value": {
"S": "50"
}
},
"Mango" : {
"M": {
"aa": {
"S": "284_454_51.0.0"
},
"bb": {
"S": "291"
},
"cc": {
"S": "4"
}
}
}
},
"product": {
"S": "fruit"
}
}
I would also consider putting the different values in different "rows" in the table and using queries to build the objects.
Below is what the table structure looks like in DynamoDb when I scan the table using
aws dynamodb scan --table-name "hotel" --endpoint-url http://localhost:8088
{
"Count": 2,
"Items": [
{
"dc": {
"N": "0"
},
"sw": {
"L": [
{
"N": "1"
}
]
}
},
{
"dc": {
"N": "0"
},
"sw": {
"L":[]
},
}
],
"ScannedCount": 2,
"ConsumedCapacity": null
}
Now I want to query the table where sw: []. I am using following query to retrieve the results.
aws dynamodb query --table-name "hotel" --key-conditions file:////tables/key1.json --endpoint-url http://localhost:8088 where
key1.json
{
"sw":{
"ComparisonOperator":"EQ",
"AttributeValueList": [ {"L":[]} ]
}
}
But I get following error
An error occurred (ValidationException) when calling the Query operation: Query condition missed key schema element
Please suggest how can I query the table to retrieve the results.
The way you have structured you table its hard to query the array by its fields.Try to save each Item as a row.
I hope I can express myself a way you can all understand.
Here's the thing, I'm developing a back-end for an app in nodeJS. There's a table named 'chat' in dynamoDB with the ID of the chat's creator and a list of messages IDs, also there's a table named 'messages' with the author of each message and the text.
When the front-end asks the server for all the chats it has to send a list of chats objects with this sctructure:
[
{
"ID_CHAT": {
"S": "asd123asd"
},
"author": {
"name": "Name",
"lastname": "Lastname"
},
"date": {
"S": "19/11/2017 16:00"
},
"USER_DESTINATION": {
"S": "222asd222"
},
"messages": [
{
"Item": {
"author": {
"name": "Name",
"lastname": "Lastname"
},
"text": {
"S": "Hi!"
},
"MESSAGE_ID": {
"S": "456asd456"
},
"date": {
"S": "19/11/2017 16:05"
}
}
},
{
"Item": {
"author": {
"name": "Name",
"lastname": "Lastname"
},
"text": {
"S": "zup?"
},
"MESSAGE_ID": {
"S": "789asd789"
},
"date": {
"S": "19/11/2017 16:06"
}
}
}
]
}
{
"ID_CHAT": {
"S": "123asd123asd"
},
"author": {
"S": "123asd123"
}, and so on...
]
But in order to send this I need to get the chat's authors data from dynamodb (asynchronous function) and the list of messages data again from dynamodb.
I tried combining map() to get the chats authors and Promise.all() for the messages but it won't work cause the second iteration of the map() function runs after the Promise.then().
I'll appretiate your help!
I got a table in dynamodb
aws dynamodb describe-table --table-name work
{
"Table": {
"TableArn": "../work",
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"ProvisionedThroughput": {
"NumberOfDecreasesToday": 0,
"WriteCapacityUnits": 5,
"ReadCapacityUnits": 5
},
"TableSizeBytes": 0,
"TableName": "work",
"TableStatus": "ACTIVE",
"KeySchema": [
{
"KeyType": "HASH",
"AttributeName": "id"
}
],
"ItemCount": 0,
"CreationDateTime": 1505847222.951
}
and my data was like
id(P.K) location
1 location1
2 location1
3 location2
And in my key.json
{
"location": {"S": "location1"}
}
And when I
aws dynamodb delete-item --table-name work --key file://key.json
An error occurred (ValidationException) when calling the DeleteItem operation: The provided key element does not match the schema
{
"id": {"S": "location1"}
}
You key.json is in wrong format.
Check the documentation,
https://docs.aws.amazon.com/cli/latest/reference/dynamodb/delete-item.html
Given the table schema defined below (create-table.json) I am getting the following error after I call put-item using add-event1.json followed by add-event2.json:
A client error (ConditionalCheckFailedException) occurred when calling the PutItem operation: The conditional request failed
Why doesn't the ConditionExpression allow me to write both records? (I expect to have 2 records after the second operation)
I suspected that it was because of the non-key conditions used, but I don't see anything in the docs that indicates a lack of support for non-key conditions.
Create Table
$ aws dynamodb create-table --cli-input-json file://create-table.json
create-table.json
{
"TableName": "EVENTS_TEST",
"KeySchema": [
{ "AttributeName": "aggregateId", "KeyType": "HASH" },
{ "AttributeName": "streamRevision", "KeyType": "RANGE" }
],
"AttributeDefinitions": [
{ "AttributeName": "aggregateId", "AttributeType": "S" },
{ "AttributeName": "streamRevision", "AttributeType": "N" }
],
"ProvisionedThroughput": {
"ReadCapacityUnits": 10,
"WriteCapacityUnits": 10
}
}
Add First Record
$ aws dynamodb put-item --cli-input-json file://add-event1.json
add-event1.json
{
"TableName": "EVENTS_TEST",
"Item": {
"aggregateId": { "S": "id" },
"id": { "S": "119" },
"context": { "S": "*" },
"aggregate": { "S": "*" },
"streamRevision": { "N": "0" },
"commitId": { "S": "1119" },
"commitSequence": { "N": "0" },
"commitStamp": { "N": "1470185631511" },
"dispatched": { "BOOL": false },
"payload": { "S": "{ \"event\": \"bla\" }" }
},
"ExpressionAttributeNames": { "#name": "aggregate" },
"ConditionExpression": "attribute_not_exists(aggregateId) and attribute_not_exists(streamRevision) and #name <> :name and context <> :ctx",
"ExpressionAttributeValues": {
":name": { "S": "*" },
":ctx": { "S": "*" }
}
}
Add Second Record
$ aws dynamodb put-item --cli-input-json file://add-event2.json
add-event2.json
{
"TableName": "EVENTS_TEST",
"Item": {
"aggregateId": { "S": "id" },
"id": { "S": "123" },
"context": { "S": "myCtx" },
"aggregate": { "S": "myAgg" },
"streamRevision": { "N": "0" },
"commitId": { "S": "1123" },
"commitSequence": { "N": "0" },
"commitStamp": { "N": "1470185631551" },
"dispatched": { "BOOL": false },
"payload": { "S": "{ \"event\": \"bla2\" }" }
},
"ExpressionAttributeNames": { "#name": "aggregate" },
"ConditionExpression": "aggregateId <> :id and streamRevision <> :rev and #name <> :name and context <> :ctx",
"ExpressionAttributeValues": {
":id": { "S": "id" },
":rev": { "N": "0" },
":name": { "S": "myAgg" },
":ctx": { "S": "myCtx" }
}
}
Your goal is to save both records. There are 2 issues here
With your choice of hash and range key it is impossible to save both records
The combination of hash and range key make a record unique.
Event 1 and event 2 have the same values for hash and range key.
Therefore the second put-item wil simply overwrite the first record.
Your ConditionExpression prevents the replacement of record 1 by record 2
The ConditionExpression is evaluated just before putting a record. Your expression fails because when your event 2 is about to be inserted, DynamoDB discovers that a record with aggregateId “id1” already exists. The condition fails on "attribute_not_exists(aggregateId)", and you receive the ConditionalCheckFailedException This expression prevents overwriting of record 1 by record 2.
If you want to save both records you will have to come up with another choice of hash key and/or range key that better represents the unicity of your data item. You can not solve that with a ConditionExpression.
I received similar error.
The conditional request failed (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException; Request ID: SMNKMSKJNSHBSGHVGHSB)
In my case I had not added sort key in my table and my second item had the same primary key as of first item. It worked after I added sort key.
SOLUTION
An item with that ID allready exists in the Table.
You need to create a unique ID for the item you try to add.