Problem:
The AWS Amplify CLI does not offer options to move away from "on-demand" usage, to provisioned for DynamoDB tables. Thus, I would like to specify this information in the CloudFormation stack.
What is the best practise for this? I would like to sustain the changes written as infrastructure as code for future deployments.
Here are the steps:
Go to your amplify/backend/api/[projectName]/parameters.json folder
Find the DynamoDBBillingMode and change it to "PAY_PER_REQUEST"
Add "DynamoDBModelTableReadIOPS": 1, below DynamoDBBillingMode to set the reads for all tables
Add "DynamoDBModelTableWriteIOPS": 1, below DynamoDBModelTableReadIOPS to set the writes for all tables
These three key/value pairs should be in the route table.
Final JSON:
{
"AppSyncApiName": "name",
"DynamoDBBillingMode": "PROVISIONED",
"DynamoDBModelTableReadIOPS": 1,
"DynamoDBModelTableWriteIOPS": 1,
...
}
How To Customise Reads and Writes Per Table:
Create an override.ts file in the same folder as your parameters.json file
Specify your billing mode like this example: resources.models["Contact"].modelDDBTable.billingMode = "PROVISIONED";
Specify your read and write capacity units like this example: resources.models["Contact"].modelDDBTable.provisionedThroughput = { "readCapacityUnits": 1, "writeCapacityUnits": 1 };
Final JSON:
{
resources.models["Contact"].modelDDBTable.billingMode = "PROVISIONED";
resources.models["Contact"].modelDDBTable.provisionedThroughput = { "readCapacityUnits": 1, "writeCapacityUnits": 1 };
}
Related
I'm using the Cosmos Data migration tool to migrate data between environments. During my migration, I need to update the hostname of the website in the data. I was able to do this pretty easily with a query like this with the top level object data:
SELECT Farms["name"], Farms["farmerInfo"], REPLACE(Farms["websiteLink"], "thiswebsite", "newHostName") AS websiteLink FROM Farms
My Cosmos DB data is structured like (data is just for the example):
{
"name": "Red's Farm",
"websiteLink": "www.thiswebsite.com/goats/",
"farmerInfo": {
"name": "Bob",
"websiteLink": "www.thiswebsite.com/goats/",
"hasGoats": true,
"numGoats": 17
}
}
I don't actually need to modify any of the top level data. The data I need to modify is within the "farmerInfo" object. I've tried a few things but I've had no luck. How can I replace a string in this object using the SQL api?
I want the data to look like this after the migration:
{
"name": "Red's Farm",
"websiteLink": "www.thiswebsite.com/goats/",
"farmerInfo": {
"name": "Bob",
"websiteLink": "www.newHostName.com/goats/", <--- Updated data
"hasGoats": true,
"numGoats": 17
}
}
You can use a SELECT statement in your SELECT statement to build up the sub objects. As example:
SELECT
c.name,
c.websiteLink,
(
SELECT
c.farmerInfo.name,
REPLACE(c.farmerInfo.websiteLink, "thiswebsite", "newHostName") AS websiteLink
) AS farmerInfo
FROM c
I have a table with PK (String) and SK (Integer) - e.g.
PK_id SK_version Data
-------------------------------------------------------
c3d4cfc8-8985-4e5... 1 First version
c3d4cfc8-8985-4e5... 2 Second version
I can do a conditional insert to ensure we don't overwrite the PK/SK pair using ConditionalExpression (in the GoLang SDK):
putWriteItem := dynamodb.Put{
TableName: "example_table",
Item: itemMap,
ConditionExpression: aws.String("attribute_not_exists(PK_id) AND attribute_not_exists(SK_version)"),
}
However I would also like to ensure that the SK_version is always consecutive but don't know how to write the expression. In pseudo-code this is:
putWriteItem := dynamodb.Put{
TableName: "example_table",
Item: itemMap,
ConditionExpression: aws.String("attribute_not_exists(PK_id) AND attribute_not_exists(SK_version) **AND attribute_exists(SK_version = :SK_prev_version)**"),
}
Can someone advise how I can write this?
in SQL I'd do something like:
INSERT INTO example_table (PK_id, SK_version, Data)
SELECT {pk}, {sk}, {data}
WHERE NOT EXISTS (
SELECT 1
FROM example_table
WHERE PK_id = {pk}
AND SK_version = {sk}
)
AND EXISTS (
SELECT 1
FROM example_table
WHERE PK_id = {pk}
AND SK_version = {sk} - 1
)
Thanks
A conditional check is applied to a single item. It cannot be spanned across multiple items. In other words, you simply need multiple conditional checks. DynamoDb has transactWriteItems API which performs multiple conditional checks, along with writes/deletes. The code below is in nodejs.
const previousVersionCheck = {
TableName: 'example_table',
Key: {
PK_id: 'prev_pk_id',
SK_version: 'prev_sk_version'
},
ConditionExpression: 'attribute_exists(PK_id)'
}
const newVersionPut = {
TableName: 'example_table',
Item: {
// your item data
},
ConditionExpression: 'attribute_not_exists(PK_id)'
}
await documentClient.transactWrite({
TransactItems: [
{ ConditionCheck: previousVersionCheck },
{ Put: newVersionPut }
]
}).promise()
The transaction has 2 operations: one is a validation against the previous version, and the other is an conditional write. Any of their conditional checks fails, the transaction fails.
You are hitting your head on some of the differences between a SQL and a no-SQL database. DynamoDB is, of course, a no-SQL database. It does not, out of the box, support optimistic locking. I see two straight forward options:
Use a software layer to give you locking on your DynamoDB table. This may or may not be feasible depending on how often updates are made to your table. How fast 'versions' are generated and the maximum time your application can be gated on the lock will likely tell you if this can work foryou. I am not familiar with Go, but the Java API supports this. Again, this isn't a built-in feature of DynamoDB. If there is no such Go API equivalent, you could use the technique described in the link to 'lock' the table for updates. Generally speaking, locking a no-SQL DB isn't a typical pattern as it isn't exactly what it was created to do (part of which is achieving large scale on unstructured documents to allow fast access to many consumers at once)
Stop using an incrementor to guarantee uniqueness. Typically, incrementors are frowned upon in DynamoDB, in part due to the lack of intrinsic support for it and in part because of how DynamoDB shards you don't want a lot of similarity between records. Using a UUID will solve the uniqueness problem, but if you are porting an existing application that means more changes to the elements that create that ID and updates to reading the ID (perhaps to include a creation-time field so you can tell which is the newest, or the prepending or appending of an epoch time to the UUID to do the same). Here is a pertinent link to a SO question explaining on why to use UUIDs instead of incrementing integers.
Based on Hung Tran's answer, here is a Go example:
checkItem := dynamodb.TransactWriteItem{
ConditionCheck: &dynamodb.ConditionCheck{
TableName: "example_table",
ConditionExpression: aws.String("attribute_exists(pk_id) AND attribute_exists(version)"),
Key: map[string]*dynamodb.AttributeValue{"pk_id": {S: id}, "version": {N: prevVer}},
},
}
putItem := dynamodb.TransactWriteItem{
Put: &dynamodb.Put{
TableName: "example_table",
ConditionExpression: aws.String("attribute_not_exists(pk_id) AND attribute_not_exists(version)"),
Item: data,
},
}
writeItems := []*dynamodb.TransactWriteItem{&checkItem, &putItem}
_, _ = db.TransactWriteItems(&dynamodb.TransactWriteItemsInput{TransactItems: writeItems})
I'm new to NoSQL database. Currently I'm trying to use the Firebase and integrate it with iOS. When it comes to predefine the database, with trial and error, I try to make it look like this:
When I tried to retrieve the "stories" path in iOS, I get json structure like this:
[
<null>,
{
comments: [
<null>,
1,
2,
3
],
desc: "Blue versus red in a classic battle of good versus evil and right versus wrong.",
duration: 30,
rating: 4.42,
tags: [
<null>,
"fantasy",
"scifi"
title: "The Order of the Midnight Sun",
writer: 1
]
}
]
My question is, why there's always a null at the beginning of each array? What should I do in the database editor to avoid the null?
It looks like you start pushing data to index 1 and not 0, inserting/retrieving data to/from a list starts with index 0:
In my app I have some videos and users can like it. video metadata saves in the node "videos", something like this:
videos: {
xxxxx: {
name: "Funny video",
likes: 255,
....
}
}
And I have another node "userVideos" only with video ids.
userVideos: {
xxxxx: true,
yyyyy: true,
}
In the UI I want to show the ranking of the video. For example, I have 5 videos: v01 has 5 likes, v02 - 10, v03 - 1, v04 - 100, v05 - 3.
So, when I get from userVideos the video with id "v03", I should know that his ranking is 5 ('cause if I get the videos list with orderBy DESC my videos node, the position of v03 is 5).
I think that the cloud function should calculate this ranking.
My question is next: is I have a possibility to get the position in the sorted node by key?
Realtime Database queries can't tell you the position of a child snapshot in a sorted query. You have to write code to maintain that data for yourself.
A single terraform alarm looks something like this:
resource "aws_cloudwatch_metric_alarm" "ecs_cpu_reservation" {
alarm_name = "ecs-cpu-reservation-${var.environment}"
alarm_description = "my description"
namespace = "AWS/ECS"
metric_name = "CPUReservation"
dimensions {
ClusterName = "${var.environment}"
}
statistic = "Average"
period = "300"
evaluation_periods = "${var.acceptable_cpu_reservation_eval_period}"
comparison_operator = "GreaterThanOrEqualToThreshold"
threshold = "${var.acceptable_cpu_reservation}"
alarm_actions = ["${data.terraform_remote_state.vpc.my_topic_arn}"]
actions_enabled = "${var.alerting_enabled}"
}
I have 10 alarms per table, and 50 tables.
Therefore, the tf file will contain 500 of those resource blocks. That's a huge file!
The vast majority of the alarms are identical... the only difference being what table the alarm is for.
Is there a way to loop over a table name list and create the alarms?
From what I read, using the "count" variable (or iterating over a list) will lead to maintenance nightmares.
One other option to avoid looping would be to wrap things with a module so you need to source that just once to get everything inside the module.
So if you had a module that looked like this:
variable "dynamodb_table_name" {}
variable "consumed_read_units_threshold" {}
variable "consumed_write_units_threshold" {}
...
resource "aws_cloudwatch_metric_alarm" "consumed_read_units" {
alarm_name = "dynamodb_${var.dynamodb_table_name}_consumed_read_units"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = "2"
metric_name = "ConsumedReadCapacityUnits"
namespace = "AWS/DynamoDB"
period = "120"
statistic = "Average"
dimensions {
TableName = "${var.dynamodb_table_name}"
}
threshold = "${var.consumed_read_units_threshold}"
alarm_description = "This metric monitors DynamoDB ConsumedReadCapacityUnits for ${var.dynamodb_table_name}"
insufficient_data_actions = []
}
resource "aws_cloudwatch_metric_alarm" "consumed_write_units" {
alarm_name = "dynamodb_${var.dynamodb_table_name}_consumed_write_units"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = "2"
metric_name = "ConsumedWriteCapacityUnits"
namespace = "AWS/DynamoDB"
period = "120"
statistic = "Average"
dimensions {
TableName = "${var.dynamodb_table_name}"
}
threshold = "${var.consumed_write_units_threshold}"
alarm_description = "This metric monitors DynamoDB ConsumedWriteCapacityUnits for ${var.dynamodb_table_name}"
insufficient_data_actions = []
}
...
You could define all of your DynamoDB table metrics in a single place and then source that whenever you create your DynamoDB table and they would all get metric alarms for every thing you care about. It's then easy to add/remove/modify these alarms in a single place (the module) and have that automatically applied to your tables on the next terraform apply.
Ideally you'd be able to create a single DynamoDB module that created the table as well as the alarms at the same time but the DynamoDB table resource isn't particularly flexible so it would be a nightmare to design a module that would allow for the flexibility you need (different defined attributes and indexes mostly).
If you wanted you could combine this with looping to reduce some of the module code in exchange for issues around changing the looped things because Terraform will see the resources for things changing and force recreations of even things that shouldn't be affected. In the case where you are just creating alarms (nothing stateful or needs to be up 100%) this isn't a huge issue but be aware that it could require a second terraform apply to fully apply changes done like that.