I'm using AWS Lambdas enabled by Serverless framework with DynamoDB as my primary DB.
Now, out table nomenclature is dependent on the stage to which we deploy our functions to.
For example,
for prod : prod-primary
for test : test-primary
for local : local-primary
In the table we also have 4-5 GSIs.
For our production table, we want to use DynamoDB On-Demand.
But for my testing table on which only I am testing, I would rather like to use ProvisionedCapacity setting to save unnecessary testing costs.
Right now, my table config looks like this :
Resources:
myData:
Type: AWS::DynamoDB::Table
Properties:
TableName: "${opt:stage, 'local'}-primary"
AttributeDefinitions:
- AttributeName: PK
AttributeType: S
- AttributeName: SK
AttributeType: S
- AttributeName: AK
AttributeType: S
- AttributeName: createdBy
AttributeType: S
- AttributeName: itemType
AttributeType: S
- AttributeName: itemStatus
AttributeType: S
KeySchema:
- AttributeName: PK
KeyType: HASH
- AttributeName: SK
KeyType: RANGE
BillingMode: PAY_PER_REQUEST
StreamSpecification:
StreamViewType: NEW_AND_OLD_IMAGES
GlobalSecondaryIndexes:
- IndexName: SK-PK-Index
KeySchema:
- AttributeName: SK
KeyType: HASH
- AttributeName: PK
KeyType: RANGE
Projection:
ProjectionType: ALL
- IndexName: itemType-AK-index
KeySchema:
- AttributeName: itemType
KeyType: HASH
- AttributeName: AK
KeyType: RANGE
Projection:
ProjectionType: ALL
- IndexName: createdBy-itemType-index
KeySchema:
- AttributeName: createdBy
KeyType: HASH
- AttributeName: itemType
KeyType: RANGE
Projection:
ProjectionType: ALL
TimeToLiveSpecification:
AttributeName: expireAt
Enabled: true
This works fine when I need to deploy to prod stage. However, I have to manually replace BillingMode with :
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
And also add this under all the mentioned GSIs. Is there a way to automate this?
Related
First time to dynamodb and serverless framework. I am trying to create a simple todo app. With todoId as primary key and userId as a secondary index. This is my definition of the table in serverless.yaml but when i try to get todo list of the user, i get the above error.
resources:
Resources:
GroupsDynamoDBTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: todoId
AttributeType: S
- AttributeName: userId
AttributeType: S
KeySchema:
- AttributeName: todoId
KeyType: HASH
BillingMode: PAY_PER_REQUEST
TableName: ${self:provider.environment.TODOLIST_TABLE}
GlobalSecondaryIndexes:
- IndexName: ${self:provider.environment.USER_ID_INDEX}
KeySchema:
- AttributeName: userId
KeyType: HASH
Projection:
ProjectionType: ALL
query:
const result = await docClient
.query({
TableName: toDoListTable,
KeyConditionExpression: 'userId = :userId',
ExpressionAttributeValues: {
':userId': 5
},
ScanIndexForward: false
})
.promise()
Since you are making a query with a global secondary index you must specify the name of the index that you want to use and the attributes to be returned in the query results.
The result should be this:
const result = await docClient
.query({
TableName: toDoListTable,
IndexName: "UserIdIndex", // the name specified here: self:provider.environment.USER_ID_INDEX
KeyConditionExpression: "userId = :userId",
ExpressionAttributeValues: {
":userId": 5
},
ProjectionExpression: "todoId, userId",
ScanIndexForward: false
})
.promise()
My DynamoDB table
awsRequestID (S)
ttl (N)
createdate (S) (ISO)
user_id (S)
message (S)
What I try to achieve
I want to have a global secondary index so I can query to filter all the messages of a users and I get them ordered (using the sort key) as explained in https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html
serverless.yml
resources:
Resources:
EventsTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${self:custom.eventsTable}
AttributeDefinitions:
- AttributeName: awsRequestID
AttributeType: S
- AttributeName: ttl
AttributeType: N
- AttributeName: createdate
AttributeType: S
- AttributeName: user_id
AttributeType: S
- AttributeName: message
AttributeType: S
KeySchema:
- AttributeName: awsRequestID
KeyType: HASH
GlobalSecondaryIndexes:
- IndexName: UserIdIndex
KeySchema:
- AttributeName: user_id
KeyType: HASH
- AttributeName: createdate
KeyType: RANGE
Projection:
ProjectionType: 'ALL'
TimeToLiveSpecification:
AttributeName: ttl
Enabled: true
BillingMode: PAY_PER_REQUEST
Error when I deploy
An error occurred: EventsTable - Property AttributeDefinitions is inconsistent with the KeySchema of the table and the secondary indexes.
Apparently the right syntax is this.
These were the errors:
You cannot add the ttl column to AttributeDefinitions (otherwise you get the error of the question)
You must have the columns you need for the global secondary index in the attribute definitions (otherwise you get the exact same error)
You must not have extra columns (message) in the attribute definitions (all give the exact same error message)
resources:
Resources:
EventsTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${self:custom.eventsTable}
AttributeDefinitions:
- AttributeName: awsRequestID
AttributeType: S
- AttributeName: user_id
AttributeType: S
- AttributeName: createdate
AttributeType: S
KeySchema:
- AttributeName: awsRequestID
KeyType: HASH
GlobalSecondaryIndexes:
- IndexName: UserIdIndex
KeySchema:
- AttributeName: user_id
KeyType: HASH
- AttributeName: createdate
KeyType: RANGE
Projection:
ProjectionType: 'ALL'
TimeToLiveSpecification:
AttributeName: ttl
Enabled: true
BillingMode: PAY_PER_REQUEST
Isn't LocalSecondaryIndexes an array? I tried alot of different version of it. With and without a -, indenting lines/props. But I can't get it to work.
This syntax gives me Expected params.LocalSecondaryIndexes to be an Array.
CoordinatesTable:
Type: 'AWS::DynamoDB::Table'
Properties:
KeySchema:
- AttributeName: hashKey
KeyType: HASH
- AttributeName: rangeKey
KeyType: RANGE
AttributeDefinitions:
- AttributeName: hashKey
AttributeType: "N"
- AttributeName: rangeKey
AttributeType: "S"
- AttributeName: geohash
AttributeType: "N"
LocalSecondaryIndexes:
IndexName: geohash-index
KeySchema:
- AttributeName: hashKey
KeyType: HASH
- AttributeName: geohash
KeyType: RANGE
Projection:
ProjectionType: All
ProvisionedThroughput:
ReadCapacityUnits: 10
WriteCapacityUnits: 5
TableName: CoordinatesTable
This is the table in json, take from a github projekt.
{
TableName: config.tableName,
ProvisionedThroughput: {
ReadCapacityUnits: 10,
WriteCapacityUnits: 5
},
KeySchema: [
{
KeyType: "HASH",
AttributeName: config.hashKeyAttributeName
},
{
KeyType: "RANGE",
AttributeName: config.rangeKeyAttributeName
}
],
AttributeDefinitions: [
{ AttributeName: config.hashKeyAttributeName, AttributeType: 'N' },
{ AttributeName: config.rangeKeyAttributeName, AttributeType: 'S' },
{ AttributeName: config.geohashAttributeName, AttributeType: 'N' }
],
LocalSecondaryIndexes: [
{
IndexName: config.geohashIndexName,
KeySchema: [
{
KeyType: 'HASH',
AttributeName: config.hashKeyAttributeName
},
{
KeyType: 'RANGE',
AttributeName: config.geohashAttributeName
}
],
Projection: {
ProjectionType: 'ALL'
}
}
]
};
}
}
I believe the error message is misleading and should rather state that LocalSecondaryIndexes should be a list, because that's what it is in the JSON representation. So making a list out of it in YAML should work fine:
LocalSecondaryIndexes:
- IndexName: geohash-index
KeySchema:
- AttributeName: hashKey
KeyType: HASH
- AttributeName: geohash
KeyType: RANGE
Projection:
ProjectionType: All
I seem to be having a lot of difficulty querying data with AWS.DynamoDB.DocumentClient().get().
I am using Serverless and set up my serverless.yml with this schema:
resources:
Resources:
ShortUrlsTable:
Type: "AWS::DynamoDB::Table"
Properties:
AttributeDefinitions:
- AttributeName: id
AttributeType: S
- AttributeName: longUrl
AttributeType: S
- AttributeName: shortPath
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
GlobalSecondaryIndexes:
- IndexName: longUrlIndex
KeySchema:
- AttributeName: longUrl
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
Projection:
ProjectionType: ALL
- IndexName: shortPathIndex
KeySchema:
- AttributeName: shortPath
KeyType: HASH
Projection:
ProjectionType: ALL
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
TableName: ${self:custom.tableName}
What I want to be able to do is search the DB for a shortUrlItem using either longUrl or shortPath.
So far I have this set up:
dynamoDb = new AWS.DynamoDB.DocumentClient()
app.get("/:longUrl", (req, res) => {
const {longUrl} = req.params
const getParams = {
TableName: SHORT_URLS_TABLE,
Key: {longUrl},
}
dynamoDb.get(getParams, (error, result) => {
res.send({...error, ...result})
})
})
All I seem to be getting is this error message returned to me:
"message":"The provided key element does not match the schema","code":"ValidationException","time":"2018-08-17T20:39:27.765Z","requestId":"4RKNVG7ET1ORVF10H71M7AUABRVV4KQNSO5AEMVJF66Q9ASUAAJG","statusCode":400,"retryable":false,"retryDelay":21.513795782119505,"TableName":"short-urls-table-dev"
I cannot seem to figure out if I am querying correctly or setting up my schema correctly for the secondary index to be the searchable key in my table.
I can see two mistakes
1: your getParams are wrong. You make get request on PK but you provide GSI key in the params section. It should be like
const getParams = {
TableName: SHORT_URLS_TABLE,
Key: {
id: id, // Because id is the attribute of your HASH key.
}
}
This is the reason of the error. Your hash key is not on attribute longUrl.
2: Anyway, you can't make get request on GSI. Its not have GSI's are designed. GSI does not force uniqueness so there can be multiple items in the same GSI Hash key, therefore you can only query instead of get.
What you are trying to do is something like
const queryParams = {
TableName: SHORT_URLS_TABLE,
IndexName: 'longUrlIndex',
KeyConditionExpression: 'longUrl = :longUrlValue',
ExpressionAttributeValues: {
'longUrlValue': longUrl
}
};
dynamoDb.query(queryParams, (error, result) => {
res.send({...error, ...result})
})
I want to set tags for Dynamo DB, did we have a serverless.yml configuration for it like we have it for function
functions:
createUsers:
tags:
key: value
resources:
Resources:
DynamoDbTable:
Type: 'AWS::DynamoDB::Table'
DeletionPolicy: Retain
Properties:
TableName: 'table'
AttributeDefinitions:
-
AttributeName: id
AttributeType: S
KeySchema:
-
AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
Tags:
-
Key: 'tagKey'
Value: 'tagValue'