I'm creating serverless framework project.
DynamoDB table is created by other CloudFormation Stack.
How I can refer existing dynamodb table's StreamArn in serverless.yml
I have configuration as below
resources:
Resources:
MyDbTable: //'arn:aws:dynamodb:us-east-2:xxxx:table/MyTable'
provider:
name: aws
...
onDBUpdate:
handler: handler.onDBUpdate
events:
- stream:
type: dynamodb
arn:
Fn::GetAtt:
- MyDbTable
- StreamArn
EDIT:
- If your tables were created in another Serverless service you can skip steps 1, 4 and 8.
- If your tables were created in a standard CloudFormation Stack, edit this stack to add the outputs from step 2 and skip steps 1, 4 and 8
Stuck with the same issue I came up the following workaround:
Create a new serverless service with only tables in it (you want to make a copy of your existing tables set-up):
service: MyResourcesStack
resources:
Resources:
FoosTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${opt:stage}-${self:service}-foos
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
StreamSpecification:
StreamViewType: NEW_AND_OLD_IMAGES # This enables the table's stream
(Optional) You can use serverless-dynamodb-autoscaling to configure autoscaling from the serverless.yml:
plugins:
- serverless-dynamodb-autoscaling
custom:
capacities:
- table: FoosTable # DynamoDB Resource
read:
minimum: 5 # Minimum read capacity
maximum: 50 # Maximum read capacity
usage: 0.75 # Targeted usage percentage
write:
minimum: 5 # Minimum write capacity
maximum: 50 # Maximum write capacity
usage: 0.75 # Targeted usage percentage
Set up the stack to output the tables name, Arn and StreamArn:
Outputs:
FoosTableName:
Value:
Ref: FoosTable
FoosTableArn:
Value: {"Fn::GetAtt": ["FoosTable", "Arn"]}
FoosTableStreamArn:
Value: {"Fn::GetAtt": ["FoosTable", "StreamArn"]}
Deploy the stack
Copy the data from your old tables to the newly created ones.
To do so, I used this script which works well if the old and new tables are in the same region and if the table are not huge. For larger tables, you may want to use AWS Data Pipeline.
Replace your hardcoded references to your tables in your initial service with the previously outputed variables:
provider:
environment:
stage: ${opt:stage}
region: ${self:provider.region}
dynamoDBTablesStack: "MyResourcesStack-${opt:stage}" # Your resources stack's name and the current stage
foosTable: "${cf:${self:provider.environment.dynamoDBTablesStack}.FoosTableName}"
foosTableArn: "${cf:${self:provider.environment.dynamoDBTablesStack}.FoosTableArn}"
foosTableStreamArn: "${cf:${self:provider.environment.dynamoDBTablesStack}.FoosTableStreamArn}"
functions:
myFunction:
handler: myFunction.handler
events:
- stream:
batchSize: 100
type: dynamodb
arn: ${self:provider.environment.foosStreamArn}
Deploy those changes
Test everything
Backup and delete your old tables
Related
It seems that the default doctrine listener used by FOSElasticaBundle does not support bulk index by default. I have an application where I want to add support for more complex search queries through ElasticSearch. The search engine only will perform queries through one unique entity Post. When I create, edit or delete there is not any problem, the index in elasticsearch is updated automatically through the listener. My problem comes when I want to do bulk updates to hide or show more than one post at once, the listener is not receiving the signal to make the bulk index update in elasticsearch.
I am new to FOSElasticSearch so I do not know if I am missing something. I am using FOSElasticaBundle 6, Symfony 5.2 and ElasticSearch 7
Here you can find my fos_elastica.yaml
fos_elastica:
messenger: ~
clients:
default: { host: 127.0.0.1, port: 9200 }
indexes:
product:
settings:
index:
analysis:
analyzer:
my_analyzer:
type: snowball
language: English
persistence:
# the driver can be orm, mongodb or phpcr
driver: orm
model: App\Entity\Post
listener: { enabled: true }
provider: ~
finder: ~
elastica_to_model_transformer:
ignore_missing: true
properties:
title: { boost: 10, analyzer: my_analyzer }
tags: { boost: 8, analyzer: my_analyzer }
description: { boost: 6, analyzer: my_analyzer }
ispublished: { type: integer}
And here you can find the way I am updating more than once entity element at once in PostRepository (the function is to update all post from one unique author, it is just an example):
public function bulkUpdate($ispublished, $authorid){
return $this->createQueryBuilder('p')
->update()
->set('p.ispublished', $ispublished)
->andWhere('p.authorid = :id')
->setParameter('id', $authorid)
->getQuery()
->execute();
}
Also I found that I could disable default listener, dispatch messages for each create, update or delete action through symfony/messenger and consume them async in the background. I guess that I should create my own handler and dispatch specific messages (although I could not find an example about this in the doc) in each modifying action, although at the end I also have the same problem, as I do not know how to send a bulk index update/delete action
In the other hand I was thinking in executing all time a background script in python to check what rows were modified in mysql database and update those index with the script directly through ElasticSearch Api
I do not think that I will need to update more than 1k posts at once, so I would like to keep using the default listener to update posts automatically and avoid gaps between that an entity is modified and the index is updated in ElasticSearch. I just need to find the best way to update indexes in bulk as I have everything else already implemented and working
Sorry for all the text but I wanted to give all details about what I need to do
I am using the serverless framework to create a DynamoDB table and then I want to access it from a Lambda function.
In the serverless.yml file I have the definitions below for the environment variable and CF resources.
What I was expecting was a table with the name accounts-api-dev-accounts, but what the cloudformation stack is creating for me is accounts-api-dev-accounts-SOME_RANDOM_LETTERS_AND_NUMBERS_SUFFIX.
In my lambda function the environment variable DYNAMODB_ACCOUNTS_TABLE_NAME is exposed to the function without the SOME_RANDOM_LETTERS_AND_NUMBERS_SUFFIX part. Is the CF stack supposed to add a random suffix? How do I actually retrieve the right table name?
service:
name: accounts-api
provider:
...
stage: ${opt:stage, 'dev'}
environment:
DYNAMODB_ACCOUNTS_TABLE_NAME: '${self:service}-${self:provider.stage}-accounts'
And the following CF resource:
Resources:
AccountsTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${env:DYNAMODB_ACCOUNTS_TABLE_NAME}
AttributeDefinitions:
- AttributeName: customerNumber
AttributeType: S
- AttributeName: accountNumber
AttributeType: S
KeySchema:
- AttributeName: customerNumber
KeyType: HASH
- AttributeName: accountNumber
KeyType: RANGE
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
Maybe the environment variables are not updated yet at the time of the creation of the table definition? I'm not sure.
Try ${self:provider.environment.DYNAMODB_ACCOUNTS_TABLE_NAME} instead of ${env:DYNAMODB_ACCOUNTS_TABLE_NAME}.
I haven't seen this behavior yet (random characters after deploy), it could be a way to force uniqueness when the table has to be replaced. You could use another environment variable and have the value populated by the Table resource's output. That way, CloudFormation will inject the actual resource name to the Lambda environment variable. I haven't tried this, but this would be my first "go to".
environment:
DYNAMODB_ACCOUNTS_TABLE_NAME: '${self:service}-${self:provider.stage}-accounts'
ACTUAL_DYNAMODB_ACCOUNTS_TABLE_NAME:
Ref: AccountsTable
I am trying to add autoscaling to multiple DDB tables, since all the tables would have the same traffic pattern the autoscaling configuration for them would be identical.
I know of a way to do that by creating ScalableTargets for each table, but since I want all tables to use the same configuration I was wondering if it is possible to re-use the scalable targets
WriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: 30
MinCapacity: 5
ResourceId:
Fn::Join:
- /
- - table
- Ref: FooTable
RoleARN:
Fn::GetAtt: DynamoDBAutoScalingRole.Arn
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
WriteScalingPolicy:
Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
Properties:
PolicyName: WriteAutoScalingPolicy
PolicyType: TargetTrackingScaling
ScalingTargetId:
Ref: WriteCapacityScalableTarget
TargetTrackingScalingPolicyConfiguration:
TargetValue: 80
ScaleInCooldown: 10
ScaleOutCooldown: 10
PredefinedMetricSpecification:
PredefinedMetricType: DynamoDBWriteCapacityUtilization
DynamoDBAutoScalingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- application-autoscaling.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'dynamodb:DescribeTable'
- 'dynamodb:UpdateTable'
- 'cloudwatch:PutMetricAlarm'
- 'cloudwatch:DescribeAlarms'
- 'cloudwatch:GetMetricStatistics'
- 'cloudwatch:SetAlarmState'
- 'cloudwatch:DeleteAlarms'
Resource: "*"
This works fine to enable autoscaling on a single table but when I try the same with multiple tables it fails, I am trying this like -
WriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: 30
MinCapacity: 5
ResourceId:
Fn::Join:
- /
- - table
- Ref: FooTable
- table
- Ref: BarTable
RoleARN:
Fn::GetAtt: DynamoDBAutoScalingRole.Arn
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
The above target fails to create, we can re-use the scalable targets for indexes in a same way but is it also possible for multiple tables?
If yes, how to achieve that?
I am looking for the best way of creating a stack, in a number of separate steps.
I would like in the first template, to only get up the compute nodes and the network configuration.
In the second template, I would like to create the storage nodes and attach them to the already existing compute nodes.
What do you think is the best way to do this?
Following is one possible approach.
1) Define first template for your compute nodes and network configuration. But define outputs in your first template to expose your compute node IDs. For example, if you create a OS::Nova::Server with name mynode1, you can expose its ID as the output for that template as follows:
outputs:
mynode1_id:
description: ID of mynode1
value: {getattr: [mynode1, id]}
Once you instantiate a heat stack, say mystack1, with this first template, then you can access the ID of mynode1 as follows:
heat output-show mystack1 mynode1_id
2) Create your second template for storage with IDs of your compute nodes from step1 as input parameters. For example:
parameters:
mynode1_id:
type: string
description: ID for mynode1
Then you can use that in your "resources:" section as follows:
resources:
...
...
my_volume_attach:
type: OS::Cinder::VolumeAttachment
properties:
instance_uuid: {get_param: mynode1_id}
...
3) Invoke your second heat stack creation as follows:
heat stack-create -f second-template.yaml -P mynode1_id=`heat output-show mystack1 mynode1_id` mystack2
You might also want to define dependencies between your resources, using the depends_on attribute.
From your description it doesn't seem like using several templates is the correct solution.
for example - if you want objects 3,4 created after objects 1,2, you can define a template as follows:
heat_template_version: '2015-10-15'
parameters:
param1:
type: string
description: just an example of parameter
resources:
object1:
type: OS::Neutron::XXX
properties:
property: XXX
description: object1
object2:
type: OS::Neutron::XXX
properties:
property: XXX
description: object2
object3:
type: OS::Nova::XXX
properties:
property: XXX
description: object3
depends_on: object1
object4:
type: OS::Nova::XXX
properties:
property: XXX
description: object4
depends_on: object1
In my project based on Symfony 2 + Doctrine 2 I implement model with following approach (based on FOSUserBundle source code):
All classes which belong to model are in "Model" folder of my bundle
In "Entity" folder I have classes which extend classes from Model
In "Resources/config/doctrine/" I have files with mappings in YAML format
Important: Classes for persistance (Entities) for which I want STI extend respective classes from Model, not from Entity.
The problem:
#Resources/config/doctrine/ContentElement.orm.yml
My\CoreBundle\Entity\ContentElement:
type: entity
table: content_element
inheritanceType: SINGLE_TABLE
discriminatorColumn:
name: discr
type: string
length: 255
discriminatorMap:
contentElement: ContentElementList
htmlContentElement: HtmlContentElement
id:
id:
type: integer
generator: { strategy: AUTO }
fields:
anchor_id:
type: string
anchor_text:
type: string
#Resources/config/doctrine/HtmlContentElement.orm.yml
My\CoreBundle\Entity\HtmlContentElement:
type: entity
fields:
html:
type: text
When I try to update database I've got errors from YAML driver until I specify additionally 'id' (which should be inherited as I thought)
After adding mapping for id I have sql queries in which I see 2 separate tables for each entity.
I suspect that this happens because HtmlContentElement extends Model\HtmlContentElement but not Entity\ContentElement.
Am I right and is there known solution to my problem?