I am trying to add autoscaling to multiple DDB tables, since all the tables would have the same traffic pattern the autoscaling configuration for them would be identical.
I know of a way to do that by creating ScalableTargets for each table, but since I want all tables to use the same configuration I was wondering if it is possible to re-use the scalable targets
WriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: 30
MinCapacity: 5
ResourceId:
Fn::Join:
- /
- - table
- Ref: FooTable
RoleARN:
Fn::GetAtt: DynamoDBAutoScalingRole.Arn
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
WriteScalingPolicy:
Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
Properties:
PolicyName: WriteAutoScalingPolicy
PolicyType: TargetTrackingScaling
ScalingTargetId:
Ref: WriteCapacityScalableTarget
TargetTrackingScalingPolicyConfiguration:
TargetValue: 80
ScaleInCooldown: 10
ScaleOutCooldown: 10
PredefinedMetricSpecification:
PredefinedMetricType: DynamoDBWriteCapacityUtilization
DynamoDBAutoScalingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- application-autoscaling.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'dynamodb:DescribeTable'
- 'dynamodb:UpdateTable'
- 'cloudwatch:PutMetricAlarm'
- 'cloudwatch:DescribeAlarms'
- 'cloudwatch:GetMetricStatistics'
- 'cloudwatch:SetAlarmState'
- 'cloudwatch:DeleteAlarms'
Resource: "*"
This works fine to enable autoscaling on a single table but when I try the same with multiple tables it fails, I am trying this like -
WriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: 30
MinCapacity: 5
ResourceId:
Fn::Join:
- /
- - table
- Ref: FooTable
- table
- Ref: BarTable
RoleARN:
Fn::GetAtt: DynamoDBAutoScalingRole.Arn
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
The above target fails to create, we can re-use the scalable targets for indexes in a same way but is it also possible for multiple tables?
If yes, how to achieve that?
Related
I have micrnaut based java application. which uses two databases, i have set up the two datasources
but how can i use flyway to load different data into the two databases
directory is as follows, which works for single db as following examples:
resources -> db->migration -> V1__data.sql
Assuming you have two datasources as below:
datasources:
default:
url: 'jdbc:h2:mem:flywayDb1;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE'
username: 'sa'
password: ''
driverClassName: 'org.h2.Driver'
other:
url: 'jdbc:h2:mem:flywayDb2;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE'
username: 'sa'
password: ''
driverClassName: 'org.h2.Driver'
You can configure Flyway as below:
flyway:
datasources:
default:
enabled: true
locations:
- classpath:db/migrations/default
other:
enabled: true
locations:
- classpath:db/migrations/other
Check the micronaut Flyway manual for more information.
Does anyone have a gremlin-config.yaml file that would more accurately reflect how Gremlin acts in Neptune?
I am trying to run as much as I can using a local docker container, and I've substituted properties like gremlin.tinkergraph.vertexIdManager=ANY so that the Vertex IDs can be strings.
But I'm still missing details like multiple labels, which I think is only available via the Neo4Js config, unsure what else this will change.
But yeah, generally looking for a config that represents how Neptune functions as closely as possible
Current:
host: 172.17.0.2
port: 8182
evaluationTimeout: 30000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
graph: conf_local/tinkergraph-custom.properties}
scriptEngines: {
gremlin-groovy: {
plugins: { org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/empty-sample.groovy]}}}}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0] }} # application/json
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 } # application/vnd.graphbinary-v1.0
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }} # application/vnd.graphbinary-v1.0-stringd
processors:
- { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
- { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
consoleReporter: {enabled: true, interval: 180000},
csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: true},
slf4jReporter: {enabled: true, interval: 180000}}
strictTransactionManagement: false
idleConnectionTimeout: 0
keepAliveInterval: 0
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 10485760
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
enabled: false}
gremlin.tinkergraph.vertexIdManager=ANY
gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
version: '3'
services:
gremlin-server:
container_name: gwent_onboarding_neptune
image: tinkerpop/gremlin-server:3.5.0
user: $USER
volumes:
- ./configuration/gremlin-conf:/opt/gremlin-server/conf_local
# - ./gremlin-console/data:/gremlin-server/scripts
ports:
- 8182:8182
command: ./conf_local/gremlin-serverr.yaml
Edit:
Started down this rabbit hole when I was trying to get tinkerpop to work with multiple labels
It looks as if you have already done the things I usually recommend (such as enabling and using strings for the ID values). Given TinkerGraph does not support transactions currently, to simulate those you would likely need to consider using something like JanusGraph running in "inmemory" mode via a Gremlin Server. For the remaining differences, at the present time, it's mostly a case of avoiding any unsupported features such as meta properties in the code and queries that you write. Other than multiple labels as you mentioned, you should be able to do a lot of development and testing of queries locally but, of course, you will at some point still need to test against an actual Neptune cluster to verify the workloads that you have behave as expected when using the full Neptune cluster architecture.
I'm creating serverless framework project.
DynamoDB table is created by other CloudFormation Stack.
How I can refer existing dynamodb table's StreamArn in serverless.yml
I have configuration as below
resources:
Resources:
MyDbTable: //'arn:aws:dynamodb:us-east-2:xxxx:table/MyTable'
provider:
name: aws
...
onDBUpdate:
handler: handler.onDBUpdate
events:
- stream:
type: dynamodb
arn:
Fn::GetAtt:
- MyDbTable
- StreamArn
EDIT:
- If your tables were created in another Serverless service you can skip steps 1, 4 and 8.
- If your tables were created in a standard CloudFormation Stack, edit this stack to add the outputs from step 2 and skip steps 1, 4 and 8
Stuck with the same issue I came up the following workaround:
Create a new serverless service with only tables in it (you want to make a copy of your existing tables set-up):
service: MyResourcesStack
resources:
Resources:
FoosTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${opt:stage}-${self:service}-foos
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
StreamSpecification:
StreamViewType: NEW_AND_OLD_IMAGES # This enables the table's stream
(Optional) You can use serverless-dynamodb-autoscaling to configure autoscaling from the serverless.yml:
plugins:
- serverless-dynamodb-autoscaling
custom:
capacities:
- table: FoosTable # DynamoDB Resource
read:
minimum: 5 # Minimum read capacity
maximum: 50 # Maximum read capacity
usage: 0.75 # Targeted usage percentage
write:
minimum: 5 # Minimum write capacity
maximum: 50 # Maximum write capacity
usage: 0.75 # Targeted usage percentage
Set up the stack to output the tables name, Arn and StreamArn:
Outputs:
FoosTableName:
Value:
Ref: FoosTable
FoosTableArn:
Value: {"Fn::GetAtt": ["FoosTable", "Arn"]}
FoosTableStreamArn:
Value: {"Fn::GetAtt": ["FoosTable", "StreamArn"]}
Deploy the stack
Copy the data from your old tables to the newly created ones.
To do so, I used this script which works well if the old and new tables are in the same region and if the table are not huge. For larger tables, you may want to use AWS Data Pipeline.
Replace your hardcoded references to your tables in your initial service with the previously outputed variables:
provider:
environment:
stage: ${opt:stage}
region: ${self:provider.region}
dynamoDBTablesStack: "MyResourcesStack-${opt:stage}" # Your resources stack's name and the current stage
foosTable: "${cf:${self:provider.environment.dynamoDBTablesStack}.FoosTableName}"
foosTableArn: "${cf:${self:provider.environment.dynamoDBTablesStack}.FoosTableArn}"
foosTableStreamArn: "${cf:${self:provider.environment.dynamoDBTablesStack}.FoosTableStreamArn}"
functions:
myFunction:
handler: myFunction.handler
events:
- stream:
batchSize: 100
type: dynamodb
arn: ${self:provider.environment.foosStreamArn}
Deploy those changes
Test everything
Backup and delete your old tables
I am trying to create a cluster using Heat Templates of Openstack. I have following template defining my resource group.
cluster:
type: OS::Heat::ResourceGroup
properties:
count: { get_param: instance_count }
resource_def:
type: ../templates/vm.yaml
properties:
image: { get_param: image }
flavor: { get_param: flavor }
private_network : { get_attr : [network, name] }
This works, but the name of all these servers is very cryptic. I was wondering if it would be possible to provide a prefix to name to each of the instances.
Or another way could be is I can str_replace a template value with the current index of the cluster count.
Is there a way to achieve this?
Nevermind, got it from the ResourceGroup documentation. Use %index%.
Here is the example from the documentation.
resources:
my_indexed_group:
type: OS::Heat::ResourceGroup
properties:
count: 3
resource_def:
type: OS::Nova::Server
properties:
# create a unique name for each server
# using its index in the group
name: my_server_%index%
image: CentOS 6.5
flavor: 4GB Performance
I am looking for the best way of creating a stack, in a number of separate steps.
I would like in the first template, to only get up the compute nodes and the network configuration.
In the second template, I would like to create the storage nodes and attach them to the already existing compute nodes.
What do you think is the best way to do this?
Following is one possible approach.
1) Define first template for your compute nodes and network configuration. But define outputs in your first template to expose your compute node IDs. For example, if you create a OS::Nova::Server with name mynode1, you can expose its ID as the output for that template as follows:
outputs:
mynode1_id:
description: ID of mynode1
value: {getattr: [mynode1, id]}
Once you instantiate a heat stack, say mystack1, with this first template, then you can access the ID of mynode1 as follows:
heat output-show mystack1 mynode1_id
2) Create your second template for storage with IDs of your compute nodes from step1 as input parameters. For example:
parameters:
mynode1_id:
type: string
description: ID for mynode1
Then you can use that in your "resources:" section as follows:
resources:
...
...
my_volume_attach:
type: OS::Cinder::VolumeAttachment
properties:
instance_uuid: {get_param: mynode1_id}
...
3) Invoke your second heat stack creation as follows:
heat stack-create -f second-template.yaml -P mynode1_id=`heat output-show mystack1 mynode1_id` mystack2
You might also want to define dependencies between your resources, using the depends_on attribute.
From your description it doesn't seem like using several templates is the correct solution.
for example - if you want objects 3,4 created after objects 1,2, you can define a template as follows:
heat_template_version: '2015-10-15'
parameters:
param1:
type: string
description: just an example of parameter
resources:
object1:
type: OS::Neutron::XXX
properties:
property: XXX
description: object1
object2:
type: OS::Neutron::XXX
properties:
property: XXX
description: object2
object3:
type: OS::Nova::XXX
properties:
property: XXX
description: object3
depends_on: object1
object4:
type: OS::Nova::XXX
properties:
property: XXX
description: object4
depends_on: object1