I have a DynamoDB GSI with only certain fields in the projection(attributes). I would like to add a new field to this list of attributes. Is it possible to do this without deleting the GSI and recreating it? I did not find an option to do that in DynamoDB console or in update_table cli.
According to UpdateTable API it is only possible to create and delete GSI. So to update the current GSI, the old GSI needs to be deleted and recreated again.
Related
I have one table, name is Ticket
Ticket {
id,
usage,
affiliationOrganization
createdAt,
....
}
GSI1:
PartionKey: usage
SortKey: affiliationOrganization
After a development time, i want update SortKey of GSI1
PartionKey: usage
SortKey: affiliationOrganization#createdAt
But after modifying the GSI, the old data is no longer in this GSI table.
Only when adding new data will these new data be automatically typed into the GSI table.
(My english is stupid, I'm sorry if I wrote it wrong)
I want to automatically update old data to GSI after it is modified
Avoid data loss in GSI table, query is not missing data
The old data will automatically replicate to the GSI should it be eligible. Please ensure all your old items have an attribute named affiliationOrganization#createdAt.
Note that this must be a single attribute, DynamoDB does not combine attributes automatically, you must do so manually.
I need to change the index column of GlobalSecondaryIndex in DynamoDB. At first, I just change the "AttributeName" of GSI. It didn't work. Then I researched online and found that I should delete the current GSI and create a new GSI. Here, I use IndexStatus: DELETING to remove the current GSI. However, it always showed error:Encountered unsupported property IndexStatus. So how to fix this error? I'm coding in yaml.
To change the column of a GSI you'll have to use a two step process.
Remove the GSI from your table.
Add the GSI back to the table with the new column.
Because your question is tagged with yaml I assume you are using CloudFormation, or some form of IaC. You'll need to remove the GSI, deploy the change without the GSI, add the GSI with the new column, and deploy that.
A safer approach might be to create a new GSI with the new column.
Add the new GSI with the new column and deploy the table
Update your code to use the new GSI and deploy the new code
Remove the old GSI with the old column and deploy the table
I want to migrate my data from DynamoDb to Redshift. I dont want to scan the whole table at once as this might result in throttling.
My Table is as below:
acountId(hash key), lastUpdatedTime.
I thought I can create GSI on lastUpdatedTime and then I can query like give me the data between day1 to day5. Again next day I can do give me data between day6 to day7.
But even with GSI my understanding is that It will scan the whole table As I wont have any hash key to provide. I just have some range of timestamp to query.
Creating a GSI is the right solution indeed. However the GSI creation operation might be a bit slow/expensive if you set GSI to project all attributes. I would recommend creating the GSI on lastUpdatedTime, and project only the partition key (and order key if you have one) using KEYS_ONLY. Then, when you scan, you will only retrieve the item keys and query the item there and then, when migrating.
I recommend reading up on GSIs here: https://docs.aws.amazon.com/fr_fr/amazondynamodb/latest/developerguide/GSI.html
First of all, I have table structure like this,
Users:{
UserId
Name
Email
SubTable1:[{
Column-111
Column-112
},
{
Column-121
Column-122
}]
SubTable2:[{
Column-211
Column-212
},
{
Column-221
Column-222
}]
}
As I am new to DynamoDB, so I have couple of questions regarding this as follows:
1. Can I create structure like this?
2. Can we set primary key for subtables?
3. Luckily, I found DynamoDB helper class to do some operations into my DB.
https://www.gopiportal.in/2018/12/aws-dynamodb-helper-class-c-and-net-core.html
But, don't know how to fetch only perticular subtable
4. Can we fetch only specific columns from my main table? Also need suggestion for subtables
Note: I am using .net core c# language to communicate with DynamoDB.
Can I create structure like this?
Yes
Can we set primary key for subtables?
No, hash key can be set on top level scalar attributes only (String, Number etc.)
Luckily, I found DynamoDB helper class to do some operations into my DB.
https://www.gopiportal.in/2018/12/aws-dynamodb-helper-class-c-and-net-core.html
But, don't know how to fetch only perticular subtable
When you say subtables, I assume that you are referring to Array datatype in the above sample table. In order to fetch the data from DynamoDB table, you need hash key to use Query API. If you don't have hash key, you can use Scan API which scans the entire table. The Scan API is a costly operation.
GSI (Global Secondary Index) can be created to avoid scan operation. However, it can be created on scalar attributes only. GSI can't be created on Array attribute.
Other option is to redesign the table accordingly to match your Query Access Pattern.
Can we fetch only specific columns from my main table? Also need suggestion for subtables
Yes, you can fetch specific columns using ProjectionExpression. This way you get only the required attributes in the result set
Is it possible to modify the Rangekey column after table creation. Such as adding new column/attribute and assigning as RangeKey for the table. Tried searching but cant ble to find any articles about changing the Range or Hash key
No, unfortunately it's not possible to change the hash key, range key, or indexes after a table is created in DynamoDB. The DynamoDB UpdateItem API Documentation is clear about the fact that indexes cannot be modified. I can't find a reference to anywhere in the docs that explicitly states that the table keys cannot be modified, but at present they cannot be changed.
Note that DynamoDB is schema-less other than the hash and range key, and you can add other attributes to new items with no problems. Unfortunately, if you need to modify either your hash key or range key, you'll have to make a new table and migrate the data.
Edit (January 2014): DynamoDB now has support for on the fly global secondary indexes
To change or create an additional sort key, you will need to create a new table and migrate over to it, as both actions cannot be done on existing tables.
DynamoDB streams enable us to migrate tables without any downtime. I've done this to great effective, and the steps I've followed are:
Create a new table (let us call this NewTable), with the desired key structure, LSIs, GSIs.
Enable DynamoDB Streams on the original table
Associate a Lambda to the Stream, which pushes the record into NewTable. (This Lambda should trim off the migration flag in Step 5)
[Optional] Create a GSI on the original table to speed up scanning items. Ensure this GSI only has attributes: Primary Key, and Migrated (See Step 5).
Scan the GSI created in the previous step (or entire table) and use the following Filter:
FilterExpression = "attribute_not_exists(Migrated)"
Update each item in the table with a migrate flag (ie: “Migrated”: { “S”: “0” }, which sends it to the DynamoDB Streams (using UpdateItem API, to ensure no data loss occurs).
NOTE: You may want to increase write capacity units on the table during the updates.
The Lambda will pick up all items, trim off the Migrated flag and push it into NewTable.
Once all items have been migrated, repoint the code to the new table
Remove original table, and Lambda function once happy all is good.
Following these steps should ensure you have no data loss and no downtime.
I've documented this on my blog, with code to assist:
https://www.abhayachauhan.com/2018/01/dynamodb-changing-table-schema/