I have a simple table consisting of orderID as PK and userID as SK, I found out that in dynamoDB you need to specify both PK and SK to use query. so in this case, how is it possible for me to get all the orders for userID x since I can't ignore orderID since they're the partition key of this table? another way to solve this which works but not recomended is using a scan filter, which scans the whole table then filters the result. it will eventually slow down as the table grow. I wonder how do you guys do it with this scenario?
You can create a Global Secondary Index (GSI) based on userId and query based on that index.
You can read more about indexes and GSI’s in the AWS docs here.
Related
I want to migrate my data from DynamoDb to Redshift. I dont want to scan the whole table at once as this might result in throttling.
My Table is as below:
acountId(hash key), lastUpdatedTime.
I thought I can create GSI on lastUpdatedTime and then I can query like give me the data between day1 to day5. Again next day I can do give me data between day6 to day7.
But even with GSI my understanding is that It will scan the whole table As I wont have any hash key to provide. I just have some range of timestamp to query.
Creating a GSI is the right solution indeed. However the GSI creation operation might be a bit slow/expensive if you set GSI to project all attributes. I would recommend creating the GSI on lastUpdatedTime, and project only the partition key (and order key if you have one) using KEYS_ONLY. Then, when you scan, you will only retrieve the item keys and query the item there and then, when migrating.
I recommend reading up on GSIs here: https://docs.aws.amazon.com/fr_fr/amazondynamodb/latest/developerguide/GSI.html
I am using aws-sdk to connect with DynamoDb and I have got into a scenario where I got one dynamodb table with different partition/hash-key and I have to scan and filter to get the results. Scanning the entire table would be a costly operation . Is there a way to scan only a certain partion/haskey of a table ?
You have to use Dynamo DB Query. You can query any table or secondary index that has a composite primary key (a partition key and a sort key).
In my opinion you shouldn't use scan, because it very costly and slow.
You didn't write what is the program language, but here is some example to query:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#query-property
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
https://www.dynamodbguide.com/querying/
About indices:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SQLtoNoSQL.Indexes.html
UPDATE #1:
Maybe that will help:
Add a new column to your table. The values will be static. (For example: Col name: const_value Values: const)
Create a new secondary index to your table.
'partition key': 'const_value'
'sort key': the column what you want to filter
You can use Query.
select * from tableName where columnName="value";
How can I fetch a similar result in DynamoDB using java, without using primary key as my attribute (Need to group data based on a value for a particular column).
I have gone through articles regarding getbatchitems, QuerySpec but all these require me to pass the primary key.
Can someone give a lead here?
Short answer is you can't. Whenever you use the Query or GetItem operations in DynamoDB you must always supply the table or index primary key.
You have two options:
Perform a Scan operation on the table and filter by columnName="value". However this requires DynamoDB to look at every item in the table so it is likely to be slow and expensive.
Add a Global Secondary Index to your table. This will require you to define a primary key for the index that contains the columnName you want to query
I'm new to AWS DynamoDB and wanted to clarify something. Is it possible to query a table and filter base on a non-primary key attribute. My table looks like the following
Store
Id: PrimaryKey
Name: simple string
Location: simple string
Now I want to query on the Name, but I think I have to give the key as well from what I know? Apart from that I can use the scan but then I will be loading all the data.
From the docs:
The Query operation finds items based on primary key values. You can query any table or secondary index that has a composite primary key (a partition key and a sort key).
DynamoDB requires queries to always use the partition key.
In your case your options are:
create a Global Secondary Index that uses Name as a primary key
use a Scan + Filter if the table is relatively small, or if you expect the result set will include the majority of the records in the table
There are few designs principals that you can follow while you are using DynamoDB. If you are coming from a relational background, you have already witnessed the query limitations from primary key attributes.
Design your tables, for querying and separating hot and cold data.
Create Indexes for Querying from Non Key attributes (You have two options, Global Secondary Index which you can define at any time and Local Secondary Index which you need to specify at table creation time).
With the Global Secondary Index you can promote any NonKey attribute as the Partition Key for the Index and select another attribute for Sort Key for querying. For Local Secondary Index, you can promote any Non Key attribute as the Sort Key keeping the same Partition Key.
Using Indexes for query is important also to improve the efficiency in using provisioned throughput.
Although having indexes consumes the read throughput from the table, it also saves read through put from in a way that, if you project the right amount of attributes to read, it can give a huge benefit in reading. Check the following example.
Lets say you have a DynamoDB table that has items of 40KB. If you read directly from the table to list 10 items, it consumes 100 Read Throughput Units (For one item 10 Units since one unit can read 4KB and multiply it by 10). If you have an index defined just to project the attributes needed to list which will be having 4KB per item, then it will be consuming only 10 Read Throughput Units(One Unit per item) which makes a huge difference in terms of cost.
With DynamoDB its really important how you define Indexes to optimize for Querying not only from Query capability but also in terms of throughput.
You can not query based non-primary key attribute in Dynamo Db.
If you wanted to still do that you can do it using scan query,but scan is costly operation in DyanmoDB and if table is large, then it will affect performance and not recommended because it will scan each item in table and AWS cost you for all item it scan for that query.
There are two ways to achieve it
Keep Store Id as your PrimaryKey/ Partaion key of Dyanmo DB table and add Name/Location as sort Key (only one as Dyanmo DB accept only one Attribute as sort key by design.
Create Global Secondary Indexes for Querying from Non Key attributes which you are more frequenly required.
There are 3 ways to created GSI in Dyanamo DB, In your case select GSI with option INCLUDE and add Name , Location and store ID in Idex.
KEYS_ONLY – Each item in the index consists only of the table partition key and sort key values, plus the index key values. The KEYS_ONLY option results in the smallest possible secondary index.
INCLUDE – In addition to the attributes described in KEYS_ONLY, the secondary index will include other non-key attributes that you specify.
ALL – The secondary index includes all of the attributes from the source table. Because all of the table data is duplicated in the index, an ALL projection results in the largest possible secondary index.
Is it possible to retrieve rows from the dynamodb Global secondary index using batchgetitem api? If my aim is to retrieve data from the main table based on some non-key attribute also , but data should be retrieved in the batch of 100 items - is the GSI index won't fit here?
Also is BatchItemGet API available for Query? Say a table has the primary key and sort key and same primary key can have multiple sort keys can I retrieve multiple primary keys using batchItemGet with just primary key only or it won't fir here?
There is no way to specify the index name in the BatchGetItem API operation according to the docs. That means using BatchGetItem (and GetItem for that matter) on a secondary index isn't possible. Both of these operate on the primary index.
If you want to retrieve data from a secondary index, you need to use Query or Scan. Both support the IndexName attribute according to the documentation. When using Query you have to specify the partition key and can optionally filter based on the sort key. If you don't filter on the sort key, you will get all items with the partition key, which should take care of your second requirement.
To retrieve data from a secondary index based on different partition keys, you'd need to issue multiple Query operations for the separate values of these keys, there is no batching here.
You can use PartiQL with WHERE IN clause for that:
SELECT * FROM Orders WHERE OrderID IN [100, 300, 234]
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ql-reference.select.html