I'm trying to create an index on a nested field, using the Dashboard in AWS Developer Console. E.g. if I have the following schema:
{ 'id': 1,
'nested': {
'mode': 'mode1',
'text': 'nice text'
}
}
I was able to create the index on nested.mode, but whenever I then go to query by index, nothing ever comes back. It makes me think that DynamoDB created the index on a field name nested.mode instead of the mode field of nested. Any hints re. what I might be doing wrong?
You cannot (currently) create a secondary index off of a nested attribute. From the Improving Data Access with Secondary Indexes in DynamoDB documentation (emphasis mine):
For each secondary index, you must specify the following:
...
The key schema for the index. Every attribute in the index key schema must be a top-level attribute of type String, Number, or Binary. Nested attributes and multi-valued sets are not allowed. Other requirements for the key schema depend on the type of index:
You can, however, create an index on any top level JSON element.
Related
I've got an object like this:
{
"id": (UUID generated at call time)
"referenceId": (Foreign key for another system)
"referenceType": (enum)
}
The primary index is just
Primary key: id
And I've got a secondary index like
Primary key: referenceId
Secondary key: referenceType
So what I want to do, is query the secondary index with the referenceId and referenceType, and then if that's empty, write the record to the table.
The problem is, Conditional Expressions can only check same index you're querying. And I've looked into DynamoDb Transactions, but they say you can't have two transactions target the same item.
Is there a way to make this work?
If I understand your question correctly, your item has three attributes: id,
referenceId and referenceType. You've also defined a global secondary index with a composite primary key of referenceId and referenceType.
Assuming all these attributes are part of the same item, you shouldn't need to read the secondary index before deciding to write to the table. Rather, you could perform a PUT operation on the condition that referenceId and reference type don't yet exist on that item.
ddbClient.putItem({
"TableName": "YOUR TABLE NAME",
"Item": { "PK": "id" },
"ConditionExpression": "attribute_exists(referenceId) AND attribute_exists(referenceType)"
})
You may also want to check out this fantastic article on DynamoDB condition expressions.
As I understand the question, your table PK is generated "at call time" so it would be different for two different requests with the same reference ID and reference type.
That being the case, your answer is no.
If you've got a requirement for uniqueness of a key set by a foreign system, you should consider using the same key or some mashup of it for your DDB table.
Alternately, modify the foreign system to set the UUID for a given reference ID & type.
First of all, I have table structure like this,
Users:{
UserId
Name
Email
SubTable1:[{
Column-111
Column-112
},
{
Column-121
Column-122
}]
SubTable2:[{
Column-211
Column-212
},
{
Column-221
Column-222
}]
}
As I am new to DynamoDB, so I have couple of questions regarding this as follows:
1. Can I create structure like this?
2. Can we set primary key for subtables?
3. Luckily, I found DynamoDB helper class to do some operations into my DB.
https://www.gopiportal.in/2018/12/aws-dynamodb-helper-class-c-and-net-core.html
But, don't know how to fetch only perticular subtable
4. Can we fetch only specific columns from my main table? Also need suggestion for subtables
Note: I am using .net core c# language to communicate with DynamoDB.
Can I create structure like this?
Yes
Can we set primary key for subtables?
No, hash key can be set on top level scalar attributes only (String, Number etc.)
Luckily, I found DynamoDB helper class to do some operations into my DB.
https://www.gopiportal.in/2018/12/aws-dynamodb-helper-class-c-and-net-core.html
But, don't know how to fetch only perticular subtable
When you say subtables, I assume that you are referring to Array datatype in the above sample table. In order to fetch the data from DynamoDB table, you need hash key to use Query API. If you don't have hash key, you can use Scan API which scans the entire table. The Scan API is a costly operation.
GSI (Global Secondary Index) can be created to avoid scan operation. However, it can be created on scalar attributes only. GSI can't be created on Array attribute.
Other option is to redesign the table accordingly to match your Query Access Pattern.
Can we fetch only specific columns from my main table? Also need suggestion for subtables
Yes, you can fetch specific columns using ProjectionExpression. This way you get only the required attributes in the result set
I am referring to a thread creating an index with JSON
I have a column called data in my DynamoDB table. This is in JSON and the structure of this file looks like this:
{
"config": "aasdfds",
"state":"PROCESSED",
"value" "asfdasasdf"
}
The AWS documentation says that I can create an index with the top level JSON attribute. However I don't know how to do this exactly. When I create the index, should I specify the partition key as data.state, then, in my code, use a query with the column data.state with the value set to PROCESSED, or should I create the partition key as data, then, in my code, look for the column data with the value set to state = "PROCESSED" ?
Top level attribute means DynamoDB supports creating index on Scalar attributes only (String, Number, or Binary).
The JSON attribute is stored as Document data type. So, index can't be created on Document data type.
The key schema for the index. Every attribute in the index key schema
must be a top-level attribute of type String, Number, or Binary. Other
data types, including documents and sets, are not allowed.
Scalar Types – A scalar type can represent exactly one value. The
scalar types are number, string, binary, Boolean, and null.
Document Types – A document type can represent a complex structure
with nested attributes—such as you would find in a JSON document. The
document types are list and map.
Set Types – A set type can represent multiple scalar values. The set
types are string set, number set, and binary set.
I have a dynamo table with the following attributes :
id (Number - primary key )
title (String)
created_at (Number - long)
tags (StringSet - contains a set of tags say android, ios, etc.,)
I want to be able to query by tags - get me all the items tagged android. How can I do that in DynamoDB? It appears that global secondary index can be built only on ScalarDataTypes (which is Number and String) and not on items inside a set.
If the approach I am taking is wrong, an alternative way for doing it either by creating different tables or changing the attributes is also fine.
DynamoDB is not designed to optimize indexing on set values. Below is a copy of the amazon's relevant documentation (from Improving Data Access with Secondary Indexes in DynamoDB).
The key schema for the index. Every attribute in the index key schema
must be a top-level attribute of type String, Number, or Binary.
Nested attributes and multi-valued sets are not allowed. Other
requirements for the key schema depend on the type of index: For a
global secondary index, the hash attribute can be any scalar table
attribute. A range attribute is optional, and it too can be any scalar
table attribute. For a local secondary index, the hash attribute must
be the same as the table's hash attribute, and the range attribute
must be a non-key table attribute.
Amazon recommends creating a separate one-to-many table for these kind of problems. More info here : Use one to many tables
This is a really old post, sorry to revive it, but I'd take a look at "Single Table Design"
Basically, stop thinking about your data as structured data - embrace denormalization
id (Number - primary key )
title (String)
created_at (Number - long)
tags (StringSet - contains a set of tags say android, ios, etc.,)
Instead of a nosql table with a "header" of this:
id|title|created_at|tags
think of it like this:
pk|sk |data....
id|id |{title, created_at}
id|id+tag|{id, tag} <- create one record per tag
You can still return everything by querying for pk=id & sk begins with id and join the tags to the id records in your app logic
and you can use a GSI to project id|id+tag into tag|id which will still require you to write two queries against your data to get items of a given tag (get the ids then get the items), but you won't have to duplicate your data, you wont have to scan and you'll still be able to get your items in one query when your access pattern doesn't rely on tags.
FWIW I'd start by thinking about all of your access patterns, and from there think about how you can structure composite keys and/or GSIs
cheers
You will need to create a separate table for this query.
If you are interested in fetching all items based on a tag then I suggest keeping a table with a primary key:
hash: tag
range: id
This way you can use a very simple Query to fetch all items by tag.
What will happen if I create a PutItem request like this:
{
"Expected":
{
"testAttribute" :
{
"Exists": "false",
}
},
"Item":
{
"testAttribute" :
{
"S": "testValue"
}
},
"TableName": "TableName"
}
where "testAttribute" is not part of the primary key.
Will DynamoDB scan the table to see if there is an item with attribute "testAttribute" == "testValue" ?
If not, how will DynamoDB determine the presence of a "testAttribute" == "testValue" ?
I can't find anything in the docs describing how this works.
According to the documentation of the PutItem action, you are not allowed to issue that request. It says:
Item: A map of attribute name/value pairs, one for each attribute. Only the primary key attributes are required; you can optionally provide other attribute name-value pairs for the item.
(emphasis mine)
You must provide a value for each attribute of the primary key whenever you use PutItem.
This way, as you will surely agree, it is very simple and fast for DynamoDB to check the condition you defined on the Expected clause: no scan is needed, it just has to look at the single item that could match the request. Otherwise, as you noted, DynamoDB would need to perform a full table scan (and it would possibly be very slow, and they would certainly charge you for that) or it would need to maintain a consistent index of every single item in a table, and they would charge you for the SSD space used to store it!
Also, note that the meaning of the expected clause is a little bit different than what you described in the question. Supposing you fix your request and add all the primary key attributes, the request would mean:
"If the item identified by this primary key does not exist, create it; if it does exist and does not contain an attribute named testAttribute, replace the item with the one whose attributes are described in this request; if the item does exist and does contain an attribute named testAttribute, do nothing".
Your description says that DynamoDB would check if the value of testAttribute is testValue, but it is not what happens when you use the Expected/Exists clause. To achieve the effect you described, you need to use the Expected/Value clause, and then you specify the value you are expecting in that clause -- the value specified for the attribute in the Item property of the request is just used to define the new value of the attribute, if an update (or insert) is to occur.