Need to query with mac_addr, such as:
Find all data with devices & mac_addr
but as suggested in dynamo, for querying we can't use list.
Device JSON
{
"issuer" : "device.stn100bom.ctr.sophos.com"
"id" : "5ef29826-3b7a-11ed-a261-0242ac120002"
"aid" : "65d30626-3b7a-11ed-a261-0242ac120002"
"mac_addr" : ["82:c5:50:a7:ac:01",...]
}
One possible option could be creating a separate table, which has deviceId and mac_addr, acting as a lookup table.
Please suggest if there is a better option available.
Related
First of all, I have table structure like this,
Users:{
UserId
Name
Email
SubTable1:[{
Column-111
Column-112
},
{
Column-121
Column-122
}]
SubTable2:[{
Column-211
Column-212
},
{
Column-221
Column-222
}]
}
As I am new to DynamoDB, so I have couple of questions regarding this as follows:
1. Can I create structure like this?
2. Can we set primary key for subtables?
3. Luckily, I found DynamoDB helper class to do some operations into my DB.
https://www.gopiportal.in/2018/12/aws-dynamodb-helper-class-c-and-net-core.html
But, don't know how to fetch only perticular subtable
4. Can we fetch only specific columns from my main table? Also need suggestion for subtables
Note: I am using .net core c# language to communicate with DynamoDB.
Can I create structure like this?
Yes
Can we set primary key for subtables?
No, hash key can be set on top level scalar attributes only (String, Number etc.)
Luckily, I found DynamoDB helper class to do some operations into my DB.
https://www.gopiportal.in/2018/12/aws-dynamodb-helper-class-c-and-net-core.html
But, don't know how to fetch only perticular subtable
When you say subtables, I assume that you are referring to Array datatype in the above sample table. In order to fetch the data from DynamoDB table, you need hash key to use Query API. If you don't have hash key, you can use Scan API which scans the entire table. The Scan API is a costly operation.
GSI (Global Secondary Index) can be created to avoid scan operation. However, it can be created on scalar attributes only. GSI can't be created on Array attribute.
Other option is to redesign the table accordingly to match your Query Access Pattern.
Can we fetch only specific columns from my main table? Also need suggestion for subtables
Yes, you can fetch specific columns using ProjectionExpression. This way you get only the required attributes in the result set
Suppose I have an object like
{
epochTime : 1527174282
action : create
state : fail
}
From the documentation of AWS, to query
You must specify the partition key name and value as an equality
condition.
You can optionally provide a second condition for the sort key (if
present).
In the case where I just want to query some item from A to B from the whole dataset ( Just want to use the sort key ) how should I chooe the hash key and query for this kind of data to work effectively?
You need to define secondary index for the attribute with the primary key (hash) as the field and perform scan on the secondary index
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html
Working with scans
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html
Also refer this java sdk examples for working with secondary index and scans https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSIJavaDocumentAPI.Example.html
https://aws.amazon.com/about-aws/whats-new/2015/02/10/secondary-index-scan-a-simpler-way-to-scan-dynamodb-table/
I want to write a terraform module that will create dynamoDb tables. The attributes are expected to be read from .tfvars or default variable instead of being already named in .tf as in the resource guide here
To explain further, say a list of attributes is being used to achieve this pseudo-code:
resource "aws_dynamodb_table" "basic-dynamodb-table" {
name = "GameScores"
... #Other required feilds
...
...
# attributes is a list of names
for(attribute_name:${length(var.attributes)}){
attribute {
name = "${var.attributes[i]}"
type = "N"
}
}
}
How can I iterate over the attribute list and create the attribute{ } during terraform plan/apply ? The number of attribute blocks cannot be static like shown in the terraform docs, and their names must be read from variables.
very late but must be util for another people that are looking at.
I use this doc and there I can use some like this
dynamic "attribute" {
for_each = var.table_attributes
content {
name = tag.value["name"]
type = tag.value["type"]
}
}
In my variables.tf I have the variable table_attributes with a map of objects to use.
Maybe it helps someone.
When you create a DynamoDB table, the only attributes you need to specify are the partition key and, optionally, the sort key. All other attributes are stored as part of each document (or item) you store in the table.
The same applies to Global Secondary Indexes as well. You only need to specify the partition key and sort key for each index.
If you don't have static attributes, then you can't create a table. The names of the partition and sort keys must be the same for the lifetime of a table/index.
Finally, it's not clear from the question, but please don't use Teraform to load data into your table. It's not the right tool for that!
I understand this query might be inefficient since it can involve a full table scan in worst case, but I need to fetch only a single item at a time.
For example, I have a table containing values like this:
{
id: 'bc63a25e-b92b-483e-9ad3-ad6d474dfae2',
domain: 'xyz.com',
template_url: `https://s3.us-east-2.amazonaws.com/bucket/some-random-url.html`,
data_elements: {
message_link: 'http://www.google.com'
zodiac_sign: 'Scorpio'
}
}
I have a GSI with domain as the hash key. Now I want to fetch items from this table:
WHERE domain == 'xyz.com'
AND id not in <a list of ids>
LIMIT 1;
How can I achieve this type of query ? I checked the documentation and I could see there is IN operator but could not find any NOT IN operator.
I had the same issue, and I don't think you can. You will need to use the key for the 'get' method and the 'scan' method. The only alternative (I think) would be to fetch all items and then to do a string comparison on each and everyone. I don't think I need to mention how incredibly expensive that would be.
As mentioned, I had to deal with the same issue and I ended up changing my data structure. It was a bit cumbersome to begin with and I have twice the data entries of a relational db but it is negligible and the query is incredibly fast even on the micro AWS instance.
You can't always do the same operations on a NoSQL db that you can do on a MySQL db and this is a prime example of that.
I am not sure why you have mentioned about scan as you have hashkey of the GSI. You can use the Query API with the below params.
var idArray = ["1", "2"];
var params = {
TableName : "tablename",
IndexName : 'your_index_name',
KeyConditionExpression : 'domain = :domainVal',
FilterExpression : "NOT #id IN (:idValue)",
ExpressionAttributeNames: { "#id": "id" },
ExpressionAttributeValues : {
":domainVal" : 'xyz.com',
":idValue" : idArray
}
};
I have tested the NOT IN on my table. It works fine for me.
You can run SQL queries on DynamoDB if you use EMR Hive or Redshift. In this case you can use any SQL operators to query your data.
Of course this is not intended for interactive queries and intended only for some analytics queries that are executed infrequently.
Here is how to use DynamoDB with Redshift.
Here is how to use DynamoDB with EMR Hive.
Is it possible to retrieve an entity from gae datastore using a property and not using the key?
I could see I can retrieve entities with key using the below syntax.
quote = mgr.getObjectById(Students.class, id);
Is there an alternative that enables us to use a property instead of key?
Or please suggest any other ways to achieve the requirement.
Thanks,
Karthick.
Of course this is possible. Think of the key of an entity being like the primary key of an SQL row (but please, don't stretch the analogy too far - the point is it's a primary key - the implementations of these two data storage systems are very different and it causes people trouble when they don't keep this in mind).
You should look either here (JDO) to read about JDO queries or here (JPA) to read about JPA queries, depending what kind of mgr your post refers to. For JDO, you would do something like this:
// begin building a new query on the Cat-kind entities (given a properly annotated
// entity model class "Cat" somewhere in your code)
Query q = pm.newQuery(Cat.class);
// set filter on species property to == param
q.setFilter("species == speciesParam");
// set ordering for query results by age property descending
q.setOrdering("age desc");
// declare the parameters for this query (format is "<Type> <name>")
// as referenced above in filter statement
q.declareParameters("String speciesParam");
// run the query
List<Cat> results = (List<Cat>) q.execute ("siamese");
For JPA, you would use JPQL strings to run your queries.