I have a dynamo DB table (id(pk),name(sk),email,date,itemId(number))
and GSI on (itemId pk, date(sk)
trying to query for an array of itemIds [1,2,3,4] but getting error using the IN statement in KeyExperssionValue when doing
aws.DocClient.query
const IdsArrat = [1,2,3,4,5];
const query: {
IndexName: 'accountId-createdAt-index',
KeyConditionExpression: 'itemId IN (:a1,:a2,:a3)',
ExpressionAttributeValues: {
{
':a1':1,
':a2':2,
.......
}
},
ScanIndexForward: false,
},
getting error using the IN statement in.
This it possible to query for multiple values on GSI in dynamoDb ?
You're trying to query for multiple different partition key's in a GSI. This can only be done by doing multiple individual queries (3 in the example). It's also possible with a GSI that multiple values would get returned for a single Partition key lookup, so it's better to query the partition key "itemId" individually.
See the following for reference:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-KeyConditionExpression
It's not possible to have a IN and join multiple values in a query , but it's possible to use BatchGetItem to request multiple queries that are solved in parallel . This is actually very close to the IN solution you want.
The result will be a list of the elements in the table.
There are limits in the number of queries in the size of the result set < 16 MB and the number of queries < 100.
Please check this document for details :
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html
refering to this answer https://stackoverflow.com/a/70494101/7706503, you could try partiQL to construct similar statement for querying gsi table with multiple key,
select * from table."gsi_index_name" where partition_key in [key1,key2]
then you could send the statement with low level api in one shot, for example, in dotnet, it's called ExecuteStatementAsync
Related
Consider Below is my sample json.
{
"servletname": "cofaxEmail",
"servlet-class": "org.cofax.cds.EmailServlet",
"init-param": {
"mailHost": "mail1",
"mailHostOverride": "mail2"
}
i have chosen "servletname" as my primary key as i will receive it in every request plus few 1000 server names are there it could be the best PK.
My Question is, to make the partition key work for me.
Do i have to specify the partition key option seperately like below
ItemResponse<ServerDto> ServerDtoResponse = await this.container.ReadItemAsync<ServerDto>(bocServerDto.mailHost, new PartitionKey(bocServerDto.servletname));
or
Including the partition key in the select query itself , without adding seperate new PartitionKey(), like
select * from r where r.servletname='cofaxEmail' and r.mailHost='mail1';
Crux of the question is: By passing partitionKey object in where condition of select query is it enough to utilize the partition key feature?
Thanks
For any crud operation you would pass in the value for the partition key. For example, on a point read.
ItemResponse<ServerDto> ServerDtoResponse = await this.container.ReadItemAsync<ServerDto>(bocServerDto.mailHost, new PartitionKey("cofaxEmail"));
For a query, you can either pass it in the queryRequest options or use it in the query as the first filter predicate. Here is an example of using the queryRequest options.
thanks.
I am using boto3 to create a simple DynamoDB query as follows:
response = table.query(
KeyConditionExpression=Key('course-lesson-id-part').eq(1)
)
This query works and return the single item just like it should.
course-lesson-id-part is my primary partition key and is numeric; I have two items in my table:
course-lesson-id-part: 1
course-lesson-id-part: 2
Ideally, and according to this reference, I should be able to use the following to query all items with course-lesson-id-part > 0:
response = table.query(
KeyConditionExpression=Key('course-lesson-id-part').gt(0)
)
None of the other DB Query conditions seem to be working other than .eq, which does me no good here.
Anyone have any ideas?
Only eq is supported for partition keys.
You must specify the partition key name and value as an equality condition.
Source: Working with Queries
I'm modeling a dynamodb diagram for an invoice app and I'm looking for generate the unique invoice id that need to be incremented from 1 to X. There is in 2019 a solution about this kind of problem with aws appsync and dynamodb as datasource ?
Auto-incrementing integers are not a recommended pattern in DynamoDB although it is possible to implement something similar using application level logic. A DynamoDB table is distributed to many logical partitions according to the table's partition key. Items are then sorted within that partition according to their sort key. You will need to decide what structure makes sense for you app and what an auto-incrementing means for your app. The simplest case would be to omit a sort key and treat the auto-incremented id as the partition key which would guarantee its uniqueness but also has implications that every row lives in its own partition and thus listing all invoices would have to be a Scan and thus does not preserve order which may or may not make sense for your app.
As mentioned in this SO post (How to use auto increment for primary key id in dynamodb) you can use code like this:
const params = {
TableName: 'CounterTable',
Key: { HashKey : 'auto-incrementing-counter' },
UpdateExpression: 'ADD #a :x',
ExpressionAttributeNames: {'#a' : "counter_value"},
ExpressionAttributeValues: {':x' : 1},
ReturnValues: "UPDATED_NEW" // ensures you get value back the new key
};
new AWS.DynamoDB.DocumentClient().update(params, function(err, data) {});
to atomically increment the integer stored in the CounterTable row designated by the partition key "auto-incrementing-counter". After the atomic increment, you can use the returned id to create the new Invoice.
You can implement this pattern using DynamoDB & AppSync but the first thing to decide is if it suits your use case. You may also be interested in the RDS integration via the RDS data API which would have more native support for auto-incrementing IDs but would lose out on the set it and forget it scaling of DynamoDB.
I understand this query might be inefficient since it can involve a full table scan in worst case, but I need to fetch only a single item at a time.
For example, I have a table containing values like this:
{
id: 'bc63a25e-b92b-483e-9ad3-ad6d474dfae2',
domain: 'xyz.com',
template_url: `https://s3.us-east-2.amazonaws.com/bucket/some-random-url.html`,
data_elements: {
message_link: 'http://www.google.com'
zodiac_sign: 'Scorpio'
}
}
I have a GSI with domain as the hash key. Now I want to fetch items from this table:
WHERE domain == 'xyz.com'
AND id not in <a list of ids>
LIMIT 1;
How can I achieve this type of query ? I checked the documentation and I could see there is IN operator but could not find any NOT IN operator.
I had the same issue, and I don't think you can. You will need to use the key for the 'get' method and the 'scan' method. The only alternative (I think) would be to fetch all items and then to do a string comparison on each and everyone. I don't think I need to mention how incredibly expensive that would be.
As mentioned, I had to deal with the same issue and I ended up changing my data structure. It was a bit cumbersome to begin with and I have twice the data entries of a relational db but it is negligible and the query is incredibly fast even on the micro AWS instance.
You can't always do the same operations on a NoSQL db that you can do on a MySQL db and this is a prime example of that.
I am not sure why you have mentioned about scan as you have hashkey of the GSI. You can use the Query API with the below params.
var idArray = ["1", "2"];
var params = {
TableName : "tablename",
IndexName : 'your_index_name',
KeyConditionExpression : 'domain = :domainVal',
FilterExpression : "NOT #id IN (:idValue)",
ExpressionAttributeNames: { "#id": "id" },
ExpressionAttributeValues : {
":domainVal" : 'xyz.com',
":idValue" : idArray
}
};
I have tested the NOT IN on my table. It works fine for me.
You can run SQL queries on DynamoDB if you use EMR Hive or Redshift. In this case you can use any SQL operators to query your data.
Of course this is not intended for interactive queries and intended only for some analytics queries that are executed infrequently.
Here is how to use DynamoDB with Redshift.
Here is how to use DynamoDB with EMR Hive.
I have 2 LSI in my table with a primary partition Key with primary sort key
Org-ID - primary partition Key
ClientID- primary sort Key
Gender - LSI
Section - LSI
I have no issue with querying a table with one LSI, but how to mention 2 LSI in a table schema.
var params = {
TableName:"MyTable",
IndexNames: ['ClientID-Gender-index','ClientID-Section-index'],
KeyConditionExpression : '#Key1 = :Value1 and #Key2=:Value2 and #Key3=:Value3',
ExpressionAttributeNames:{
"#Key1":"Org-ID",
"#Key2":"Gender",
"#Key3":"Section"
},
ExpressionAttributeValues : {
':Value1' :"Microsoft",
':Value2':"Male",
':Value3':"Cloud Computing"
}};
Can anyone fix the issue in IndexName(line 3) or KeyConditionExpression(line 4), I'm not sure about it.
Issue
Condition can be of length 1 or 2 only
You can only query a single DynamoDB index at a time. You cannot use multiple indexes in the same query.
A simple alternative is to use a single index and apply a query filter, but this will potentially require a lot of records to be scanned and the filter only reduces the amount of data transferred over the network.
A more advanced alternative is to make a compound key. You would most likely want to use a GSI, rather than an LSI for this use case. By making a single new column that is the string concatenation of Key1, Key2, and Key3 you can use this GSI to search all three keys at the same time. This will make each individual record bigger by repeating data but it allows for a more complex query pattern.