I am having parition key as id that is VIN(vehicle identification number) sort key as timestamp, I want to execute query and want to know the latest record having the maximum timestamp(only single record based on vin with latest timestamp).
You can use Query API with ScanIndexForward as false and Limit equal to 1 to achieve the result.
ScanIndexForward = false -> means arranging the sort key in descending
order
Limit = 1 -> means return only one item
Sample code:-
var params = {
TableName : "yourtablename",
KeyConditionExpression: "#VIN = :vinvalue",
ExpressionAttributeNames:{
"#VIN": "VIN"
},
ExpressionAttributeValues: {
":vinvalue":"somevinvalue"
},
ScanIndexForward : false,
Limit : 1
};
docClient.query(params, function(err, data) {
if (err) {
console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
} else {
console.log("Query succeeded.");
data.Items.forEach(function(item) {
console.log(" -", item.year + ": " + item.title);
});
}
});
public EventLogEntity fetch(String vin) {
EventLogEntity eventLogEntityResult = null;
EventLogEntity entity = new EventLogEntity();
entity.setId(vin);
DynamoDBQueryExpression<EventLogEntity> queryExpression =
new DynamoDBQueryExpression<EventLogEntity>().withHashKeyValues(entity);
//DynamoDBQueryExpression Works only for Partition, Sort and Indexes in Dynamo Db
queryExpression.setScanIndexForward(false); // Will give you result in descending order
queryExpression.withLimit(1);// Will always give you single record
List<EventLogEntity> result = dynamoDBMapper.queryPage(EventLogEntity.class, queryExpression).getResults();
if (result != null && !result.isEmpty()) {
eventLogEntityResult = result.get(0);
}
return eventLogEntityResult;
}
Related
I am new to dynamoDB. I am having difficulty developing a table structure. I have data that can best be thought of as a folder structure. There are folders which are nested in parent folders. Most of the time, I will be querying for all folders with a given parent folder, however, there are times when I will be querying individual folders.
If I use the parent_id (parent folder) as the partition key and the id of the individual folder as the sort key, I believe that this creates a table where all related files are stored together and I can query them efficiently. However, I have questions.
First, the query "works" in that it returns the data, but is it written so that it queries the data correctly and is not merely scrolling through the whole table?
router.get("/api/children_folders/:parent_id", (req, res, next) => {
let parent_id = req.params.parent_id;
let params = {
TableName: tableName,
KeyConditionExpression: "parent_id = :pid",
ExpressionAttributeValues: {
":pid": parent_id,
},
ScanIndexForward: false,
};
docClient.query(params, (err, data) => {
if (err) {
console.log(err);
return res.status(err.statusCode).send({
message: err.message,
status: err.statusCode,
});
} else {
return res.status(200).send(data);
}
});
});
Second, if I want to query for individual tags, do I need to pass in a combination of the parent folder ID and the actual ID, or is this OK?
router.get("/api/folder/:folder_id", (req, res, next) => {
let tag_id = req.params.folder_id;
let params = {
TableName: tableName,
KeyConditionExpression: "folder_id = :fid",
ExpressionAttributeValues: {
":fid": folder_id,
},
Limit: 1,
};
docClient.query(params, (err, data) => {
if (err) {
console.log(err);
return res.status(err.statusCode).send({
message: err.message,
status: err.statusCode,
});
} else {
if (!_.isEmpty(data.Items)) {
return res.status(200).send(data.Items[0]);
} else {
return res.status(404).send();
}
}
});
});
I just feel like I am missing some thing here and I want to make sure that I am grabbing the data correctly.
The PK, should be something that would divide the load equally (ideally). I don't the fully picture of your problem but assuming you can chose a good parent folder as a partition key, then you can insert every file/dir with a sort key representing its full path
For example:
PK SK
/home /username/pictures/cat.jpg
This way if you want to get a specific item you can use the get item request
var params = {
Key: {
"PK": { "S": "/home" },
"SK": { "S": "/username/pictures/cat.jpg" }
},
TableName: tableName
};
var result = await dynamodb.getItem(params).promise()
Now if you want to list all the files in "/home/username/pictures" you can use begins with query
const params = {
TableName: 'tablenName',
KeyConditionExpression: '#PK = :root_path and begins_with(#SK, :sub_path)',
ExpressionAttributeNames:{
"#user_id": "root_path",
"#user_relation": 'sub_path'
},
ExpressionAttributeValues: {
":root_path": "/home",
":sub_path": "/username/pictures"
}
}
I've used following stored procedure and try to parameterize the query by passing the number parameter. However it gives me error while executing the stored procedure. Any insights is really helpful
function uspGetUsersByPage(number) {
//Set Environment
let context = getContext();
let coll = context.getCollection();
let link = coll.getSelfLink();
let response = context.getResponse();
let query = {
query: 'SELECT * FROM a WHERE a.DocumentType = "UserRole" and a.AuditFields.IsLatest = true and a.AuditFields.IsDeleted = false OFFSET #number LIMIT 20'
, parameters: [{ name: '#number', value: number }]
};
//Execute the query against the collection
let runquery = coll.queryDocuments(link, query, {}, callbackfn);
//Call function to throw an error(if any) or display the output
function callbackfn(err, queryoutput) {
if (err) {
throw err;
}
if (!queryoutput || !queryoutput.length) {
response.setBody(null);
}
else {
response.setBody(queryoutput);
}
};
//Display standard output if query doesnt get any results
if (!runquery) { throw Error('Unable to retrieve requested information'); }
};
Please see my simple test following your description.
Data:
Stored procedure:
function sample(prefix) {
var collection = getContext().getCollection();
var query = {query: "SELECT c.id,c.number FROM c offset #num limit 1", parameters:
[{name: "#num", value: prefix}]};
console.log(query);
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
query,
function (err, feed, options) {
if (err) throw err;
// Check the feed and if empty, set the body to 'no docs found',
// else take 1st element from feed
if (!feed || !feed.length) {
var response = getContext().getResponse();
response.setBody('no docs found');
}
else {
var response = getContext().getResponse();
response.setBody(feed);
}
});
if (!isAccepted) throw new Error('The query was not accepted by the server.');
}
Input:
Output:
I have something like this:
{
Records:{}
}
and I just want to add data inside records like this:
{
Id: 123,
Records:{
10001: { event : "item1" }
}
}
My 1st attempt:
var params = {
TableName : "Records",
Key : { Id : 123 },
UpdateExpression: 'set Record.#k1.event = :v1',
ExpressionAttributeNames: { '#k1' : 10001},
ExpressionAttributeValues: { ':v1' : 'item1' }
};
Because Record.10001 doesn't exist, it give me error:
The document path provided in the update expression is invalid for update
"ADD" only support NUMBER and SET type so its not suitable in my case.
My 2nd attempt (trying to create a empty map 1st):
var params = {
TableName : "Records",
Key : { Id : 123 },
UpdateExpression: 'set Record.#k1 = {}',
ExpressionAttributeNames: { '#k1' : 10001},
};
It fails as my syntax is probably incorrect and I cannot find how to create a empty map to an attribute.
So my question is how do I add structured object as a map or empty map to an existing item like the example above?
I can easily modifying the data from the amazon DynamoDB data management page so at least I know there exist a way to do it. I just need to know how.
Thanks
I somehow successfully done it this way:
var params = {
TableName: "Records",
Key: { Id: 123 },
UpdateExpression: 'set Records.#k1 = :v1',
ExpressionAttributeNames: {'#k1': '10001',},
ExpressionAttributeValues: {':v1': { event: 'Yay~' } }
};
optionally you could use
UpdateExpression: 'set Records.#k1 = if_not_exists( Records.#k1, :v1)',
to avoid overwriting existing records.
I found a workaround for the error we get.
try:
table.update_item(
Key = {'Id' : id},
UpdateExpression = 'SET Records.#key1.#key2 = :value1',
ExpressionAttributeNames = {'#key1': '10001', '#key2' : 'event'},
ExpressionAttributeValues = {':value1': 'singing'}
)
except:
table.update_item(
Key = {'Id' : id},
UpdateExpression = 'SET Records = :value1',
ExpressionAttributeValues = {':value1': {'10001': {'event' : 'dance'}}}
)
When we try to update for the first time we get "The document path provided in the update expression is invalid for update" error. An exception will be thrown since there will be no nested JSON.
Thereafter, any updates say for example Record.10002 will not throw exception. Update function in the try block will be executed.
Please comment on this if this is the correct way of handling this kind of scenario.
one of the possible resolution to get rid of such problem: you have to have an empty object created and exist in DB in advance.
for instance,
during generation of your object (original creation in DB):
field: { }
to update this field use:
TableName: ....,
Key: {
.....
},
UpdateExpression: 'set field.#mapKey = :v',
ExpressionAttributeNames: {
'#mapKey': myName
},
ExpressionAttributeValues: {
':v': myValue
}
This can be done in c# as follows.
If you have the following json:
"invoice":{
"M":{
":number":{
"S":"XPN1052"
}":date":{
"S":"02-17-2022"
}
}
}
You can use the following code to insert an invoice record:
ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
{":invoice", new AttributeValue { M = new Dictionary<string, AttributeValue>{
{":date", new AttributeValue { S = invoice.TxnDate.ToString("MM-dd-yyyy") }},
{":number", new AttributeValue{ S = invoice.DocNumber } } }
} }},
UpdateExpression = "SET invoice = :invoice",
var params = {
TableName : "Records",
Key : { Id : 123 },
UpdateExpression: 'set #Records = :obj',
ExpressionAttributeNames: { '#Records' : 'Records'},
ExpressionAttributeValues: { ':obj' : {'10001': { 'event' : 'item1' }}}
};
In order to insert/update Dynamodb items dynamically based on a given dictionary with various keys (eventually columns in Dynamodb) we can construct query string as shown below:
# given dictionary with needs to be dumped into dynamodb
dict_to_insert = {'column1':'value1',
'column2.subdocument1':{'subkey1':'subvalue1',
'subkey2':'subvalue2'}...}...
# placeholder variables
# Using upsert for insert/update mllog records on dynamodb
query = "SET"
values_dict = {}
names_dict = {}
i = 0
# iterating over given dictionary and construct the query
for key, value in dict_to_insert.items():
# if item key column provide in the given dictionary
# then it should be skipped
if key == 'ITEM_KEY_NAME':
i += 1
continue
# None values causing error on update_item
# So, better to ignore them as well
if value is not None:
# if_not_exists method prevents overwriting existing
# values on columns
query += " " + "#k_" + str(i) + " = if_not_exists(#k_" + str(i) + \
", :v_" + str(i)+")"
if key != list(dict_to_insert.keys())[-1]: # if not the last attribute add ,
query += ","
# Defining dynamodb columns and their corresponding values
values_dict["#k_" + str(i)] = key
names_dict[":v_" + str(i)] = value
i += 1
Once we construct the query string as shown above, then we can confidently run our update_item method on Dynamodb:
# table obj here is boto3 dynamodb resource instance
table.update_item(
TableName='DYNAMODB_TABLENAME',
Key={'ITEM_KEY_NAME': 'ANY_VALUE_AS_KEY'},
UpdateExpression=query.strip(),
ExpressionAttributeNames=names_dict,
ExpressionAttributeValues=values_dict)
How do you "upsert" a property to a DynamoDB row. E.g. SET address.state = "MA" for some item, when address does not yet exist?
I feel like I'm having a chicken-and-egg problem because DynamoDB doesn't let you define a sloppy schema in advance.
If address DID already exist on that item, of type M (for Map), the internet tells me I could issue an UpdateExpression like:
SET #address.#state = :value
with #address, #state, and :value appropriately mapped to address, state, and MA, respectively.
But if the address property does not already exist, this gives an error:
'''
ValidationException: The document path provided in the update expression is invalid for update
'''
So.. it appears I either need to:
Figure out a way to "upsert" address.state (e.g., SET address = {}; SET address.state = 'MA' in a single command)
or
Issue three (!!!) roundtrips in which I try it, SET address = {}; on failure, and then try it again.
If the latter.... how do I set a blank map?!?
Ugh.. I like Dynamo, but unless I'm missing something obvious this is a bit crazy..
You can do it with two round trips, the first conditionally sets an empty map for address if it doesn't already exist, and the second sets the state:
db.update({
UpdateExpression: 'SET #a = :value',
ConditionExpression: 'attribute_not_exists(#a)',
ExpressionAttributeValues: {
":value": {},
},
ExpressionAttributeNames: {
'#a': 'address'
}
}, ...);
Then:
db.update({
UpdateExpression: 'SET #a.#b = :v',
ExpressionAttributeNames: {
'#a': 'address',
'#b': 'state'
},
ExpressionAttributeValues: {
':v': 'whatever'
}
}, ...);
You cannot set nested attributes if the parent document does not exist. Since address does not exist you cannot set the attribute province inside it. You can achieve your goal if you set address to an empty map when you create the item. Then, you can use the following parameters to condition an update on an attribute address.province not existing yet.
var params = {
TableName: 'Image',
Key: {
Id: 'dynamodb.png'
},
UpdateExpression: 'SET address.province = :ma',
ConditionExpression: 'attribute_not_exists(address.province)',
ExpressionAttributeValues: {
':ma': 'MA'
},
ReturnValues: 'ALL_NEW'
};
docClient.update(params, function(err, data) {
if (err) ppJson(err); // an error occurred
else ppJson(data); // successful response
});
By the way, I had to replace state with province as state is a reserved word.
Another totally different method is to simply create the address node when creating the parent document in the first place. For example assuming you have a hash key of id, you might do:
db.put({
Item: {
id: 42,
address: {}
}
}, ...);
This will allow you to simply set the address.state value as the address map already exists:
db.update({
UpdateExpression: 'SET #a.#b = :v',
AttributeExpressionNames: {
'#a': 'address',
'#b': 'state'
},
AttributeExpressionValues: {
':v': 'whatever'
}
}, ...);
Some kotlin code to do this recursively regardless how deep it goes. It sets existence of parent paths as condition and if condition check fails, recursively creates those paths first. It has to be in the library's package so it can access those package private fields/classes.
package com.amazonaws.services.dynamodbv2.xspec
import com.amazonaws.services.dynamodbv2.document.Table
import com.amazonaws.services.dynamodbv2.model.ConditionalCheckFailedException
import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder.attribute_exists
fun Table.updateItemByPaths(hashKeyName: String, hashKeyValue: Any, updateActions: List<UpdateAction>) {
val parentPaths = updateActions.map { it.pathOperand.path.parent() }
.filter { it.isNotEmpty() }
.toSet() // to remove duplicates
try {
val builder = ExpressionSpecBuilder()
updateActions.forEach { builder.addUpdate(it) }
if (parentPaths.isNotEmpty()) {
var condition: Condition = ComparatorCondition("=", LiteralOperand(true), LiteralOperand(true))
parentPaths.forEach { condition = condition.and(attribute_exists<Any>(it)) }
builder.withCondition(condition)
}
this.updateItem(hashKeyName, hashKeyValue, builder.buildForUpdate())
} catch (e: ConditionalCheckFailedException) {
this.updateItemByPaths(hashKeyName, hashKeyValue, parentPaths.map { M(it).set(mapOf<String, Any>()) })
this.updateItemByPaths(hashKeyName, hashKeyValue, updateActions)
}
}
private fun String.parent() = this.substringBeforeLast('.', "")
Here is a helper function I wrote in Typescript that works for this a single level of nesting using a recursive method.
I refer to the top-level attribute as a column.
//usage
await setKeyInColumn('customerA', 'address', 'state', "MA")
// Updates a map value to hold a new key value pair. It will create a top-level address if it doesn't exist.
static async setKeyInColumn(primaryKeyValue: string, colName: string, key: string, value: any, _doNotCreateColumn?:boolean) {
const obj = {};
obj[key] = value; // creates a nested value like {address:value}
// Some conditions depending on whether the column already exists or not
const ConditionExpression = _doNotCreateColumn ? undefined:`attribute_not_exists(${colName})`
const AttributeValue = _doNotCreateColumn? value : obj;
const UpdateExpression = _doNotCreateColumn? `SET ${colName}.${key} = :keyval `: `SET ${colName} = :keyval ` ;
try{
const updateParams = {
TableName: TABLE_NAME,
Key: {key:primaryKeyValue},
UpdateExpression,
ExpressionAttributeValues: {
":keyval": AttributeValue
},
ConditionExpression,
ReturnValues: "ALL_NEW",
}
const resp = await docClient.update(updateParams).promise()
if (resp && resp[colName]) {
return resp[colName];
}
}catch(ex){
//if the column already exists, then rerun and do not create it
if(ex.code === 'ConditionalCheckFailedException'){
return this.setKeyInColumn(primaryKeyValue,colName,key, value, true)
}
console.log("Failed to Update Column in DynamoDB")
console.log(ex);
return undefined
}
}
I've got quite similar situation. I can think of only a one way to do this in 1 query/atomically.
Extract map values to top level attributes.
Example
Given I have this post item in DynamoDB:
{
"PK": "123",
"SK": "post",
"title": "Hello World!"
}
And I want to later add an analytics entry to same partition:
{
"PK": "123",
"SK": "analytics#december",
"views": {
// <day of month>: <views>
"1": "12",
"2": "457463",
// etc
}
}
Like in your case, it's not possible to increment/decrement views days counters in single query if analytics item nor views map might not exist (could be later feature or don't want to put empty items).
Proposed solution:
{
"PK": "123",
"SK": "analytics#december",
// <day of month>: <views>
"1": "12", // or "day1" if "1" seems too generic
"2": "457463",
// etc
}
}
Then you could do something like this (increment +1 example):
{
UpdateExpression: "SET #day = if_not_exists(#day, 0) + 1",
AttributeExpressionNames: {
'#day': "1"
}
}
if day attribute value doesn't exist, set default value to 0
if item in database doesn't exist, update API adds a new one
This is my code when I insert multi record into my Collection first time it work well but when i insert second time it say duplicate _id, how to fix it.
//Save Journal Detail
$("#item-list tr").each(function (e) {
//debugger;
var yearMonth = $('#journalDate').val();
var date = moment(yearMonth).format("YYMM");
var prefix = curBranch + "-" + date;
var chartAccountId = $(this).find(".chart-account-id option:selected").val();
if (chartAccountId == "" || chartAccountId == null) {
return;
}
var journalDetailId = idGenerator.genWithPrefix(Acc.Collection.JournalDetail, prefix, 8);
var debit = parseFloat($(this).find(".debit").val());
var credit = parseFloat($(this).find(".credit").val());
if (debit > 0) {
Acc.Collection.JournalDetail.insert({
_id: journalDetailId,
journalId: doc._id,
chartAccountId: chartAccountId,
drCr: debit,
split: "Hello"
});
} else {
Acc.Collection.JournalDetail.insert({
_id: journalDetailId,
journalId: doc._id,
chartAccountId: chartAccountId,
drCr: -credit,
split: "Hello"
});
}
});
return doc;
The problem is likely to be that your $.each() iterations do not wait for your insert calls to finish, therefore the last id in the collection stays the same, and therefore idGenerator.genWithPrefix gives you the same id for multiple successive insert calls.
In the Meteor client, "insert never blocks". So you would have to wait on each insert call before getting the next id via idGenerator.genWithPrefix, or else your next generated id will end up being the same as the previous one you started to insert.
One easy solution could be to transfer that logic to the server (where insert calls are blocking) using a method call, by only giving it the list of documents you wish for it to insert.
On the client:
//Save Journal Detail
var listToInsert = [];
var yearMonth = $('#journalDate').val();
var date = moment(yearMonth).format("YYMM");
var prefix = curBranch + "-" + date;
$("#item-list tr").each(function (e) {
//debugger;
var chartAccountId = $(this).find(".chart-account-id option:selected").val();
if (chartAccountId == "" || chartAccountId == null) {
return;
}
var debit = parseFloat($(this).find(".debit").val());
var credit = parseFloat($(this).find(".credit").val());
listToInsert.push({
journalId: doc._id,
chartAccountId: chartAccountId,
drCr: debit > 0 ? debit : -credit, // simplified with a ternary operator
split: "Hello"
});
});
Meteor.call("insertEntries", listToInsert, prefix, function () {
//done!
});
return doc;
On the server:
Meteor.methods({
'insertEntries': function (listToInsert, prefix) {
for (var i = 0; i < listToInsert.length; i++) {
listToInsert[i]._id = idGenerator.genWithPrefix(Acc.Collection.JournalDetail, prefix, 8);
// Since we're on the server, this insert will be blocking
Acc.Collection.JournalDetail.insert(listToInsert[i]);
}
}
});