How to Remove from List Of Maps in DynamoDB (Must be Atomic) - amazon-dynamodb

I have this Schema:
{
product: S // Primary Key, // my Hash
media: L // List of Maps
}
Each media item will be like this:
[
{
id: S, // for example: id: uuid()
type: S, // for example: "image"
url: S // for example: id: "http://domain.com/image.jpg"
}
]
Sample Data:
{
product: "IPhone 6+",
media: [
{
id: "1",
type: "image",
url: "http://www.apple.com/iphone-6-plus/a.jpg"
},
{
id: "2",
type: "image",
url: "http://www.apple.com/iphone-6-plus/b.jpg"
},
{
id: "3",
type: "video",
url: "http://www.apple.com/iphone-6-plus/overview.mp4"
}
]
}
I want to be able to remove from media list by id.
Something like: "From product: 'IPhone 6+', remove the media with id: '2'"
After the query, the Data should be like this:
{
product: "IPhone 6+",
media: [
{
id: "1",
type: "image",
url: "http://www.apple.com/iphone-6-plus/a.jpg"
},
{
id: "3",
type: "video",
url: "http://www.apple.com/iphone-6-plus/overview.mp4"
}
]
}
How should i express query like this? I saw a post on UpdateItem but i can't find a good example for this query type.
Thanks!

Unfortunately, the API doesn't have this feature. The closest you can do is to delete an entry from "List" data type if you know the index.
I understand that most of the time the index mayn't be available. However, you can take a look at this alternate option if you don't have any other solution.
You also need to understand that even though DynamoDB started supporting the Document data types such as List, Map and Set, you can't perform some actions. Some features are yet to be added in the API. I believe this scenario is one of them.
I have used REMOVE to delete the item from list.
var params = {
TableName : "Product",
Key : {
"product" : "IPhone 6+"
},
UpdateExpression : "REMOVE media[0]",
ReturnValues : "UPDATED_NEW"
};
console.log("Updating the item...");
docClient.update(params, function(err, data) {
if (err) {
console.error("Unable to update item. Error JSON:", JSON.stringify(err, null, 2));
} else {
console.log("UpdateItem succeeded:", JSON.stringify(data));
}
});
This is just for your reference:-
The Delete operator can be used only on SET.
DELETE - Deletes an element from a set.
If a set of values is specified, then those values are subtracted from
the old set. For example, if the attribute value was the set [a,b,c]
and the DELETE action specifies [a,c], then the final attribute value
is [b]. Specifying an empty set is an error.
The DELETE action only supports set data types. In addition, DELETE
can only be used on top-level attributes, not nested attributes.

Related

How to set a DynamoDB Map property value, when the map doesn't exist yet

How do you "upsert" a property to a DynamoDB row. E.g. SET address.state = "MA" for some item, when address does not yet exist?
I feel like I'm having a chicken-and-egg problem because DynamoDB doesn't let you define a sloppy schema in advance.
If address DID already exist on that item, of type M (for Map), the internet tells me I could issue an UpdateExpression like:
SET #address.#state = :value
with #address, #state, and :value appropriately mapped to address, state, and MA, respectively.
But if the address property does not already exist, this gives an error:
'''
ValidationException: The document path provided in the update expression is invalid for update
'''
So.. it appears I either need to:
Figure out a way to "upsert" address.state (e.g., SET address = {}; SET address.state = 'MA' in a single command)
or
Issue three (!!!) roundtrips in which I try it, SET address = {}; on failure, and then try it again.
If the latter.... how do I set a blank map?!?
Ugh.. I like Dynamo, but unless I'm missing something obvious this is a bit crazy..
You can do it with two round trips, the first conditionally sets an empty map for address if it doesn't already exist, and the second sets the state:
db.update({
UpdateExpression: 'SET #a = :value',
ConditionExpression: 'attribute_not_exists(#a)',
ExpressionAttributeValues: {
":value": {},
},
ExpressionAttributeNames: {
'#a': 'address'
}
}, ...);
Then:
db.update({
UpdateExpression: 'SET #a.#b = :v',
ExpressionAttributeNames: {
'#a': 'address',
'#b': 'state'
},
ExpressionAttributeValues: {
':v': 'whatever'
}
}, ...);
You cannot set nested attributes if the parent document does not exist. Since address does not exist you cannot set the attribute province inside it. You can achieve your goal if you set address to an empty map when you create the item. Then, you can use the following parameters to condition an update on an attribute address.province not existing yet.
var params = {
TableName: 'Image',
Key: {
Id: 'dynamodb.png'
},
UpdateExpression: 'SET address.province = :ma',
ConditionExpression: 'attribute_not_exists(address.province)',
ExpressionAttributeValues: {
':ma': 'MA'
},
ReturnValues: 'ALL_NEW'
};
docClient.update(params, function(err, data) {
if (err) ppJson(err); // an error occurred
else ppJson(data); // successful response
});
By the way, I had to replace state with province as state is a reserved word.
Another totally different method is to simply create the address node when creating the parent document in the first place. For example assuming you have a hash key of id, you might do:
db.put({
Item: {
id: 42,
address: {}
}
}, ...);
This will allow you to simply set the address.state value as the address map already exists:
db.update({
UpdateExpression: 'SET #a.#b = :v',
AttributeExpressionNames: {
'#a': 'address',
'#b': 'state'
},
AttributeExpressionValues: {
':v': 'whatever'
}
}, ...);
Some kotlin code to do this recursively regardless how deep it goes. It sets existence of parent paths as condition and if condition check fails, recursively creates those paths first. It has to be in the library's package so it can access those package private fields/classes.
package com.amazonaws.services.dynamodbv2.xspec
import com.amazonaws.services.dynamodbv2.document.Table
import com.amazonaws.services.dynamodbv2.model.ConditionalCheckFailedException
import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder.attribute_exists
fun Table.updateItemByPaths(hashKeyName: String, hashKeyValue: Any, updateActions: List<UpdateAction>) {
val parentPaths = updateActions.map { it.pathOperand.path.parent() }
.filter { it.isNotEmpty() }
.toSet() // to remove duplicates
try {
val builder = ExpressionSpecBuilder()
updateActions.forEach { builder.addUpdate(it) }
if (parentPaths.isNotEmpty()) {
var condition: Condition = ComparatorCondition("=", LiteralOperand(true), LiteralOperand(true))
parentPaths.forEach { condition = condition.and(attribute_exists<Any>(it)) }
builder.withCondition(condition)
}
this.updateItem(hashKeyName, hashKeyValue, builder.buildForUpdate())
} catch (e: ConditionalCheckFailedException) {
this.updateItemByPaths(hashKeyName, hashKeyValue, parentPaths.map { M(it).set(mapOf<String, Any>()) })
this.updateItemByPaths(hashKeyName, hashKeyValue, updateActions)
}
}
private fun String.parent() = this.substringBeforeLast('.', "")
Here is a helper function I wrote in Typescript that works for this a single level of nesting using a recursive method.
I refer to the top-level attribute as a column.
//usage
await setKeyInColumn('customerA', 'address', 'state', "MA")
// Updates a map value to hold a new key value pair. It will create a top-level address if it doesn't exist.
static async setKeyInColumn(primaryKeyValue: string, colName: string, key: string, value: any, _doNotCreateColumn?:boolean) {
const obj = {};
obj[key] = value; // creates a nested value like {address:value}
// Some conditions depending on whether the column already exists or not
const ConditionExpression = _doNotCreateColumn ? undefined:`attribute_not_exists(${colName})`
const AttributeValue = _doNotCreateColumn? value : obj;
const UpdateExpression = _doNotCreateColumn? `SET ${colName}.${key} = :keyval `: `SET ${colName} = :keyval ` ;
try{
const updateParams = {
TableName: TABLE_NAME,
Key: {key:primaryKeyValue},
UpdateExpression,
ExpressionAttributeValues: {
":keyval": AttributeValue
},
ConditionExpression,
ReturnValues: "ALL_NEW",
}
const resp = await docClient.update(updateParams).promise()
if (resp && resp[colName]) {
return resp[colName];
}
}catch(ex){
//if the column already exists, then rerun and do not create it
if(ex.code === 'ConditionalCheckFailedException'){
return this.setKeyInColumn(primaryKeyValue,colName,key, value, true)
}
console.log("Failed to Update Column in DynamoDB")
console.log(ex);
return undefined
}
}
I've got quite similar situation. I can think of only a one way to do this in 1 query/atomically.
Extract map values to top level attributes.
Example
Given I have this post item in DynamoDB:
{
"PK": "123",
"SK": "post",
"title": "Hello World!"
}
And I want to later add an analytics entry to same partition:
{
"PK": "123",
"SK": "analytics#december",
"views": {
// <day of month>: <views>
"1": "12",
"2": "457463",
// etc
}
}
Like in your case, it's not possible to increment/decrement views days counters in single query if analytics item nor views map might not exist (could be later feature or don't want to put empty items).
Proposed solution:
{
"PK": "123",
"SK": "analytics#december",
// <day of month>: <views>
"1": "12", // or "day1" if "1" seems too generic
"2": "457463",
// etc
}
}
Then you could do something like this (increment +1 example):
{
UpdateExpression: "SET #day = if_not_exists(#day, 0) + 1",
AttributeExpressionNames: {
'#day': "1"
}
}
if day attribute value doesn't exist, set default value to 0
if item in database doesn't exist, update API adds a new one

Any way to have Simple Schema in Meteor validate a specific array index?

From what I understand in the docs, you can define your schema like this:
MySchema = new SimpleSchema({
//This says that the addresses key is going to contain an array
addresses: {
type: [Object],
},
// To indicate the presence of an array, use a $:
"addresses.$.street": {
type: String,
},
"addresses.$.city": {
type: String,
}
});
Ok, I get this part. But what if I wanted to validate the contents in a specific array index? I want something like this:
MySchema = new SimpleSchema({
//This says that the itemsOrdered key is going to contain an array
itemsOrdered: {
type: [Object],
},
// Here I want to validate certain indexes in the array.
"itemsOrdered.0.sku": {
type: String
},
"itemsOrdered.0.price": {
type: Number
},
"itemsOrdered.1.sku": {
type: String
},
"itemsOrdered.1.price": {
type: Number
},
"itemsOrdered.1.quantity": {
type: Number
},
"itemsOrdered.2.sku": {
type: String
},
"itemsOrdered.2.price": {
type: Number
},
"itemsOrdered.2.customerNotes": {
type: String
optional: true
}
});
So here I'm trying to validate the values inside array index 0, 1, and 2. Each array index has a different item that has been ordered.
Normally I would use a hash table data structure, but for this purpose I need to preserve order which is why I'm using an array.
When I try to run this code I get an error Cannot read property 'blackbox' of undefined
Have you considered custom validation?
https://github.com/aldeed/meteor-simple-schema/blob/master/README.md#custom-validation
According to the doc within the function the key property of this will provide the information you want. So you could have something like:
MySchema = new SimpleSchema({
//This says that the itemsOrdered key is going to contain an array
itemsOrdered: {
type: [Object],
},
// Here I want to validate certain indexes in the array.
"itemsOrdered.$.sku": {
type: String,
custom: function () {
var key = this.key,
re = /\d+/;
var index = Number(key.match(re)[0]);
// Do some custom validation
}
},
"itemsOrdered.$.price": {
type: Number
},
"itemsOrdered.$.quantity": {
type: Number,
optional: true
},
"itemsOrdered.$.customerNotes": {
type: String,
optional: true
}
});
Here I put the validation logic in the sku field since it's required.

Kendo UI Grid with Drop down Template Not updating value after save to server

We are using a server based simple grid. The grid reads and updates data for a remote data source. It has two columns that are using drop down editors. Everything seems to work fine except that after saving, when editor closes, the changed values are not displayed in the edited row. It still shows the old value. When we try to refresh the grid using the sync event, it changes the order of the rows however, it does update the values on refresh.
It seems like the template function is not executed after the update is completed. How to edit the grid / code to ensure that the changed value is reflected in the grid?
Grid Definition code:
$("#servicetype-grid").kendoGrid({
pageable: true,
toolbar: [{name: "create", text: ""}, { template: kendo.template($("#servicetype-search-template").html())}],
columns: [
{
field: "serviceName", title: "Service Name"
},
{
field: "billCalculationTypeId",
editor: calculationTypeDropDownEditor,
template: function(dataItem) {
return kendo.htmlEncode(dataItem.billCalculationTypeName);
},
title: "Bill Calculation Type"
},
{
field: "payCalculationTypeId",
editor: calculationTypeDropDownEditor,
template: function(dataItem) {
return kendo.htmlEncode(dataItem.payCalculationTypeName);
},
title: "Pay Calculation Type"
},
{
command: [
{ name: "edit", text: { edit: "", cancel: "", update: "" }},
{ name: "destroy", text:""}
],
title: "Actions"
}
],
dataSource: dataSource,
sortable: {
mode: "single",
allowUnsort: false
},
dataBound: function(e) {
setToolTip();
},
edit: function(e) {
$('.k-grid-update').kendoTooltip({content: "Update"});
$('.k-grid-cancel').kendoTooltip({content: "Cancel"});
},
cancel: function(e) {
setToolTip();
},
editable: {
mode: "inline",
confirmation: "Are you sure that you want to delete this record?"
}
});
Drop down function is defined as:
function calculationTypeDropDownEditor(container, options) {
$('<input required data-text-field="name" data-value-field="id" data-bind="value:' + options.field + '"/>')
.appendTo(container)
.kendoDropDownList({
autoBind: false,
dataSource: {
transport: {
read: {
dataType: "jsonp",
url: baseURL + "rips/services/calculationType/lookupList"
}
}
}
});
}
After doing some search on Google and browsing through different examples on Kendo site, I finally was able to resolve this issue. Following is my understanding of the problem and solution:
When we are using drop down box or combo box as a custom editor, generally we have a different datasource that contains a list options with an id and a value to show. I defined the template as "#=field.property#" of the field that I was looking up. In my case it was calculation type. Following is my model:
model: {
id: "serviceId",
fields: {
serviceName: { field:"serviceName", type: "string", validation: { required: { message: "Service Name is required"}} },
billCalculationType: { field: "billCalculationType", validation: { required: true}},
payCalculationType: { field: "payCalculationType", validation: { required: true} }
}
In above, billCalculationType and payCalculationType are supposed to be drop down list values displaying the calculation type name but storing the id of the corresponding calculation type. So, in my grid definition, I added following:
{ field: "billCalculationType",
editor: calculationTypeDropDownEditor,
template:"#=billCalculationType.name#",
title: "Bill Calculation Type" },
Where calculation dropdown editor is a function that builds a drop down from external data source. So, it works fine. However, for the template definition to work in (field.property) format, the server must return the value as a class for this field and not a simple text. So, in my server service, I returned in following format:
{"billCalculationType":{"id":"4028828542b66b3a0142b66b3b780001","name":"Hourly","requiredDetails":false},"payCalculationType":{"id":"4028828542b66b3a0142b66b3b960002","name":"Kilometers","requiredDetails":false},"serviceId":"4028828542b66b3a0142b66b3c16000a","serviceName":"XYZ"}
Notice that the billCalculationType and payCalculationType are returned as objects with name and id as properties. This allows the template to work properly.
Hope this helps.

Grid content JSON conversion

I have a grid where user and add new rows as many as they want. After adding all the rows, they click the "Save" button. On Save button click, I want to send all the data entered by the user in JSON format to the server side code (i.e. a servlet in my case)
Below is the model and store definition:
Ext.define('Plant', {
extend: 'Ext.data.Model',
fields: [
// the 'name' below matches the tag name to read, except 'availDate'
// which is mapped to the tag 'availability'
{name: 'common', type: 'string'},
{name: 'botanical', type: 'string'},
{name: 'light'},
{name: 'price', type: 'float'},
// dates can be automatically converted by specifying dateFormat
{name: 'availDate', mapping: 'availability', type: 'date', dateFormat: 'm/d/Y'},
{name: 'indoor', type: 'bool'}
]
});
// create the Data Store
var store = Ext.create('Ext.data.Store', {
// destroy the store if the grid is destroyed
autoDestroy: true,
model: 'Plant'
});
On click of the save button, I am able to get the store like this:
{
text: 'Save',
handler : function(){
//Getting the store
var records = grid.getStore();
console.log(records.getCount());
Ext.Ajax.request({
url: '/CellEditing/CellEditingGridServlet',
method: 'POST',
jsonData: {
//How to assign the store here such that
//it is send in a JSON format to the server?
},
callback: function (options, success, response) {
}
});
}
But I don't know like how to convert the store content into JSON and send it in the jsonData of the ajax request.
I want the JSON data something like this in the server side:
{"plantDetails":
[
{
"common": Plant1,
"light": 'shady',
"price": 25.00,
"availDate": '05/05/2013',
"indoor": 'Yes'
},
{
"common": Plant2,
"light": 'most shady',
"price": 15.00,
"availDate": '12/09/2012',
"indoor": 'No'
},
]
}
Please let me know how to achieve this.
Regards,
Agreed with Neil, the right way to do this is through an editable store outfited with a proxy and a writer. See example here: http://docs.sencha.com/ext-js/4-1/#!/example/grid/cell-editing.html
Store
writer :
{
type : 'json',
allowSingle : true
}
Experiment with allowSingle as per your use case
In your controller
//if you want to set extra params
yourStore.getProxy().setExtraParam("anyParam",anyValue);
// sync the store
yourStore.sync({success : function() {
yourGrid.setLoading(false);
.. },
scope : this // within the scope of the controller
});
You should be creating the model with a new id ( you can ignore it at the server side and use your own key generation , but it lets extjs4 for its internal purposes know that a new record has been created).
creating a model instance
var r = Ext.create('yourModel', { id: idGen++, propA : valA , ... });
insert to grid
store.insert(0,r);
var editPlugin = grid.getPlugin(editPluginId);
editPlugin.startEdit(0,1);
Once you receive a response back the id's can be update to their true value.
in the Store
reader :
{
type : 'json',
root : 'yourdata',
successProperty : 'success',
idProperty : 'id'
}
If you were to use the same grid for handling and editing then you could use the write event or the appropriate event
for more advanced handling in the Store
listeners :
{
write : function(store,operation, eOpts)
{
var insertedData = Ext.decode(operation.response.responseText);
.. do something
}
}
I would recommend using the mvc architecture of Extjs4
This is what I tried and it seems to work:
var store = Ext.create('Ext.data.Store', {
// destroy the store if the grid is destroyed
autoDestroy: true,
model: 'Plant',
proxy: {
type: 'ajax',
url: '/CellEditing/CellEditingGridServlet',
writer: {
type: 'json',
root: 'plantDetails'
}
}
handler : function(){
grid.getStore().sync();
But I am getting an additional parameter in the JSON at the server side:
"id": null
I don't have this id set in my model then where is this coming from? Is there some way to set some values to it rather than having a default null value?

ExtJS4 - Reconfiguring a grid in ASP.NET - JSON structure issue

One of ASP.NET's security features is proving to be a mountain to scale here - the "d" property addition when returning a JSON response appears to be confusing ExtJS when I attempt to reconfigure a gridpanel dynamically, causing it to fail when attempting to generate new column structure.
I followed this solution by nicholasnet:
http://www.sencha.com/forum/showthread.php?179861-Dynamic-grid-columns-store-fields
and it works beautifully, until the JSON payload is wrapped around the "d" property, e.g.
{"d":{
"metaData": {
"root": "data",
"fields": [{
"type": "int",
"name": "id",
"hidden": true,
"header": "id",
"groupable": false,
"dataIndex": "id"
}, ...omitted for brevity...]
},
"success": true,
"data": [{
"id": "1",
"controller": "Permissions",
"description": "Allow to see permission by roles",
"administrator": true,
"marketing": false
}]
}
}
I can't work out how to tell ExtJS to skirt around this problem. I've tried setting the "root" property of the AJAX reader to "d.data" but that results in the grid showing the correct number of rows but no data at all.
I've all the property descriptors required for column metadata ("name", "header", "dataIndex") in the JSON so I don't believe the JSON structure to be the cause. My main lead at the moment is that on the event handler:
store.on
({
'load' :
{
fn: function(store, records, success, operation, eOpts)
{
grid.reconfigure(store,store.proxy.reader.fields);
},
scope: this
}
}, this);
The fields in historyStore.proxy.reader.fields part is undefined when I pass the "d"-wrapped JSON. Anyone have any ideas on why this is or how to solve this issue?
edit: my Store/proxy
Ext.define('pr.store.Store-History', {
extend: 'Ext.data.Store',
model: 'pr.model.Model-History',
proxy: {
type: 'ajax',
url: '/data/history.json',
reader: {
type: 'json',
root: 'd'
}
}
});
Ext.define('pr.store.Store-History', {
extend: 'Ext.data.Store',
model: 'pr.model.Model-History',
proxy: {
type: 'ajax',
url: '/data/history.json',
reader: {
type: 'json',
root: 'data',
readRecords: function(data) {
//this has to be before the call to super because we use the meta data in the superclass readRecords
var rootNode = this.getRoot(data);
if (rootNode.metaData) {
this.onMetaChange(rootNode.metaData); // data used to update fields
}
/**
* #deprecated will be removed in Ext JS 5.0. This is just a copy of this.rawData - use that instead
* #property {Object} jsonData
*/
this.jsonData = rootNode;
return this.callParent([rootNode]); // data used to get root element and then get data from it
},
}
}
});
Update:
you are not getting fields in reader because the default code for getting fields from data doesn't handle your wrapped data, so you need to change 'readRecords' function to handle your custom data

Resources