Update or create nested element in dynamoDB - amazon-dynamodb

I want to update or create DynamoDB item to get next element:
{
"id": 156,
"date": 12323233.000,
"countries": {
"ua": 1,
"ru": 2}
}
I use python and boto3. So I can check if field countries exist and if not add it. But that will mean 2 DB requests.
table.update_item(
Key={
'id': 156,
'date': date,
},
UpdateExpression='SET countries = if_not_exists(countries, :countries)',
ExpressionAttributeValues={
':countries': {},
},
)
table.update_item(
Key={
'id': 156,
'date': date,
},
UpdateExpression='ADD countries.#country :inc',
ExpressionAttributeNames={"#country": country},
ExpressionAttributeValues={
':inc': 1
},
)
Is there any way to merge this 2 requests in one?

I had to do something like this recently, and took a while to figure out. I've got a count I want to increment if a page doesn't already exist in my set of "done" pages. If not, it increments and adds the page number to the set. Took a while to realize you can 'append' to a list, but have to 'add' to a set.
try:
res = dbt.update_item(
Key={'pk': 'doc3', 'sk': 'na'},
ReturnConsumedCapacity='INDEXES', ReturnValues='ALL_NEW',
ExpressionAttributeNames={
'#count': 'count',
'#done': 'done',
},
ExpressionAttributeValues={
':1': 1,
':page': page,
':pagelist': set([page]),
},
ConditionExpression="(NOT contains(done, :page))",
UpdateExpression="ADD #done :pagelist, #count :1",
)
print(f'rand int page={page} count={res["Attributes"]["count"]}'
f' CU={res["ConsumedCapacity"]["Table"]}')
except ClientError as err:
if err.response['Error']['Code'] == 'ConditionalCheckFailedException':
print('Already got page=%s (%s)' % (page, err))
else:
raise

Related

Great Expectations - Result validation for row_count and column_freshness

I would like to validate results for row count and column freshness on some data on AWS. I am using a check_config.json file to configure the checks. I use terraform to make a Glue job to run the check and throw the result to DynamoDB. The result in DynamoDB is not elaborate and I would like the result to be more specific on the exact results obtained before marking a check as fail or pass. I would like to see, for example, when was the table last modified(column freshness) and number of rows obtained after a count (expect_row_count).
Below is the current result in DynamoDB:
Below is the json code:
{
"table": "table1",
"checks": [
{
"check": "custom_expect_column_to_be_fresh",
"parameters": {
"columns": [
"column1"
],
"strftime_format": "%Y-%m-%d",
"threshold_days": 0,
"threshold_hours": 10
}
},
{
"check": "expect_table_row_count_to_be_between",
"result_format" : "COMPLETE",
"include_config": "True",
"parameters": {
"min_value": 1,
"max_value": 100000
},
"alarm" : {
"threshold": 100,
"period": 3600
}
}
]
}
I was expecting a more elaborate result on how many rows were obtained before the row_count is marked as a failure and I also want to see the last table modification timestamp before column freshness marks as a failure.

dynamodb: how to increment a value in map

I am trying to use dynamodb to maintain a map of names with their values
eg. {"scores": {"player-a": 10}}
I also wish to use increment operator to perform atomic increments.
However, i could find very little documentation on how to use/update dynamodb maps.
Here's the python code I have so far
import boto3
ddb = boto3.client('dynamodb')
ddb.update_item(TableName='ledger', Key={'week': {'S': '06-12'}},
UpdateExpression='SET scores.player-a = scores.player-a + :val',
ExpressionAttributeValues={':val': {'N': '12'}})
DynamoDB update item uses ExpressionAttributeNames to prevent special characters in an attribute name from being misinterpreted in an expression.
Your update item consists of "player-a" as a key name which has "-" (hyphen) in it.
ddb.update_item(
TableName='ledger',
Key={
'week': {
'S': '06-12'
}
},
UpdateExpression='SET scores.#s = scores.#s + :val",
ExpressionAttributeNames={
"#s": "player-a"
},
ExpressionAttributeValues={
':val': {
'N': '12'
}
}
)

Multiple range keys in couchdb views

I've been searching for a solution since few hours without success...
I just want to do this request in couchdb with a view:
select * from database where (id >= 3000000 AND id <= 3999999) AND gyro_y >= 1000
I tried this:
function(doc) {
if(doc.id && doc.Gyro_y){
emit([doc.id,doc.Gyro_y], null);
}
}
Here is my document (record in couchdb):
{
"_id": "f97968bee9674259c75b89658b09f93c",
"_rev": "3-4e2cce33e562ae502d6416e0796fcad1",
"id": "30000002",
"DateHeure": "2016-06-16T02:08:00Z",
"Latitude": 1000,
"Longitude": 1000,
"Gyro_x": -242,
"Gyro_y": 183,
"Gyro_z": -156,
"Accel_x": -404,
"Accel_y": -2424,
"Accel_z": -14588
}
I then do an HTTP request like so:
http://localhost:5984/arduino/_design/filter/_view/bygyroy?startkey=["3000000",1000]&endkey=["3999999",9999999]&include_docs=true
I get this as an answer:
{
total_rows: 10,
offset: 8,
rows: [{
id: "f97968bee9674259c75b89658b09f93c",
key: [
"01000002",
183
],
value: null,
doc: {
_id: "f97968bee9674259c75b89658b09f93c",
_rev: "3-4e2cce33e562ae502d6416e0796fcad1",
id: "30000002",
DateHeure: "2016-06-16T02:08:00Z",
Latitude: 1000,
Longitude: 1000,
Gyro_x: -242,
Gyro_y: 183,
Gyro_z: -156,
Accel_x: -404,
Accel_y: -2424,
Accel_z: -14588
}
}
]
}
So it's working for the id but it's not working for the second key gyro_y.
Thanks for your help.
When you specify arrays as your start/end keys, the results are filtered in a "cascade". In other words, it moves from left to right, and only if something was matched by the previous key, will it be matched by the next key.
In this case, you'll only find Gyro_y >= 1000 when that document also matches the first condition of 3000000 <= id <= 3999999.
Your SQL example does not translate exactly to what you are doing in CouchDB. In SQL, it'll find both conditions and then find the intersection amongst your resulting rows. I would read up on view collation to understand these inner-workings of CouchDB.
To solve your problem right now, I would simply switch the order you are emitting your keys. By putting the Gyro_y value first, you should get the results you've described.

Query to get exact matches of Elastic Field with multile values in Array

I want to write a query in Elastic that applies a filter based on values i have in an array (in my R program). Essentially the query:
Matches a time range (time field in Elastic)
Matches "trackId" field in Elastic to any value in array oth_usr
Return 2 fields - "trackId", "propertyId"
I have the following primitive version of the query but do not know how to use the oth_usr array in a query (part 2 above).
query <- sprintf('{"query":{"range":{"time":{"gte":"%s","lte":"%s"}}}}',start_date,end_date)
view_list <- elastic::Search(index = "organised_recent",type = "PROPERTY_VIEW",size = 10000000,
body=query, fields = c("trackId", "propertyId"))$hits$hits
You need to add a terms query and embed it as well as the range one into a bool/must query. Try updating your query like this:
terms <- paste(sprintf("\"%s\"", oth_usr), collapse=", ")
query <- sprintf('{"query":{"bool":{"must":[{"terms": {"trackId": [%s]}},{"range": {"time": {"gte": "%s","lte": "%s"}}}]}}}',terms,start_date,end_date)
I'm not fluent in R syntax, but this is raw JSON query that works.
It checks whether your time field matches given range (start_time and end_time) and whether one of your terms exact matches trackId.
It returns only trackId, propertyId fields, as per your request:
POST /indice/_search
{
"_source": {
"include": [
"trackId",
"propertyId"
]
},
"query": {
"bool": {
"must": [
{
"range": {
"time": {
"gte": "start_time",
"lte": "end_time"
}
}
},
{
"terms": {
"trackId": [
"terms"
]
}
}
]
}
}
}

Kendo - Grid - Aggregate with Complex Objects

I have a Kendo UI grid. The grid has a datasource with complex object data. For example, {"foo": {"bar" : 10}}. Although the column field can navigate the object graph (i.e. foo.bar), the aggregate field doesn't seem to be able to.
Here's the code:
var grid = $("#grid").kendoGrid({
dataSource: {
data: [
{"foo": {"bar": 10}},
{"foo": {"bar": 20}}
],
aggregate: [
{field: "foo.bar", aggregate: "sum"}
]
},
columns: [
{
field: "foo.bar",
footerTemplate: "Sum: #= sum # "
}
]
}).data("kendoGrid");
Here's the fiddle:
http://jsfiddle.net/e6shF/1/
Firebug reports "TypeError: data.foo is undefined" in line 8 of kendo.all.min.js.
Am I doing something incorrectly? If this is a bug in Kendo, is there a way to work around this? I have to keep the objects complex.
Here's a "better" anwser from Kendo Support:
The behavior you are experiencing is caused by the fact that the "path" you have specified will be used as a key in the map created as result of the aggregation. Producing a object similar to the following:
{ "foo.bar" : { sum: 30 } }
Unfortunately, this construct is not supported by the footer template generation and will not be resolved correctly. A possible workaround for this scenario is to use a function instead. I have modify the sample in order to illustrate this.
var grid = $("#grid").kendoGrid({
dataSource: {
data: [
{"foo": {"bar": 10}},
{"foo": {"bar": 20}}
],
aggregate: [
{field: "foo.bar", aggregate: "sum"}
]
},
columns: [
{
field: "foo.bar",
footerTemplate: function(data) { return "Sum: " + data["foo.bar"].sum; }
}
]
}).data("kendoGrid");
It is not possible to have complex objects in aggregates since dynamically generated function for evaluating it, thinks that foo.bar is the name of the field (just one field)?
Do you really need that complex field?
I might understand that the server (providing the data of the grid) sends that complex foo but you can always flatten it using parse or data functions in the datasource. Something like this:
var grid = $("#grid").kendoGrid({
dataSource:{
data:[
{"foo":{"bar":10}},
{"foo":{"bar":20}}
],
aggregate:[
{field:"foo_bar", aggregate:"sum"}
],
schema: {
parse:function (data) {
var res = [];
$.each(data, function (idx, elem) {
res.push({ "foo_bar":elem.foo.bar })
});
return res;
}
}
},
columns: [
{
field: "foo_bar",
footerTemplate:"Sum: #= sum # "
}
]
}).data("kendoGrid");
Where I transform received foo.bar into foo_bar and use this for aggregation.

Resources