This is my source format:
{
"message":[
{"name":"sensorID","value":"5"},
{"name":"eventT","value":"2021-04-16T19:11:26.149Z"},
{"name":"pressure","value":"150"}
]
}
Looking to flatten it out into a table:
sensorID
eventT
pressure
5
"2021-04-16T19:11:26.149Z"
150
Cannot for the life of me figure it out.
Splitting the array just gets me a more nested array:
test
| project ray=array_split(message, 1)
And using mv-expand gets me two separate rows:
test
| mv-expand message
At my wits end. any help greatly appreciated.
if the schema is unknown in advance, you could try something like this (using mv-apply, summarize make_bag() and bag_unpack())
datatable(d:dynamic)
[
dynamic({
"message":[
{"name":"sensorID","value":"5"},
{"name":"eventT","value":"2021-04-16T19:11:26.149Z"},
{"name":"pressure","value":"150"}
]}),
dynamic({
"message":[
{"name":"sensorID","value":"55"},
{"name":"eventT","value":"2021-03-16T19:11:26.149Z"},
{"name":"pressure","value":"1515"}
]})
]
| mv-apply d.message on (
summarize b = make_bag(pack(tostring(d_message.name), d_message.value))
)
| project b
| evaluate bag_unpack(b)
eventT
pressure
sensorID
2021-03-16 19:11:26.1490000
1515
55
2021-04-16 19:11:26.1490000
150
5
Related
I have a kusto query which summarize an array based on values in an id column.
The query is
Table
| where {some condition}
| extend d = parse_json(events)
| mv-expand d
| extend value=parse_json(d)["intValue"]
| project value, Id
| summarize make_list(value) by Id
The result is
Id
value
ab8e
[1664379494002, 1664379485020, 1664379487998, 1664379473022]
3dBc
[ 1664383366022,1664383372025, 1664381572017, 1664381566025]
My question is can I do math operation in the array 'value'?
e.g. I want to array[1] - array[0] + array[1] - array[2] for each row?
yes, you can do that:
datatable(Id:string, value:dynamic)
[
'ab8e', dynamic([1664379494002, 1664379485020, 1664379487998, 1664379473022]),
'3dBc', dynamic([ 1664383366022,1664383372025, 1664381572017, 1664381566025]),
]
| extend result = tolong(value[1]) - tolong(value[0]) + tolong(value[1]) - tolong(value[2])
Id
value
result
ab8e
[1664379494002,1664379485020,1664379487998,1664379473022]
-11960
3dBc
[1664383366022,1664383372025,1664381572017,1664381566025]
1806011
I have DynamoDB records using this structure, category being the partition key:
{ 'category':'1',
'name' : 'wind instruments',
'instruments' : [
{ 'name' : 'oboe',
'count' : 3 },
{ 'name' : 'recorder',
'count' : 2 },
{ 'name' : 'trumpet',
'count' : 2} ]
}
{ 'category':'1',
'name' : 'string instruments',
'instruments' : [
{ 'name' : 'violin',
'count' : 6 },
{ 'name' : 'cello',
'count' : 3 } ]
}
I would like to run the following queries on this DynamoDB table :
how many kinds of instrument do I have ? (answer is 5)
how many instruments do I have in total ? (answer is 16)
By either using :
AWS CLI
PartiQL
Python code
Thanks for your help !
side note: please do not suggest any change in the data model, it's designed like this on purpose (The app reading the instruments inventory per instrument category is the principal access pattern).
DynamoDB does not do these types of queries for you, there is no count or distinct for example. My suggestion is to either have an aggregate table or store aggregate data back to the table. To do this you would use DynamoDB Streams.
----------------------------------------------------------------------------
| pk | sk | instruments | inst_kind | inst_type |
----------------------------------------------------------------------------
| category1 | wind | map of instrumnets | | |
----------------------------------------------------------------------------
| category1 | string | map of instruments | | |
----------------------------------------------------------------------------
| category1 | aggregate | | 5 | 16 |
----------------------------------------------------------------------------
To update the aggregates you use Streams and Lambda and adjust the values each time you modify the item.
To retrieve the aggregates, you simply do a GetItem where PK = "{categoryId}" and SK = "aggreagate"
The following query returns the data that I need:
let timeSpn = bin(ago(60m),1m);
requests
| where cloud_RoleName == "myApp"
| where success == "False"
| where timestamp > timeSpn
| make-series count() on timestamp from timeSpn to now() step 1m by application_Version
The problem is that the result consist of 2 lines (one for each application_Version and not 120 lines (one for each minute and for each version).
I have to use make-series and not the simple summarize because I need the "zero" values.
You can do it using the mv-expand operator
Here's an example from Back-fill Missing Dates With Zeros in a Time Chart:
let start=floor(ago(3d), 1d);
let end=floor(now(), 1d);
let interval=5m;
requests
| where timestamp > start
| make-series counter=count() default=0
on timestamp in range(start, end, interval)
| mvexpand timestamp, counter
| project todatetime(timestamp), toint(counter)
| render timechart
I would like to create a short Catalog for myself out of a database which would show what tables and fields are available there with the combination of SAPPLY(), LAPPLY() etc. and DBListNames.
So far I only get this far but it returns a "0" character variable:
catalog <- lapply(list_of_tables, function(t) dbListFields(con, name = paste0(t)))
So I would like to create an output like this:
+------------+--------------------+
| DB TABLES | FIELDS |
+------------+--------------------+
| ORDERS | "PRODUCT", "TIME" |
| CLIENTS | "ID", "NAME" |
| PROMOTIONS | "DATE", "DISCOUNT" |
+------------+--------------------+
I haven't used these kind of loops and I would like to start it here..
Thank you for your support in advance!
I am trying to get the json data (in form of a list of key-value pairs) in one of my data table cells and convert that into a dynamic table of sorts.
T
| where id == "xyz"
| project telem_obj
The data in the telem_obj cell is of the format
[
{
"Value": "SomeKey01",
"Key": "0"
},
{
"Value": "SomeKey02",
"Key": "1"
}
]
My end objective is to get a table of the form;
|Key | Value |
|SomeValue01 | 0 |
|SomeValue02 | 1 |
I have managed to do this by taking out the static data and creating atable out of it.
print EnumVals = dynamic(
[
{
"Value": "SomeKey01",
"Key": "0"
},
{
"Value": "SomeKey02",
"Key": "1"
}
]
)
| mvexpand EnumVals
| evaluate bag_unpack(EnumVals)
I am not sure how can I go about taking result of my query, extracting this list of json objects from it and convert it into a new dynamic table. I cannot find any example which works on a list of objects.
After a good night's sleep, i found how to do it
T
| take 1
| mvexpand telem_obj
| evaluate bag_unpack(telem_obj)
| project Value, Key
my mistake was I was trying to force the actual query inside a dynamic function.
print EnumVals = dynamic(
T
| where id == "xyz"
| project telem_obj
)
| mvexpand EnumVals
| evaluate bag_unpack(EnumVals)