Am trying to do a simple if/then/else using JMESPath
For example: 'if the input is a string, return the string, else return the "value" property of the input'. An input of "abc" would return "abc". An input of {"value":"def"} would return "def"
With jq this is easy: if .|type == "string" then . else .value end
With JMESPath, I can get the type
type(#)
or the input:
#
or the value property:
value
but I have not found a way to combine them into an if-then-else. Is there any way to do this?
It is possible but not cleanly. The general form is to:
Make the value you are testing an array (wrap in square braces)
Apply the map function to map the filtered array to what value you want if true
At this point you have an array that is populated with one (true) item if the array filter passed, otherwise it is empty
Concat to that array one item (the false value)
Finally, take item at index 0 in this array - which will be the result of the condition
This should allow you to also derive possible transformations for both the false and true conditions
For example, if the test data is as so:
{
"test": 11
}
Depending on the value you can get either produce the results (using test data 11 and 2 as example):
"Yes, the value is 11 which is greater than 10"
OR
"No the value is 2 which is less than or equal to 10"
Like so:
[
map(
&join(' ', ['Yes, the value is', to_string(#), 'which is greater than 10']),
[test][? # > `10`]
),
join(' ', ['No the value is', to_string(test), ' which is less than or equal to 10'])
][] | #[0]
So to abstract a template:
[
map(
&<True Expression Here>,
[<Expression you are testing>][? # <Test Expression>]
),
<False Expression Here>)
][] | #[0]
people[?general.id !=100] || people
{
"people": [
{
"general": {
"id": 100,
"age": 20,
"other": "foo",
"name": "Bob"
},
"history": {
"first_login": "2014-01-01",
"last_login": "2014-01-02"
}
},
{
"general": {
"id": 101,
"age": 30,
"other": "bar",
"name": "Bill"
},
"history": {
"first_login": "2014-05-01",
"last_login": "2014-05-02"
}
}
]
}
if else condition works here
Related
Suppose we have a structure:
{
"nested_items": [
{
"nested_sample0": "1",
"nested_sample1": "test",
"nested_sample2": "test",
"nested_sample3": {
"type": "type"
},
"nested_sample": null
},
{
"nested_sample0": "1",
"nested_sample1": "test",
"nested_sample2": "test",
"nested_sample3": {
"type": "type"
},
"nested_sample1": null
},
...
],
"sample1": 1233,
"id": "ed68ca34-6b59-4687-a557-bdefc9ec2f4b",
"sample2": "",
"sample3": "test",
"sample4": "test",
"_ts": 1656503348
}
I want to retrieve documents by id by with limit of "nested_items" field .As I know limit and offset not supported in sub queries. Any way to do this except of divide into two queries? Maybe some udf or else?
You can use the function ARRAY_SLICE assuming the array is ordered.
Example data:
{
"name": "John",
"details": [
{
"id": 1
},
{
"id": 2
},
{
"id": 3
}
]
}
Example queries
-- First 2 items from nested array
SELECT c.name, ARRAY_SLICE(c.details, 0, 2) as details
FROM c
-- Last 2 items from nested array
SELECT c.name, ARRAY_SLICE(c.details, ARRAY_LENGTH(c.details) - 2, 2) as details
FROM c
I have a cell of a table-column that is a dynamic. This was ingested from .Net as a Dictionary, but in Kusto it looks like an array of objects, that has a property key and value:
[
{"key":"ProjectId","value":"1234"},
{"key":"ProjectName","value":"Albatros"},
{"key":"User","value":"Bond"}
]
I want to convert the contents of the cell in my Kusto query to the following dynamic:
{
"ProjectId": "1234",
"ProjectName": "Albatros",
"User": "Bond"
}
I cant figure out how to write the expression, that converts it form the array into the new dynamic format.
Can anyone point me in the right direction?
you can use a combination of mv-apply and make_bag():
print d = dynamic([
{"key": "value"},
{"ProjectId": "1234"},
{"ProjectName": "Albatros"},
{"User": "Bond"}
])
| mv-apply d on (
summarize result = make_bag(d)
)
result
{ "key": "value", "ProjectId": "1234", "ProjectName": "Albatros", "User": "Bond"}
UPDATE based on your change to the original question:
print d = dynamic([
{"key":"ProjectId","value":"1234"},
{"key":"ProjectName","value":"Albatros"},
{"key":"User","value":"Bond"}
])
| mv-apply d on (
summarize result = make_bag(pack(tostring(d.key), d.value))
)
result
{ "ProjectId": "1234", "ProjectName": "Albatros", "User": "Bond"}
I have a JSON file that looks like this:
{
"InstanceId": "i-9KwoRGF6jbhYdZi823aE4qN",
"Tags": [
{
"Key": "blah",
"Value": "server-blah"
},
{
"Key": "environment",
"Value": "ops"
},
{
"Key": "server_role",
"Value": "appserver"
},
{
"Key": "Name",
"Value": "some_name"
},
{
"Key": "product",
"Value": "some_server"
}
]
}
{
...more objects like the above...
}
I need to display the InstanceId where "Key" == "environment" and "Value" == "ops".
I have jq-1.6.
If I say:
cat source.json | jq '
{ InstanceId, Tags } |
(.Tags[] | select( .Key == "environment" ))
'
I get some of what I want, but I cannot figure out how to include InstanceId in the output nor how to incorporate the "and" part of the select.
Here is a simple but efficient approach using any:
select( any(.Tags[]; .Key=="environment" and .Value == "ops") )
| .InstanceId
An alternative approach that avoids .Tags[]:
{"Key": "environment", "Value": "ops"} as $object
| select( .Tags | index($object) )
| .InstanceId
I'm not sure if this is the exact output you're looking for (comment if it isn't), but this will output the InstanceIds of JSON objects that contain a Tag with Key environment and Value ops.
jq 'select( .Tags[] | (.Key == "environment" and .Value == "ops")) | .InstanceId' < source.json
We are trying to do DynamoDB migration from prod account to stage account.
In the source account, we are making use of "Export" feature of DDB to put the compressed .json.gz files into destination S3 bucket.
We have written a glue script which will read the exported .json.gz files and writes it to DDB table.
We are making the code generic, so we should be able to migrate any DDB table from prod to stage account.
As part of that process, while testing we are facing issues when we are trying to write a NUMBER SET data to target DDB table.
Following is the sample snippet which is raising ValidationException when trying to insert into DDB
from decimal import Decimal
def number_set(datavalue):
# datavalue will be ['0', '1']
set_of_values = set()
for value in datavalue:
set_of_values.add(Decimal(value))
return set_of_values
When running the code, we are getting following ValidationException
An error occurred while calling o82.pyWriteDynamicFrame. Supplied AttributeValue is empty, must contain exactly one of the supported datatypes (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: UKEU70T0BLIKN0K2OL4RU56TGVVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
However, if instead of Decimal(value) if we use int(value) then no ValidationException is being thrown and the job succeeds.
I feel that write_dynamic_frame_from_options will try to infer schema based on the values the element contains, if the element has "int" values then the datatype would be "NS", but if the element contains all "Decimal type" values, then it is not able to infer the datatype.
The glue job we have written is
dyf = glue_context.create_dynamic_frame_from_options(
connection_type="s3",
connection_options={
"paths": [file_path]
},
format="json",
transformation_ctx = "dyf",
recurse = True,
)
def number_set(datavalue):
list_of_values = []
for value in datavalue:
list_of_values.append(Decimal(value))
print("list of values ")
print(list_of_values)
return set(list_of_values)
def parse_list(datavalue):
list_of_values = []
for object in datavalue:
list_of_values.append(generic_conversion(object))
return list_of_values
def generic_conversion(value_dict):
for datatype,datavalue in value_dict.items():
if datatype == 'N':
value = Decimal(datavalue)
elif datatype == 'S':
value = datavalue
elif datatype == 'NS':
value = number_set(datavalue)
elif datatype == 'BOOL':
value = datavalue
elif datatype == 'M':
value = construct_map(datavalue)
elif datatype == 'B':
value = datavalue.encode('ascii')
elif datatype == 'L':
value = parse_list(datavalue)
return value
def construct_map(row_dict):
ddb_row = {}
for key,value_dict in row_dict.items():
# value is a dict with key as N or S
# if N then use Decimal type
ddb_row[key] = generic_conversion(value_dict)
return ddb_row
def map_function(rec):
row_dict = rec["Item"]
return construct_map(row_dict)
mapped_dyF = Map.apply(frame = dyf, f = map_function, transformation_ctx = "mapped_dyF")
datasink2 = glue_context.write_dynamic_frame_from_options(
frame=mapped_dyF,
connection_type="dynamodb",
connection_options={
"dynamodb.region": "us-east-1",
"dynamodb.output.tableName": destination_table,
"dynamodb.throughput.write.percent": "0.5"
},
transformation_ctx = "datasink2"
)
can anyone help us in how can we unblock from this situation?
Record that we are trying to insert
{
"region": {
"S": "to_delete"
},
"date": {
"N": "20210916"
},
"number_set": {
"NS": [
"0",
"1"
]
},
"test": {
"BOOL": false
},
"map": {
"M": {
"test": {
"S": "value"
},
"test2": {
"S": "value"
},
"nestedmap": {
"M": {
"key": {
"S": "value"
},
"nestedmap1": {
"M": {
"key1": {
"N": "0"
}
}
}
}
}
}
},
"binary": {
"B": "QUFBY2Q="
},
"list": {
"L": [
{
"S": "abc"
},
{
"S": "def"
},
{
"N": "123"
},
{
"M": {
"key2": {
"S": "value2"
},
"nestedmaplist": {
"M": {
"key3": {
"S": "value3"
}
}
}
}
}
]
}
}
I have a record as
firstMap = [ name1:[ value1:10, value2:'name1', value3:150, value4:20 ],
name2:[ value1:10, value2:'name2', value3:150, value4:20 ] ]
I have a list where the values are name1, name2, etc.
I want to pull the list depending on the name1 as
[ name1:[ value1:10, value2:'name1', value3:150, value4:20 ]
firstMap.subMap(["name1"]), did work for me, but I have a list and by looping the list I need to pull the values
namesList.each{record ->
newMap = firstmap.subMap(record)
}
I have tried subMap([offer]), subMap(["offer"]), subMap(["offer?.stringValue()"]), subMap(['offer']), etc. But none of them work for me.
You don't need submap at all, that's only really useful when you want to grab a few keys at once or if you need the original key in the result
Try:
firstMap = [ name1:[ value1:10, value2:'name1', value3:150, value4:20 ],
name2:[ value1:10, value2:'name2', value3:150, value4:20 ] ]
def namesList = [ 'name1', 'name2' ]
namesList.each { name ->
println firstMap[ name ]
}
Or if you need a Map result with the original query key:
namesList.each { name ->
println firstMap.subMap( [ name ] )
}
Or indeed:
namesList.each { name ->
println( [ (name):firstMap[ name ] ] )
}
Would give you the same (ie: create a new map with the key name and the value of my first example)