Ansible flatten/merge dict but not fully - dictionary

I have the following dict
dict3: [[[{"x": "1","y": "2"},{"z": 3,"h": 4}],{"w": 5}],[[{"x": "333","y": "444"},{"z": "555","h": "777"}],{"w": "999"}]]
I need to make it like this:
dict": [
{
"x": "1",
"y": "2",
"z": "3",
"h": "4"
"w": 5
}
{
"x": "333",
"y": "444"
"z": 555,
"h": 777
"w": 999
}
]
The following command flattens it fully, which I don't want and also does not merge x,y,z,h and w (for each group - there can be many grups).
- name: "'unnest' all elements into single list"
ansible.builtin.debug:
msg: "all in one list {{lookup('community.general.flattened', dict)}}"
This is the result with full flattening (and also not merging):
"_raw_params": [
{
"x": "1",
"y": "2"
},
{
"h": 4,
"z": 3
},
{
"w": 5
},
{
"x": "333",
"y": "444"
},
{
"h": "777",
"z": "555"
},
{
"w": "999"
}
]
BTW here is the code
- hosts: localhost
gather_facts: false
vars:
dict: [[[{"x": "1","y": "2"},{"z": 3,"h": 4}],{"w": 5}],[[{"x": "333","y": "444"},{"z": "555","h": "777"}],{"w": "999"}]]
tasks:
- name: UNNEST
set_fact:
"{{lookup('community.general.flattened',dict)}}"
Is there a way to do this?
Thanks!

What you want to do is:
flatten individually each element of your top list. You then get a list of lists of dicts
combine individually each element (i.e. the dict lists) from the above result to merge every dicts into a single one.
All of the above is done by applying the corresponding filters to the elements of the top list using the map filter
The following example playbook:
---
- hosts: localhost
gather_facts: false
vars:
dict3: [[[{"x": "1","y": "2"},{"z": 3,"h": 4}],{"w": 5}],[[{"x": "333","y": "444"},{"z": "555","h": "777"}],{"w": "999"}]]
tasks:
- debug:
msg: "{{ dict3 | map('flatten') | map('combine') }}"
Gives:
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [debug] ***************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
{
"h": 4,
"w": 5,
"x": "1",
"y": "2",
"z": 3
},
{
"h": "777",
"w": "999",
"x": "333",
"y": "444",
"z": "555"
}
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Related

Finding the latest data of the object in the bundled data in Mongo

[
{
"_id": "",
"at": IsoDate(2022-11-19 10:00:00),
"areaId": 3,
"data": [
{
"name": "a",
"sec": 34,
"x": 10.3,
"y": 23.3
},
{
"name": "a",
"sec": 36,
"x": 10.3,
"y": 23.3
},
{
"name": "b",
"sec": 37,
"x": 10.3,
"y": 23.3
}
]
},
{
"_id": "",
"at": IsoDate(2022-11-19 10:00:00),
"areaId": 3,
"data": [
{
"name": "a",
"sec": 10,
"x": 10.3,
"y": 23.3
},
{
"name": "b",
"sec": 12,
"x": 10.3,
"y": 23.3
}
]
}
]
I have a table that indicates in what seconds people are in which area in packets of minutes. My goal here is to find the date the person with the specified name was last in the field with the most performance.
Can you help me for this?
Example output: Last date 'a' was found in polygon=3
2022-11-19 10:01:10 (with sec)
One option is:
Use the first 3 steps to find the matching document
The 4th step is to add the seconds to to the at date
db.collection.aggregate([
{$match: {data: {$elemMatch: {name: inputName}}}},
{$sort: {at: -1}},
{$limit: 1},
{$project: {
res: {
$dateAdd: {
startDate: "$at",
unit: "second",
amount: {$reduce: {
input: "$data",
initialValue: 0,
in: {$cond: [
{$eq: ["$$this.name", inputName]},
{$max: ["$$this.sec", "$$value"]},
"$$value"
]}
}}
}
}
}}
])
See how it works on the playground example

In jq, can you get objects back in a list, so you can index them?

So, say I have this JSON...
[
{
"a": "1",
"blah": "true"
},
{
"b": "2",
"blah": "false"
},
{
"c": "3",
"blah": "true"
}
]
...and then use jq to select certain entries...
jq '.[] | select(.blah=="true)'
I get this...
{
"a": "1",
"blah": "true"
}
{
"c": "3",
"blah": "true"
}
But I want it to look like...
[
{
"a": "1",
"blah": "true"
}
{
"c": "3",
"blah": "true"
}
]
...this, so that I can use indexing to get certain of these entries. How do I do that?
Simply indicate you want the result in a list by wrapping the expression with [];
> cat test.json | jq '[.[] | select(.blah=="true")]'
[
{
"a": "1",
"blah": "true"
},
{
"c": "3",
"blah": "true"
}
]
Using map to iterate over the array
jq 'map(select(.blah == "true"))'

Flatten JSON data with jq

I have output brakeman scan results to a json file and want to flatten the structure.
Brakeman json output sample looks like this
{
"scan_info": {
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"end_time": "2021-11-01 14:45:13 +1100",
"brakeman_version": "5.1.2"
},
"warnings": [
{
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"file": "app/controllers/my/controller.rb",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to....",
"render_path": null,
"location": {
"type": "method",
"class": "My::Controller",
"method": "request_token"
},
"user_input": "my.base_url",
"confidence": "High"
},
{.....}
],
"ignored_warnings": [],
"errors": [],
"obsolete": ["xxx","yyyy"]
}
I want to flatten the output into an array of objects similar to this. I want most data from the warnings array except .location and .render_path.
[
{
"brakeman_version": "5.1.2",
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f0d2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to...",
"confidence": "High"
},
{
"brakeman_version": "5.1.2",
"warning_type": "SQL Injection",
"check_name": "SQL",
"message": "Possible SQL injection",
"warning_code": 35,
....
"confidence": "High"
}
]
Something like this gives a nested object for each item in the warnings array but I'd like to move them up a level
jq '{brakeman_version: .scan_info.brakeman_version, start_time: .scan_info.start_time, warnings: .warnings[]}' ./brakeman.json
Result
{
"brakeman_version": "5.1.2",
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": {
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "xxxx",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"confidence": "High"
},
{...}
}
Something like this flattens the .scan_info part similarly but still leaves me with an array for warnings which is what was there in the first place.
jq '{
rails_version: .scan_info.rails_version,
security_warnings: .scan_info.security_warnings,
start_time: .scan_info.start_time,
brakeman_version: .scan_info.brakeman_version,
warnings: [.warnings[] | {warning_type: .warning_type,
warning_code: .warning_code,
fingerprint: .fingerprint,
check_name: .check_name,
confidence: .confidence} ],
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
Results
{
"rails_version": "6.1.4.1",
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": [
{
"warning_type": "SQL Injection",
"warning_code": 0,
"fingerprint": "xx",
"check_name": "SQL",
"message": "Possible SQL injection",
"line": 178,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
"confidence": "Medium"
},
{
"warning_type": "Format Validation",
"warning_code": 30,
"fingerprint": "xxx",
"check_name": "ValidationRegex",
"line": 92,
"link": "https://brakemanscanner.org/docs/warning_types/format_validation/",
"code": null,
"user_input": null,
"confidence": "High"
},
]
}
Using jq how do I correctly flatten the structure to output as desired?
I've tried some suggestions found on SO like
jq '{
rails_version: .scan_info.rails_version,
brakeman_version: .scan_info.brakeman_version,
.warnings[] as $in | $in | del(.location, .render_path) as $in2 | $in2,
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
but have been going around in circles at this stage.
With assumptions based on your description provided, I guess you are looking to do this
.scan_info + ( .warnings[] | del(.location, .render_path) )
If you want the entire results packed into an array, enclose the above filter within square brackets([ ])
jqplay demo

Watson doesn't get zeros

Let's say I have a conversation services configured in IBM Watson ready to recognize a number given in words and in pieces. For example, if I have the number 1320, it can be sent as thirteen twenty or thirteen two zero, etc.
In the first case I'll get something like this from a conversation service:
{
// ...
"entities": [
{
"entity": "sys-number",
"location": [
0,
5
],
"value": "13",
"confidence": 1,
"metadata": {
"numeric_value": 13
}
},
{
"entity": "sys-number",
"location": [
6,
12
],
"value": "20",
"confidence": 1,
"metadata": {
"numeric_value": 20
}
}
]
// ...
}
In the second case (thirteen two zero):
{
// ...
"entities": [
{
"entity": "sys-number",
"location": [
0,
5
],
"value": "13",
"confidence": 1,
"metadata": {
"numeric_value": 13
}
},
{
"entity": "sys-number",
"location": [
6,
14
],
"value": "2",
"confidence": 1,
"metadata": {
"numeric_value": 2
}
}
]
// ...
}
The big question here is: Where is my zero?
I know this question has been asked more than once, but none of the answers I found solved my current issues.
I've seen examples where a regular expression could be used, but that's for actual numbers, here I have words and Watson is the one who actually guesses the number.
Is there a way to obtain a third entry in my entities for that zero? or another work arround? or configuration I may be lacking?
I just tried this in Watson Assistant and it now gets the zero. With #sys-numbers enabled my utterance is thirteen two zero and I get these entities back:
entities: [
0: {
entity: "sys-number",
location: [0, 8],
value: "13",
confidence: 1,
metadata: {numeric_value: 13}
}
1: {
entity: "sys-number",
location: [9, 12],
value: "2",
confidence: 1,
metadata: {numeric_value: 2}
}
2: {
entity: "sys-number",
location: [13, 17],
value: "0",
confidence: 1,
metadata: {numeric_value: 0}
}
]
It might be the entities matching has been improved.

opentsdb api/query downsample result is incorrect

I used API/query downsample to query the data, but the results I get are different. I cannot explain why.
My first query:
{
"start": 1498838400,
"end": 1501516800,
"timezone": "Asia/Shanghai",
"useCalendar": true,
"delete": false,
"queries": [
{
"aggregator": "sum",
"metric": "meter.energy.active.forward.z",
"downsample": "24h-first",
"rate": false,
"filters": [
{
"type": "literal_or",
"tagk": "deviceId",
"filter": "127",
"groupBy": true
}
]
}
]
}
The result:
[{
"metric": "meter.energy.active.forward.z",
"tags": {
"deviceTypeId": "1",
"deviceNo": "340340001750",
"deviceId": "127",
"gatewayId": "72"
},
"aggregateTags": [],
"dps": {
"1498924800": 0.029999999329447746,
"1499097600": 349577.59375,
"1499184000": 410578.90625,
"1499270400": 515834.09375,
"1499356800": 616553.6875,
"1499443200": 722792.5,
"1499529600": 800983.75...}}]
For the second request, I only change 24h-first to 24h-first-nan, and the second request result is:
[{
"metric": "meter.energy.active.forward.z",
"tags": {
"deviceTypeId": "1",
"deviceNo": "340340001750",
"deviceId": "127",
"gatewayId": "72"
},
"aggregateTags": [],
"dps": {}
}]
I want the result is:
[{
"metric": "meter.energy.active.forward.z",
"tags": {
"deviceTypeId": "1",
"deviceNo": "340340001750",
"deviceId": "127",
"gatewayId": "72"
},
"aggregateTags": [],
"dps": {
"1498924800": 0.029999999329447746,
"1499011200": NaN,
"1499097600": 349577.59375,
"1499184000": 410578.90625,
"1499270400": 515834.09375,
"1499356800": 616553.6875,
"1499443200": 722792.5,
"1499529600": 800983.75...}}]
I also delete the "useCalendar", but the time is not what I want.
Do you see my issue? Can you help? Thank you!

Resources