I'm struggling evaluating a command inside a jq pipeline. Example will make it easier. Let's imagine I've got this simple json
{
"model": [{
"id": "an-id",
"path": [
"mypath1.txt"
],
"model": "foo"
},
{
"id": "an-id2",
"path": [
"mypath1.txt"
],
"model": "foo2"
}
]
}
And I want to convert into this
{
"model": [{
"id": "an-id",
"path": [
"mypath1.txt"
],
"model": "foo",
"alternative_model": "I am a computed value out of <foo>"
},
{
"id": "an-id2",
"path": [
"mypath1.txt"
],
"model": "foo2",
"alternative_model": "I am a computed value out of <foo2>"
}
]
}
I want to do something like this that allows me to delegate computing alternative model to a different bash script.
myNewJson=$(cat mappings.json | jq '[.model[]| {
id: .id,
path: .path
model: .model
alternative_model: //TODO}' ---> here I'd like to do something like "eval ./myscript $model"
])
Thanks!
Let's assume myscript contains following line :
echo "I am a computed value out of <$1>"
As jq does not allow to evaluate shell commands, you need something like :
#!/usr/bin/env bash
input=mappings.json
alternative-models(){
local result='{}' sres
for id in $(jq -r '.model[].id' $input); do
model="$(jq -r --arg id "$id" '.model[]|select(.id==$id).model' $input)"
amodel="$(./myscript $model)"
result="$(jq --arg id $id --arg amodel "$amodel" '. + { $id: $amodel }' <<< "$result")"
done
echo "$result"
}
jq --argjson amodels "$(alternative-models)" '.model|map({
id, path, model, "alternative_model": $amodels[.id] }
)' $input
Your mappings.json is not valid json because of a comma on the model line.
have a very large JSON data like below
{
"10.10.10.1": {
"asset_id": 1,
"referencekey": "ASSET-00001",
"hostname": "testDev01",
"fqdn": "ip-10-10.10.1.ap-northeast-2.compute.internal",
"network_zone": [
"DEV",
"Dev"
],
"service": {
"name": "TEST_SVC",
"account": "AWS_TEST",
"billing": "Testpay"
},
"aws": {
"tags": {
"Name": "testDev01",
"Service": "TEST_SVC",
"Usecase": "Dev",
"billing": "Testpay",
"OsVersion": "20.04"
},
"instance_type": "t3.micro",
"ami_imageid": "ami-e000001",
"state": "running"
}
},
"10.10.10.2": {
"asset_id": 3,
"referencekey": "ASSET-47728",
"hostname": "Infra_Live01",
"fqdn": "ip-10-10-10-2.ap-northeast-2.compute.internal",
"network_zone": [
"PROD",
"Live"
],
"service": {
"name": "Infra",
"account": "AWS_TEST",
"billing": "infra"
},
"aws": {
"tags": {
"Name": "Infra_Live01",
"Service": "Infra",
"Usecase": "Live",
"billing": "infra",
"OsVersion": "16.04"
},
"instance_type": "r5.large",
"ami_imageid": "ami-e592398b",
"state": "running"
}
}
}
Can I use JQ to make the conversion like below?
Or is there an easier way to solve it?
Thank you
Expected result
_key,asset_id,referencekey,hostname,fqdn,network_zone/0,network_zone/1,service/name,service/account,service/billing,aws/tags/Name,aws/tags/Service,aws/tags/Usecase,aws/tags/billing,aws/tags/OsVersion,aws/instance_type,aws/ami_imageid,aws/state
10.10.10.1,1,ASSET-00001,testDev01,ip-10-10.10.1.ap-northeast-2.compute.internal,DEV,Dev,TEST_SVC,AWS_TEST,Testpay,testDev01,TEST_SVC,Dev,Testpay,20.04,t3.micro,ami-e000001,running
10.10.10.2,3,ASSET-47728,Infra_Live01,ip-10-10-10-2.ap-northeast-2.compute.internal,PROD,Live,Infra,AWS_TEST,infra,Infra_Live01,Infra,Live,infra,16.04,r5.large,ami-e592398b,running
jq let's you do the conversion to CSV easily. The following code produces the desired output:
jq -r 'to_entries
| map([.key,
.value.asset_id, .value.referencekey, .value.hostname, .value.fqdn,
.value.network_zone[0], .value.network_zone[1],
.value.service.name, .value.service.account, .value.service.billing,
.value.aws.tags.Name, .value.aws.tags.Service, .value.aws.tags.Usecase, .value.aws.tags.billing, .value.aws.tags.OsVersion,
.value.aws.instance_type, .value.aws.ami_imageid, .value.aws.state])
| ["_key","asset_id","referencekey","hostname","fqdn","network_zone/0","network_zone/1","service/name","service/account","service/billing","aws/tags/Name","aws/tags/Service","aws/tags/Usecase","aws/tags/billing","aws/tags/OsVersion","aws/instance_type","aws/ami_imageid","aws/state"]
, .[]
| #csv' "$INPUT"
Remarks
If some nodes in the input JSON are missing, the code does not break but fills in empty values in the CSV file.
If more than two network zones are given, only the first two are covered in the CSV file
I have a json file named as param.json that looks as below:
[
{
"Value": "anshuman.ceg+Dev#gmail.com",
"Key": "AccountEmail"
},
{
"Value": "DevABC",
"Key": "AccountName"
},
{
"Value": "Security (ou-nzx5-8ajd1561)",
"Key": "ManagedOrganizationalUnit"
},
{
"Value": "anshuman.ceg+Dev#gmail.com",
"Key": "SSOUserEmail"
},
{
"Value": "John",
"Key": "SSOUserFirstName"
},
{
"Value": "Smith",
"Key": "SSOUserLastName"
}
]
I want to get only the Value for DevABC so that I can use while reading the -r line. I need only DevABC
I am using jq as follows which doesn't seem to work
jq -r .[1].Value param.json
Assuming all your Key values are distinct, you can first convert the array into an object and then access the "AccountName" property directly:
jq -r 'from_entries | .AccountName' param.json
from_entries will generate the following object, which allows you to easily access the value for a given key:
{
"AccountEmail": "anshuman.ceg+Dev#gmail.com",
"AccountName": "DevABC",
"ManagedOrganizationalUnit": "Security (ou-nzx5-8ajd1561)",
"SSOUserEmail": "anshuman.ceg+Dev#gmail.com",
"SSOUserFirstName": "John",
"SSOUserLastName": "Smith"
}
If the object keys in the input happen not to be "Key" and "Value" and you can't use from_entries, select would be a good approach:
jq --arg k 'AccountName' -r '.[] | select(.Key == $k).Value'
Here my document:
[
{
"id": "9f0e27fe-3b8f-4857-8e1d-e57e7a3f4c31",
"identifier": [
{
"system": {
"value": "urn:oid:1.3.6.1.4.1.19126.3"
},
"value": {
"value": "Y3454867M"
}
},
{
"system": {
"value": "urn:oid:2.16.724.4.9.10.2"
},
"value": {
"value": "108505134"
}
}
]
}
]
I need to pick only .identifier[where .system.value == "urn:oid:1.3.6.1.4.1.19126.3"] and project .identifier.value.value.
Desired output:
[
{
"id": "9f0e27fe-3b8f-4857-8e1d-e57e7a3f4c31",
"identifier": "Y3454867M"
}
]
I've been playing with map and select but I don't quite figure out what's the right way to get it.
Any ideas?
This approach uses first to get the first result, in case there is more than one array item matching the criteria.
jq --arg v "urn:oid:1.3.6.1.4.1.19126.3" '
map(.identifier |= first(.[] | select(.system.value == $v).value.value))
'
[
{
"id": "9f0e27fe-3b8f-4857-8e1d-e57e7a3f4c31",
"identifier": "Y3454867M"
}
]
Demo
Right on the money with the good ol' select tool, since you need data from an arbitrary index. I fumbled a bit before I unwrapped the inner array that gets piped to my select.
jq -r '.[] | [{id: .id, identifier: .identifier | .[] | select(.system.value | contains("urn:oid:1.3.6.1.4.1.19126.3")) | .value.value }]'
Still new to jq myself, so any feedback is welcome.
For an input below:
[{
"commit": {
"author": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"committer": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
"url":"https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f"
},
{
...
}
}]
How can JQ generate a delimited string from different objects as shown below?
"Stephen Dolan", "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f", "2013-06-22T16:30:59Z"
Collect the fields you want in an array and use #csv to convert to a CSV row. Make sure you get the raw output.
jq -r '.[] | [ .commit.author.name, .commit.url, .commit.author.date ] | #csv' input.json