I have following json input file. I need to get the 2 and 3rd keys like below.
Need out put as:
test, 2021.1.2
dev, 2021.2.2
test, 2021.1.2
dev, 2021.2.1
test, 2021.3.1
Input json:
[
{
"builds": {
"test": {
"2021.1.2": {
"stages": [
"functional",
"integration"
]
}
},
"dev": {
"2021.2.2": {
"stages": [
"junit",
"nls"
]
}
}
},
"dockertag": "0.1.1"
},
{
"builds": {
"test": {
"2021.1.2": {
"stages": [
"functional"
]
}
},
"dev": {
"2021.2.1": {
"stages": [
"junit",
"nls"
]
}
}
},
"dockertag": "0.1.2"
},
{
"builds": {
"test": {
"2021.3.1": {
"stages": [
"functional",
"integration"
]
}
}
},
"dockertag": "0.1.3"
}
]
I tried following code but it is incomplete. I can get the 2nd key but not sure how to further extract 3rd key:
jq -r '.[].builds | keys[] as $k | "\($k), \(.[$k] | .)"' test.json
Thanks for the help.
.[].builds | to_entries[] | "\(.key), \(.value | keys[])"
Will generate
"test, 2021.1.2"
"dev, 2021.2.2"
"test, 2021.1.2"
"dev, 2021.2.1"
"test, 2021.3.1"
Use jq --raw-output to remove the "'s around each line
Online demo
Two approaches using the path:
jq -r '.[].builds | paths | select(length == 2) | join(", ")'
test, 2021.1.2
dev, 2021.2.2
test, 2021.1.2
dev, 2021.2.1
test, 2021.3.1
Demo
jq -r 'path(.[].builds[][])[2:] | join(", ")'
test, 2021.1.2
dev, 2021.2.2
test, 2021.1.2
dev, 2021.2.1
test, 2021.3.1
Demo
Here is a ruby if your jq is broken that day:
ruby -r json -e 'JSON.parse($<.read).
each{|h| h.select{|k,v| k=="builds"}.
each{|k,v| ["test","dev"].
each{|t| puts "#{t}, #{v[t].keys[0]}" if v.key?(t)}}} ' file
test, 2021.1.2
dev, 2021.2.2
test, 2021.1.2
dev, 2021.2.1
test, 2021.3.1
Related
So, given two files with JSON data from the same source type. The JSON objects look something like:
file1:
[
{
"data": {
"id": "2",
"nodes": [
{
"stuff": "foo"
}
]
}
},
{
"data": {
"id": "6",
"nodes": [
{
"stuff": "bar"
}
]
}
},
{
"data": {
"id": "61",
"nodes": [
{
"stuff": "baz"
}
]
}
},
{
"data": {
"id": "63",
"nodes": [
{
"stuff": "qux"
}
]
}
}
]
file2:
[
{
"data": {
"id": "61",
"nodes": [
{
"stuff": "baz"
}
]
}
},
{
"data": {
"id": "63",
"nodes": [
{
"stuff": "qux"
}
]
}
}
]
I'm trying to remove objects in the array in the first file with the matching IDs in the second file so that the resultant output would be:
[
{
"data": {
"id": "2",
"nodes": [
{
"stuff": "foo"
}
]
}
},
{
"data": {
"id": "6",
"nodes": [
{
"stuff": "bar"
}
]
}
}
]
I've tried a bunch of ways to accomplish this, but I haven't found a proper solution yet.
A couple of attempts have been various permutations of the following with accompanying errors:
jq -n --argfile src /var/tmp/w-src.json --argfile dst /var/tmp/w-dst.json '
$dst
| [.data[].id] as $ids
| $src
| .data | map(select(.id | in($ids[])))
jq: error: select/0 is not defined at <top-level>, line 5:
| .data | map($ids | map(select .id == .))
jq: 1 compile error
jq -n --argfile src /var/tmp/w-src.json --argfile dst /var/tmp/w-dst.json '
$dst
| [.data[].id] as $ids
| $src
| .data[] | select(.id | in($ids[]))
'
jq: error (at <unknown>): Cannot check whether string has a string key
Ideally it would be super cool to do some kind of operation like:
$src.data[] - $dst.data[]
(kinda Ruby-ish like would be cool) and I admit, I haven't tried this but I will for kicks and giggles.
I'm trying not to have to use a function and I want to accomplish this using jq. I'm probably not too far off, but I'm at a loss. Any thoughts?
You could compile a list of IDs from the second file using input, check against it using IN, and either use del to delete the matching, or map to keep those that do not match:
jq '
(input | map(.data.id)) as $del | del(.[] | select(IN(.data.id; $del[])))
' file1.json file2.json
or
jq '
(input | map(.data.id)) as $del | map(select(IN(.data.id; $del[]) | not))
' file1.json file2.json
If you can assert that objects with identical IDs also are identical in their other parts, and you don't have many items (because it's costly), you can even just subtract the second file from the first:
jq '. - input' file1.json file2.json
I'm struggling evaluating a command inside a jq pipeline. Example will make it easier. Let's imagine I've got this simple json
{
"model": [{
"id": "an-id",
"path": [
"mypath1.txt"
],
"model": "foo"
},
{
"id": "an-id2",
"path": [
"mypath1.txt"
],
"model": "foo2"
}
]
}
And I want to convert into this
{
"model": [{
"id": "an-id",
"path": [
"mypath1.txt"
],
"model": "foo",
"alternative_model": "I am a computed value out of <foo>"
},
{
"id": "an-id2",
"path": [
"mypath1.txt"
],
"model": "foo2",
"alternative_model": "I am a computed value out of <foo2>"
}
]
}
I want to do something like this that allows me to delegate computing alternative model to a different bash script.
myNewJson=$(cat mappings.json | jq '[.model[]| {
id: .id,
path: .path
model: .model
alternative_model: //TODO}' ---> here I'd like to do something like "eval ./myscript $model"
])
Thanks!
Let's assume myscript contains following line :
echo "I am a computed value out of <$1>"
As jq does not allow to evaluate shell commands, you need something like :
#!/usr/bin/env bash
input=mappings.json
alternative-models(){
local result='{}' sres
for id in $(jq -r '.model[].id' $input); do
model="$(jq -r --arg id "$id" '.model[]|select(.id==$id).model' $input)"
amodel="$(./myscript $model)"
result="$(jq --arg id $id --arg amodel "$amodel" '. + { $id: $amodel }' <<< "$result")"
done
echo "$result"
}
jq --argjson amodels "$(alternative-models)" '.model|map({
id, path, model, "alternative_model": $amodels[.id] }
)' $input
Your mappings.json is not valid json because of a comma on the model line.
I am trying to retrieve the values containg contoso. So my output in this case should just readdata_contoso and writedata_contoso but I am having issues. I am trying jq -e -r ".roles[] | select(.name | contains("cont"))"
{
"roles": [
{
"name": "readdata_contoso",
"permissions": {
"": [
"ReadData"
]
}
},
{
"name": "writedata_contoso",
"permissions": {
"": [
"WriteData"
]
}
}
]
}
You are almost there. First, extract only the name of each role, then select those names which contain "contoso":
.roles[].name | select(contains("contoso"))
roles[].name generates a list of all names
select(contains("contoso")) only selects those names which contain "contoso"
I am trying to extract kubeconfig data with jq.
kubectl config view --raw -o json | jq ...
There is a json produced of this kind:
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [
{
"name": "some-name",
"cluster": {
"server": "https://some-url",
"certificate-authority-data": "some-cert"
}
},
{
"name": "another-name",
"cluster": {
"server": "https://another-url",
"certificate-authority-data": "another-cert"
}
}
],
"users": [
{
"name": "some-name",
"user": {
"username": "some-user",
"password": "some-password"
}
},
{
"name": "another-name",
"user": {
"username": "another-user",
"password": "another-password"
}
}
],
"contexts": [],
"current-context": "some-context"
}
Question #1:
For a given name ,"some-name", I'd like to extract json:
{
url: "https://some-url",
cert: "some-cert",
username: "some-user",
password: "some-password"
}
Question #2:
"users" sub-section can have other format
"users": [
{
"name": "...",
"user": {
"exec": {
...
}
Where .user.username or .user.password or both can be missing
In this case overall query should return "{}", even though, "clusters" query/branch has result
Question 3, as a follow up to Jeff Mercado answer:
I want to get all clusters, joined (grouped by) name:
Looking at the manual, https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions ,
section "Multiplication, division, modulo: *, /, and %", example:
jq '{"k": {"a": 1, "b": 2}} * {"k": {"a": 0,"c": 3}}' => {"k": {"a": 0, "b": 2, "c": 3}}'
gives presumably right result, assuming "k" is value of "name". So, grouping by "k", merging (*) the results.
I produced following query:
echo "${json}" | jq -r '(.clusters[] | {(.name): {url: .cluster.server, cert: .cluster["certificate-authority-data"]}}) * (.users[] | {(.name): {user: .user.username, password: .user.password}})'
First part returns {"name": {url: cert}}, second part is {"name": {username, password}}
However, result is not merge as in jq Manual, but something else ... product ?
{
"some-name": {
"url": "https://some-url",
"cert": "some-cert",
"user": "some-user",
"password": "some-password"
}
}
{
"another-name": {
"url": "https://another-url",
"cert": "another-cert"
},
"some-name": {
"user": "some-user",
"password": "some-password"
}
}
{
"some-name": {
"url": "https://some-url",
"cert": "some-cert"
},
"another-name": {
"user": "another-user",
"password": "another-password"
}
}
{
"another-name": {
"url": "https://another-url",
"cert": "another-cert",
"user": "another-user",
"password": "another-password"
}
}
Why/what is it ? Kind of following idea of the product ('*') but not of the jq tutorial as I (most likely, incorrectly) understand it
Experimentation:
I have 2 queries now producing partial result.
Let's grab original json (above) in and parse:
read -d '' json << EOF
...
EOF
queries:
echo "${json}" | jq -r '.clusters[] | select(.name=="some-name") | .cluster | {url: .server, cert: .["certificate-authority-data"]}' &&\
echo "${json}" | jq -r '.users[] | select(.name=="some-name") | .user | {user: .username, password: .password}'
Will produce the split output:
{
"url": "https://some-url",
"cert": "some-cert"
}
{
"user": "some-user",
"password": "some-password"
}
Or, with key added for further merge:
echo "${json}" | jq -r '.clusters[] | select(.name=="some-name") | {name: .name, url: .cluster.server, cert: .cluster["certificate-authority-data"]}' &&\
echo "${json}" | jq -r '.users[] | select(.name=="some-name") | {name: .name, user: .user.username, password: .user.password}'
Will produce:
{
"name": "some-name",
"url": "https://some-url",
"cert": "some-cert"
}
{
"name": "some-name",
"user": "some-user",
"password": "some-password"
}
"name" is not needed but can be used as a join operation
So you already know how to get the cluster and user by name separately, first step is to select them both within a single filter:
(.clusters[] | select(.name == $name).cluster), (.users[] | select(.name == $name).user)
This will yield two separate objects, the cluster, then the user. But we want to merge them. There's plenty of ways to do this. You could add them (+) directly or merge them (*) but no real difference there. You'll just want to remap the properties to the names you wanted where needed.
(.clusters[] | select(.name == $name).cluster | {url: .server, cert: ."certificate-authority-data"})
+
(.users[] | select(.name == $name).user | {username, password})
Pass the name in as a parameter to your filter;
$ kubectl config view --raw -o json | jq --arg name some-name '
(.clusters[] | select(.name == $name).cluster | {url: .server, cert: ."certificate-authority-data"})
+
(.users[] | select(.name == $name).user | {username, password})
'
For the second part of your question, if it turns out the mapped user is missing key properties and you want to omit them, just add another select filter to the end to test for those properties and replace with an empty object if nothing is found:
... | select(has("username") and has("password")) // {}
jqplay
I am trying to read the CIDR blocks from the VPCs in AWS on the AWS CLI. I will use this in a script when I'm done. I am using jq to parse the info:
aws ec2 describe-vpcs --region=us-east-1 | jq -r '.Vpcs[].CidrBlock'
10.200.3.0/24
However, jq only returns one of the two CIDR blocks in the VPC. This is the original json:
{
"Vpcs": [
{
"CidrBlock": "10.200.3.0/24",
"DhcpOptionsId": "dopt-d0aa95ab",
"State": "available",
"VpcId": "vpc-00de11103235ec567",
"OwnerId": "046480487130",
"InstanceTenancy": "default",
"Ipv6CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-09f19d81c2e4566b9",
"Ipv6CidrBlock": "2600:1f18:1f7:300::/56",
"Ipv6CidrBlockState": {
"State": "associated"
},
"NetworkBorderGroup": "us-east-1"
}
],
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-0511a5d459f937899",
"CidrBlock": "10.238.3.0/24",
"CidrBlockState": {
"State": "associated"
}
},
{
"AssociationId": "vpc-cidr-assoc-05ad73e8c515a470f",
"CidrBlock": "100.140.0.0/27",
"CidrBlockState": {
"State": "associated"
}
}
],
"IsDefault": false,
"Tags": [
{
"Key": "environment",
"Value": "int01"
},
{
"Key": "Name",
"Value": "company-int01-vpc"
},
{
"Key": "project",
"Value": "company"
}
]
}
]
}
Why does jq only return part of the info I'm after? I need to get all VPC CIDR blocks in the output.
You have two keys CidrBlock and CidrBlockAssociationSet under the Vpcs array.
aws ec2 describe-vpcs --region=us-east-1 |
jq -r '.Vpcs[] | .CidrBlock, .CidrBlockAssociationSet[].CidrBlock'
10.200.3.0/24
10.238.3.0/24
100.140.0.0/27
and this is an invariant solution:
aws ... | jq -r '.. | if type == "object" and has("CidrBlock") then .CidrBlock else empty end'
and, inspired by jq170727's answer, a less expressive form:
aws ... | jq -r '.. | objects | .CidrBlock // empty'
Here is a filter inspired by Dmitry's answer which is slightly shorter: .. | .CidrBlock? | values
Try it online!