could someone please explain below warning?
input file is:
{
"env": "DC",
"hosts" :
[
{
"apt_update_last_success": "1495991703",
"architecture": "amd64",
"hostname": "h1"
},
{
"apt_update_last_success": "1495991703",
"architecture": "amd64",
"hostname": "h2"
},
{
"apt_update_last_success": "1496045706",
"architecture": "amd64",
"hostname": "h3"
},
{
"apt_update_last_success": "1496045705",
"architecture": "amd64",
"hostname": "h4"
},
{
"apt_update_last_success": "1496049305",
"architecture": "amd64",
"hostname": "h5"
},
{
"apt_update_last_success": "1496049307",
"architecture": "amd64",
"hostname": "h6"
}
]
}
the jq command returns what expected but also prints the warning I dont know why:
$ jq -r '.[][] | select(.hostname=="h6")' ddd.json
jq: error: Cannot iterate over string
{
"apt_update_last_success": "1496049307",
"architecture": "amd64",
"hostname": "h6"
}
please tell me how to get rid of this.
thank you.
The issue is in your notation .[][]. Your input is just an object but you are trying to present it as a "container" of "containers" .[][].
The right way is:
jq '.hosts[] | select(.hostname=="h6")' ddd.json
{
"apt_update_last_success": "1496049307",
"architecture": "amd64",
"hostname": "h6"
}
Besides, on jq 1.5 this jq -r '.[][] | select(.hostname=="h6")' ddd.json doesn't return the expected object but only prints
jq: error (at jq1:36): Cannot iterate over string ("DC")
I got this error when piping a log file with a list of individual JSON objects to JQ:
$ cat my_log_file.txt:
{"message": "foo"}
{"message": "bar"}
$ cat my_log_file.txt | jq '.message | unique'
jq: error (at <stdin>:1): Cannot iterate over string ("foo")
jq: error (at <stdin>:2): Cannot iterate over string ("bar")
In this case you need to use jq's command line option -s (slurp) so that you can operate on the file as a single array object.
$ cat my_log_file.txt | jq -s
[
{
"message": "foo"
},
{
"message": "bar"
}
]
Related
I'm struggling evaluating a command inside a jq pipeline. Example will make it easier. Let's imagine I've got this simple json
{
"model": [{
"id": "an-id",
"path": [
"mypath1.txt"
],
"model": "foo"
},
{
"id": "an-id2",
"path": [
"mypath1.txt"
],
"model": "foo2"
}
]
}
And I want to convert into this
{
"model": [{
"id": "an-id",
"path": [
"mypath1.txt"
],
"model": "foo",
"alternative_model": "I am a computed value out of <foo>"
},
{
"id": "an-id2",
"path": [
"mypath1.txt"
],
"model": "foo2",
"alternative_model": "I am a computed value out of <foo2>"
}
]
}
I want to do something like this that allows me to delegate computing alternative model to a different bash script.
myNewJson=$(cat mappings.json | jq '[.model[]| {
id: .id,
path: .path
model: .model
alternative_model: //TODO}' ---> here I'd like to do something like "eval ./myscript $model"
])
Thanks!
Let's assume myscript contains following line :
echo "I am a computed value out of <$1>"
As jq does not allow to evaluate shell commands, you need something like :
#!/usr/bin/env bash
input=mappings.json
alternative-models(){
local result='{}' sres
for id in $(jq -r '.model[].id' $input); do
model="$(jq -r --arg id "$id" '.model[]|select(.id==$id).model' $input)"
amodel="$(./myscript $model)"
result="$(jq --arg id $id --arg amodel "$amodel" '. + { $id: $amodel }' <<< "$result")"
done
echo "$result"
}
jq --argjson amodels "$(alternative-models)" '.model|map({
id, path, model, "alternative_model": $amodels[.id] }
)' $input
Your mappings.json is not valid json because of a comma on the model line.
I am trying to extract kubeconfig data with jq.
kubectl config view --raw -o json | jq ...
There is a json produced of this kind:
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [
{
"name": "some-name",
"cluster": {
"server": "https://some-url",
"certificate-authority-data": "some-cert"
}
},
{
"name": "another-name",
"cluster": {
"server": "https://another-url",
"certificate-authority-data": "another-cert"
}
}
],
"users": [
{
"name": "some-name",
"user": {
"username": "some-user",
"password": "some-password"
}
},
{
"name": "another-name",
"user": {
"username": "another-user",
"password": "another-password"
}
}
],
"contexts": [],
"current-context": "some-context"
}
Question #1:
For a given name ,"some-name", I'd like to extract json:
{
url: "https://some-url",
cert: "some-cert",
username: "some-user",
password: "some-password"
}
Question #2:
"users" sub-section can have other format
"users": [
{
"name": "...",
"user": {
"exec": {
...
}
Where .user.username or .user.password or both can be missing
In this case overall query should return "{}", even though, "clusters" query/branch has result
Question 3, as a follow up to Jeff Mercado answer:
I want to get all clusters, joined (grouped by) name:
Looking at the manual, https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions ,
section "Multiplication, division, modulo: *, /, and %", example:
jq '{"k": {"a": 1, "b": 2}} * {"k": {"a": 0,"c": 3}}' => {"k": {"a": 0, "b": 2, "c": 3}}'
gives presumably right result, assuming "k" is value of "name". So, grouping by "k", merging (*) the results.
I produced following query:
echo "${json}" | jq -r '(.clusters[] | {(.name): {url: .cluster.server, cert: .cluster["certificate-authority-data"]}}) * (.users[] | {(.name): {user: .user.username, password: .user.password}})'
First part returns {"name": {url: cert}}, second part is {"name": {username, password}}
However, result is not merge as in jq Manual, but something else ... product ?
{
"some-name": {
"url": "https://some-url",
"cert": "some-cert",
"user": "some-user",
"password": "some-password"
}
}
{
"another-name": {
"url": "https://another-url",
"cert": "another-cert"
},
"some-name": {
"user": "some-user",
"password": "some-password"
}
}
{
"some-name": {
"url": "https://some-url",
"cert": "some-cert"
},
"another-name": {
"user": "another-user",
"password": "another-password"
}
}
{
"another-name": {
"url": "https://another-url",
"cert": "another-cert",
"user": "another-user",
"password": "another-password"
}
}
Why/what is it ? Kind of following idea of the product ('*') but not of the jq tutorial as I (most likely, incorrectly) understand it
Experimentation:
I have 2 queries now producing partial result.
Let's grab original json (above) in and parse:
read -d '' json << EOF
...
EOF
queries:
echo "${json}" | jq -r '.clusters[] | select(.name=="some-name") | .cluster | {url: .server, cert: .["certificate-authority-data"]}' &&\
echo "${json}" | jq -r '.users[] | select(.name=="some-name") | .user | {user: .username, password: .password}'
Will produce the split output:
{
"url": "https://some-url",
"cert": "some-cert"
}
{
"user": "some-user",
"password": "some-password"
}
Or, with key added for further merge:
echo "${json}" | jq -r '.clusters[] | select(.name=="some-name") | {name: .name, url: .cluster.server, cert: .cluster["certificate-authority-data"]}' &&\
echo "${json}" | jq -r '.users[] | select(.name=="some-name") | {name: .name, user: .user.username, password: .user.password}'
Will produce:
{
"name": "some-name",
"url": "https://some-url",
"cert": "some-cert"
}
{
"name": "some-name",
"user": "some-user",
"password": "some-password"
}
"name" is not needed but can be used as a join operation
So you already know how to get the cluster and user by name separately, first step is to select them both within a single filter:
(.clusters[] | select(.name == $name).cluster), (.users[] | select(.name == $name).user)
This will yield two separate objects, the cluster, then the user. But we want to merge them. There's plenty of ways to do this. You could add them (+) directly or merge them (*) but no real difference there. You'll just want to remap the properties to the names you wanted where needed.
(.clusters[] | select(.name == $name).cluster | {url: .server, cert: ."certificate-authority-data"})
+
(.users[] | select(.name == $name).user | {username, password})
Pass the name in as a parameter to your filter;
$ kubectl config view --raw -o json | jq --arg name some-name '
(.clusters[] | select(.name == $name).cluster | {url: .server, cert: ."certificate-authority-data"})
+
(.users[] | select(.name == $name).user | {username, password})
'
For the second part of your question, if it turns out the mapped user is missing key properties and you want to omit them, just add another select filter to the end to test for those properties and replace with an empty object if nothing is found:
... | select(has("username") and has("password")) // {}
jqplay
Have a JSON output as below. Trying to extract only key names like "key-user" and "oidc-cid" using JQ
[
{
"key-user": {
"contentType": "",
"enabled": true,
"isSecret": true,
"value": null
},
"oidc-cid": {
"contentType": "",
"enabled": true,
"isSecret": true,
"value": null
}
}
]
Tried the following command
jq 'keys' and jq 'keys[]' and jq '[] | keys'
Missing the dot in last command
jq '.[]|keys' data.json
[
"key-user",
"oidc-cid"
]
I am trying to read the CIDR blocks from the VPCs in AWS on the AWS CLI. I will use this in a script when I'm done. I am using jq to parse the info:
aws ec2 describe-vpcs --region=us-east-1 | jq -r '.Vpcs[].CidrBlock'
10.200.3.0/24
However, jq only returns one of the two CIDR blocks in the VPC. This is the original json:
{
"Vpcs": [
{
"CidrBlock": "10.200.3.0/24",
"DhcpOptionsId": "dopt-d0aa95ab",
"State": "available",
"VpcId": "vpc-00de11103235ec567",
"OwnerId": "046480487130",
"InstanceTenancy": "default",
"Ipv6CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-09f19d81c2e4566b9",
"Ipv6CidrBlock": "2600:1f18:1f7:300::/56",
"Ipv6CidrBlockState": {
"State": "associated"
},
"NetworkBorderGroup": "us-east-1"
}
],
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-0511a5d459f937899",
"CidrBlock": "10.238.3.0/24",
"CidrBlockState": {
"State": "associated"
}
},
{
"AssociationId": "vpc-cidr-assoc-05ad73e8c515a470f",
"CidrBlock": "100.140.0.0/27",
"CidrBlockState": {
"State": "associated"
}
}
],
"IsDefault": false,
"Tags": [
{
"Key": "environment",
"Value": "int01"
},
{
"Key": "Name",
"Value": "company-int01-vpc"
},
{
"Key": "project",
"Value": "company"
}
]
}
]
}
Why does jq only return part of the info I'm after? I need to get all VPC CIDR blocks in the output.
You have two keys CidrBlock and CidrBlockAssociationSet under the Vpcs array.
aws ec2 describe-vpcs --region=us-east-1 |
jq -r '.Vpcs[] | .CidrBlock, .CidrBlockAssociationSet[].CidrBlock'
10.200.3.0/24
10.238.3.0/24
100.140.0.0/27
and this is an invariant solution:
aws ... | jq -r '.. | if type == "object" and has("CidrBlock") then .CidrBlock else empty end'
and, inspired by jq170727's answer, a less expressive form:
aws ... | jq -r '.. | objects | .CidrBlock // empty'
Here is a filter inspired by Dmitry's answer which is slightly shorter: .. | .CidrBlock? | values
Try it online!
For an input below:
[{
"commit": {
"author": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"committer": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
"url":"https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f"
},
{
...
}
}]
How can JQ generate a delimited string from different objects as shown below?
"Stephen Dolan", "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f", "2013-06-22T16:30:59Z"
Collect the fields you want in an array and use #csv to convert to a CSV row. Make sure you get the raw output.
jq -r '.[] | [ .commit.author.name, .commit.url, .commit.author.date ] | #csv' input.json