I want to build a json with jq filter, the usecase is very simple,
I have a json file as this,
# cat input.json
{"foo": 42, "bar": "less interesting data"}
and I want to generate a json file with jq as follow,
output:
increase the value of foo,
[
{
"leafCalls": {
"text": 43
}
}
]
and the filter I am using is defined as this,
# cat results.jq
[.[] | . as { foo: $leafText, bar: $leafCode} | {
leafCall: {
text: $leafText
},
} | {
leafCalls: .leafCall |= . + 1
}]
But there is a syntax issue as said below,
# <input.json jq --slurp --from-file results.jq > output.json
jq: error: syntax error, unexpected |=, expecting '}' (Unix shell quoting issues?) at <top-level>, line 6:
leafCalls: .leafCall |= . + 1
jq: 1 compile error
Beside, there is also a problem if want to concat a string to the value of "bar",
For what I want is,
[
{
"leafCalls": {
"text": "less interesting data string"
}
}
]
The filter is
# cat results2.jq
[.[] | . as { foo: $leafText, bar: $leafCode} | {
leafCall: {
text: $leafCode },
} | {
leafCalls: .leafCall + " " + "string"
}]
and the jq is saying,
jq: error: syntax error, unexpected '+', expecting '}'
Can anyone tell what's the problem with the filter definition?
To get the first output, increase the value directly. Parentheses are needed for precedence.
jq '[. as { foo: $leafText } | {
leafCalls: {
text: ($leafText + 1)
},
}
]' input.json
And similarly for the string:
jq '[ . as { foo: $leafText, bar: $leafCode}
| { leafCalls: { text: ($leafCode + " string" ) } } ]
' input.json
Related
I was checking the jq tutorial at https://programminghistorian.org/en/lessons/json-and-jq
It makes some json reshaping, extracting some data from a json file, found at https://programminghistorian.org/assets/jq_twitter.json
At some point it makes a group_by, grouping data with the same user, extracting some user data and adding its corresponding tweet ids with the command
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]}'
so far, so good... the response looks like this (just a part is extracted):
{
"user_id": 18270633,
"user_name": "ahhthatswhy",
"user_followers": 559,
"tweet_ids": [
501064204661850100
]
}
{
"user_id": 27202261,
"user_name": "Dushan41",
"user_followers": 1201,
"tweet_ids": [
619172281751711700,
619172321564098600
]
}
{
"user_id": 2500422674,
"user_name": "pecanEgba74318",
"user_followers": 17,
"tweet_ids": [
619172331592773600
]
}
But then I would like to add a {"multiple_tweets": true} to all the objects that have more than one tweet_ids.
If I plainly pipe, like this, it works fine:
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | (select(.tweet_ids | length > 1) .multiple_tweets = true)'
a part of the result:
{
"user_id": 1653718716,
"user_name": "OAnnie8",
"user_followers": 315,
"tweet_ids": [
501064215160172540
]
}
{
"user_id": 356854246,
"user_name": "DrJLMooreIII",
"user_followers": 4888,
"tweet_ids": [
501064202904404000,
501064231387947000
],
"multiple_tweets": true
}
{
"user_id": 117155917,
"user_name": "rebekahwsm",
"user_followers": 5069,
"tweet_ids": [
501064233186893800
]
}
But if (for whatever reason, in this example is not really needed, in fact I was doing it just to understand the update-assignment) I want to use the |= operator,
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I get the error ' jq: error (at :30259): Invalid path expression with result {"user_id":1330235048,"use... '
Now the thing that I really can't understand. If instead of using the operator |= directly, I pipe through the identity operator first, it works fine.
What is the reason of this behaviour? Why does |.|= behave differently than |= ?
Why does this change anything?
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | . |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I guess I'm still not understanding how the |= operator really works.
Thank you for your help.
JQ manual explains that behavior as follows:
The left-hand side can be any general path expression; see path().
Note that the left-hand side of |= refers to a value in .. Thus $var.foo |= . + 1 won't work as expected ($var.foo is not a valid or useful path expression in .); use $var | .foo |= . + 1 instead.
Since the underlying builtin (_modify) is implemented using setpath, getpath, and delpaths; the LHS of |= must be a valid path expression that can be represented as an array; in other words, path(LHS) must not fail. See below examples.
$ jq -n 'path(1)'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 |= . + 1'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 | path(.)'
[]
$ jq -n '1 | . |= . + 1'
2
I have the the following input json:
{
"TagList": [
{
"Key": "Environment",
"Value": "foo"
},
{
"Key": "ENVIRONMENT",
"Value": "bar"
}
]
}
I want to get the values of tags with the key ENVIRONMENT using jq:
jq -r '.TagList[] | select(.Key=="ENVIRONMENT") | .Value' input.json
But as it turned out, the key could be also Environment. I try get both using this command:
jq -r '.TagList[] | select((.Key=="ENVIRONMENT") | .Value' or .Key=="Environment" | .Value)
but get the following error:
jq: error: syntax error, unexpected $end, expecting ';' or ')' (Unix shell quoting issues?) at <top-level>, line 1:
.TagList[] | select((.Key=="ENVIRONMENT") | .Value
jq: 1 compile error
How to get both of those tags?
You can use the following command:
jq '.TagList[]|select(.Key=="Environment" or .Key=="ENVIRONMENT").Value'
Generalizing a bit:
.TagList[]
| select(.Key | ascii_upcase == "ENVIRONMENT").Value
I have an array of objects of various types which reference one another with UUIDs (a terraform.tfstate file). I'd like to select one value from one such object based on the appearance of a different value in another object, where the two objects are related by one of those UUIDs.
By way of example, I can do this:
$ jq '.modules[].resources[]
| select(.type == "openstack_compute_instance_v2" and
.primary.attributes.name == "jumpbox").primary.id' terraform.tfstate
"5edfe2bf-94df-49d5-8118-3e91fb52946b"
$ jq '.modules[].resources[]
| select(.type =="openstack_compute_floatingip_associate_v2" and
.primary.attributes.instance_id == "5edfe2bf-94df-49d5-8118-3e91fb52946b").primary.attributes.floating_ip' terraform.tfstate
"10.120.241.21"
Giving me the external floating IP of the 'jumpbox' VM based on its name.
I'd like to make that all one jq call. Is that possible?
This would be easier to answer if you provided more sample data but
working backwards from your commands (with some reformatting)
$ jq '
.modules[].resources[]
| select(.type == "openstack_compute_instance_v2" and .primary.attributes.name == "jumpbox")
| .primary.id
' terraform.tfstate
"5edfe2bf-94df-49d5-8118-3e91fb52946b"
$ jq '
.modules[].resources[]
| select(.type =="openstack_compute_floatingip_associate_v2" and .primary.attributes.instance_id == "5edfe2bf-94df-49d5-8118-3e91fb52946b")
| .primary.attributes.floating_ip
' terraform.tfstate
"10.120.241.21"
we can infer you have data which looks like
{
"modules": [
{
"resources": [
{
"type": "openstack_compute_instance_v2",
"primary": {
"id": "5edfe2bf-94df-49d5-8118-3e91fb52946b",
"attributes": {
"name": "jumpbox"
}
}
},
{
"type": "openstack_compute_floatingip_associate_v2",
"primary": {
"attributes": {
"instance_id": "5edfe2bf-94df-49d5-8118-3e91fb52946b",
"floating_ip": "10.120.241.21"
}
}
}
]
}
]
}
The following filter demonstrates a solution using functions, variables and parenthesis ():
def get_primary_id($name):
select(.type == "openstack_compute_instance_v2"
and .primary.attributes.name == $name)
| .primary.id
;
def get_floating_ip($id):
select(.type =="openstack_compute_floatingip_associate_v2"
and .primary.attributes.instance_id == $id)
| .primary.attributes.floating_ip
;
.modules[]
| ( .resources[] | get_primary_id("jumpbox") ) as $id
| ( .resources[] | get_floating_ip($id) ) as $fip
| ($id, $fip)
if this filter is in filter.jq and data.json contains the sample data above
then
$ jq -M -f filter.jq data.json
produces the output:
"5edfe2bf-94df-49d5-8118-3e91fb52946b"
"10.120.241.21"
I have seen this issue. I guess it’s related to special char. I’ve read the jq manual and quote the field but still fail, I tried both with/without the bracket
abc#uswest1aprod 13:49:59 ~
$ cat test | jq .
{
"bus_v320161103-12-00-44": {
"aliases": {
"bus_v3": {}
},
"mappings": {
"business": {
"dynamic": "strict",
"_all": {
"enabled": false
}
}
}
}
abc#uswest1aprod 13:52:33 ~
$ cat test | jq ."bus_v320161103-12-00-44"
jq: error: null and number cannot be subtracted
abc#uswest1aprod 13:53:09 ~
$ cat test | jq .["bus_v320161103-12-00-44"]
error: bus_v320161103 is not defined
.[bus_v320161103-12-00-44] 1 compile error
You need to quote your filter so it isn't interpreted by the shell...
$ jq '."bus_v320161103-12-00-44"' test
Without it, it's effectively being passed in like this:
.bus_v320161103-12-00-44
Which is accessing a field called bus_v320161103 and subtracting that by 12, then 00 then 44.
Input:
{
"name":"JSON",
"good":true,
"target":"yes"
}
{
"name":"XML",
"good":false
}
I would like to exclude object WITHOUT key "target", as follow but NOT has:
jq -r ".| select(has(\"target\"))"
expected output:
{
"name":"XML",
"good":false
}
tried this:
jq -r " . | del(select(has(\"target\")))"
but there are two returned objects, one of them NULL
null
{
"good": false,
"name": "XML"
}
Select those who do not have target; that way, you do not use del:
jq -r 'select(has("target") | not)'