I have seen this issue. I guess it’s related to special char. I’ve read the jq manual and quote the field but still fail, I tried both with/without the bracket
abc#uswest1aprod 13:49:59 ~
$ cat test | jq .
{
"bus_v320161103-12-00-44": {
"aliases": {
"bus_v3": {}
},
"mappings": {
"business": {
"dynamic": "strict",
"_all": {
"enabled": false
}
}
}
}
abc#uswest1aprod 13:52:33 ~
$ cat test | jq ."bus_v320161103-12-00-44"
jq: error: null and number cannot be subtracted
abc#uswest1aprod 13:53:09 ~
$ cat test | jq .["bus_v320161103-12-00-44"]
error: bus_v320161103 is not defined
.[bus_v320161103-12-00-44] 1 compile error
You need to quote your filter so it isn't interpreted by the shell...
$ jq '."bus_v320161103-12-00-44"' test
Without it, it's effectively being passed in like this:
.bus_v320161103-12-00-44
Which is accessing a field called bus_v320161103 and subtracting that by 12, then 00 then 44.
Related
I have a JSON object where the relevant parts are of the form
{
"_meta": {
"hostvars": {
"name_1": {
"ansible_host": "10.0.0.1"
},
"name_2": {
"ansible_host": "10.0.0.2"
},
"name_3": {
"ansible_host": "10.0.0.3"
}
}
},
...
"nodes": {
"hosts": [
"name_1",
"name_2"
]
}
}
(the output of ansible-inventory --list, for reference).
I would like to use jq to produce a list of IPs of the nodes hosts by looking up the names in ._meta.hostvars. In the example, the output should be:
10.0.0.1
10.0.0.2
Note that 10.0.0.3 should not be included because name_3 is not in the .nodes.hosts list. So just doing jq -r '._meta.hostvars[].ansible_host' doesn't work.
I've tried jq '.nodes.hosts[] | ._meta.hostvars[.].ansible_host' but that fails because ._meta doesn't scan from the root after the pipe.
You can store the root in a variable before changing the context:
jq -r '. as $root | .nodes.hosts[] | $root._meta.hostvars[.].ansible_host'
But a better solution is to just inline the "hosts" query:
jq -r '._meta.hostvars[.nodes.hosts[]].ansible_host'
I was checking the jq tutorial at https://programminghistorian.org/en/lessons/json-and-jq
It makes some json reshaping, extracting some data from a json file, found at https://programminghistorian.org/assets/jq_twitter.json
At some point it makes a group_by, grouping data with the same user, extracting some user data and adding its corresponding tweet ids with the command
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]}'
so far, so good... the response looks like this (just a part is extracted):
{
"user_id": 18270633,
"user_name": "ahhthatswhy",
"user_followers": 559,
"tweet_ids": [
501064204661850100
]
}
{
"user_id": 27202261,
"user_name": "Dushan41",
"user_followers": 1201,
"tweet_ids": [
619172281751711700,
619172321564098600
]
}
{
"user_id": 2500422674,
"user_name": "pecanEgba74318",
"user_followers": 17,
"tweet_ids": [
619172331592773600
]
}
But then I would like to add a {"multiple_tweets": true} to all the objects that have more than one tweet_ids.
If I plainly pipe, like this, it works fine:
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | (select(.tweet_ids | length > 1) .multiple_tweets = true)'
a part of the result:
{
"user_id": 1653718716,
"user_name": "OAnnie8",
"user_followers": 315,
"tweet_ids": [
501064215160172540
]
}
{
"user_id": 356854246,
"user_name": "DrJLMooreIII",
"user_followers": 4888,
"tweet_ids": [
501064202904404000,
501064231387947000
],
"multiple_tweets": true
}
{
"user_id": 117155917,
"user_name": "rebekahwsm",
"user_followers": 5069,
"tweet_ids": [
501064233186893800
]
}
But if (for whatever reason, in this example is not really needed, in fact I was doing it just to understand the update-assignment) I want to use the |= operator,
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I get the error ' jq: error (at :30259): Invalid path expression with result {"user_id":1330235048,"use... '
Now the thing that I really can't understand. If instead of using the operator |= directly, I pipe through the identity operator first, it works fine.
What is the reason of this behaviour? Why does |.|= behave differently than |= ?
Why does this change anything?
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | . |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I guess I'm still not understanding how the |= operator really works.
Thank you for your help.
JQ manual explains that behavior as follows:
The left-hand side can be any general path expression; see path().
Note that the left-hand side of |= refers to a value in .. Thus $var.foo |= . + 1 won't work as expected ($var.foo is not a valid or useful path expression in .); use $var | .foo |= . + 1 instead.
Since the underlying builtin (_modify) is implemented using setpath, getpath, and delpaths; the LHS of |= must be a valid path expression that can be represented as an array; in other words, path(LHS) must not fail. See below examples.
$ jq -n 'path(1)'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 |= . + 1'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 | path(.)'
[]
$ jq -n '1 | . |= . + 1'
2
I need help in correcting jq test cases syntax. Following is output file & trying to test ID list with command below. Gives error index to string type.
[[ $(echo $output| jq -r '.output.value[] | select(.identity).id_list') == *"id2"* ]]
output = {
"resource_output": {
"value": {
"identity": [
{
"id_list": [
"/subscriptions/---/id1",
"/subscriptions/---/id2",
"/subscriptions/--/id3"
],
"principal_id": "",
"tenant_id": "",
"type": "managed"
}
]
}
}
Your query does not match the sample JSON, and you have not indicated what output you are expecting, but the following variation of your query illustrates how to use select and test with your data along the lines suggested by your attempt:
echo "$output" |
jq -r '.resource_output.identity[].id_list[] | select(test("id2"))'
Output:
/subscriptions/---/id2
Input:
{
"name":"JSON",
"good":true,
"target":"yes"
}
{
"name":"XML",
"good":false
}
I would like to exclude object WITHOUT key "target", as follow but NOT has:
jq -r ".| select(has(\"target\"))"
expected output:
{
"name":"XML",
"good":false
}
tried this:
jq -r " . | del(select(has(\"target\")))"
but there are two returned objects, one of them NULL
null
{
"good": false,
"name": "XML"
}
Select those who do not have target; that way, you do not use del:
jq -r 'select(has("target") | not)'
I am using jq to parse log data, occasionally the logs contain malformed stuff (invalid json), when this happens, jq aborts processing at that point.
Is there a way to have jq keep processing what it can, while reporting the problems via stderr?
I understand that if you have newlines in your JSON, jq may have trouble if it starts with the next line, but in such cases you will still eventually get to the point that you find the start of a legitimate json message and can continue processing.
With jq-1.5 I was able to do the following:
With this example file:
cat << EOF > example.log
{"a": 1}
{invalid
{"b": 2}
EOF
Output non-json lines as unquoted strings:
cat example.log | jq --raw-input --raw-output '. as $raw | try fromjson catch $raw'
{
"a": 1
}
{invalid
{
"b": 2
}
Silently skip non-json lines:
cat example.log | jq --raw-input 'fromjson?'
{
"a": 1
}
{
"b": 2
}
You can add --slurp if the entire input is expected to be a single multiline json blob.
Example files:
cat << EOF > valid-multiline.log
{
"a": 1,
"b": 2
}
EOF
cat << EOF > invalid-multiline.log
{
asdf
"b": 2
}
EOF
Outputs
cat valid-multiline.log | jq --slurp --raw-input --raw-output '. as $raw | try fromjson catch $raw'
{
"a": 1,
"b": 2
}
cat invalid-multiline.log | jq --slurp --raw-input --raw-output '. as $raw | try fromjson catch $raw'
{
asdf
"b": 2
}
If you have jq 1.5, the answer is: yes, though in general, preprocessing (e.g. using hjson or any-json) would be preferable.
Anyway, the idea is simply to take advantage of the try/catch feature. Here is an illustration using the inputs filter. Note that jq should in general be invoked with the -n option for this to work.
recover.jq
def handle: inputs | [., "length is \(length)"] ;
def process: try handle catch ("Failed", process) ;
process
bad.json
[1,2,3]
{id=546456, userId=345345}
[4,5,6]
See jq run:
$ jq -n -f recover.jq bad.json
[
"[1,2,3]",
"length is 3"
]
"Failed"
[
"[4,5,6]",
"length is 3"
]