I have the the following input json:
{
"TagList": [
{
"Key": "Environment",
"Value": "foo"
},
{
"Key": "ENVIRONMENT",
"Value": "bar"
}
]
}
I want to get the values of tags with the key ENVIRONMENT using jq:
jq -r '.TagList[] | select(.Key=="ENVIRONMENT") | .Value' input.json
But as it turned out, the key could be also Environment. I try get both using this command:
jq -r '.TagList[] | select((.Key=="ENVIRONMENT") | .Value' or .Key=="Environment" | .Value)
but get the following error:
jq: error: syntax error, unexpected $end, expecting ';' or ')' (Unix shell quoting issues?) at <top-level>, line 1:
.TagList[] | select((.Key=="ENVIRONMENT") | .Value
jq: 1 compile error
How to get both of those tags?
You can use the following command:
jq '.TagList[]|select(.Key=="Environment" or .Key=="ENVIRONMENT").Value'
Generalizing a bit:
.TagList[]
| select(.Key | ascii_upcase == "ENVIRONMENT").Value
Related
given this input
{
"properties": {
"hello": [
"saysay"
],
"test": [
"1"
]
},
"uri": "uri"
}
I've managed to check whether hello exists using this command
curl url | jq 'has("hello")'
Now how to check if the value of hello equals to saysay when it is a string variable ?
if hardcoded, below command return true which is expected
curl url | jq -r '.properties.hello[]=="saysay"'
when i try this one, I've got the following error : (23) Failed writing body
curl url | jq -r '.properties.$variable[]=="saysay"'
You can use --arg to set a jq variable from the command line.
field=hello
curl url | jq --arg property "$field" '.properties[$property][] == "saysay"'
I am parsing Elastic logs that look like
$ cat my_log_file.txt:
{"level": "INFO", "message": "foo"}
{"level": "WARN", "message": "bar"}
{"level": "WARN", "message": "baz"}
Because they're one per line, formerly I have used jq -s to slurp them into an actual array that I can run map on:
jq -s 'map(.message) | unique' my_log_file.txt
Now I want to select out only lines that have level != "INFO". I should be able to just use this cookbook recipe, but again jq is having trouble with each line being a separate object and not in an array.
I can't use slurp, because it doesn't work with this command:
jq 'select(."level" != "INFO")' my_log_file.txt
But when I want to map to .message again, I get the same error I got before when I wasn't using slurp:
$ jq 'select(."level" != "INFO") | map(.message) | unique' my_log_file.txt
jq: error (at <stdin>:90): Cannot index string with string "message"
How can I convert my records midstream -- after the select is done, to convert the result from
{"level": "WARN", "message": "bar"}
{"level": "WARN", "message": "bar"}
to
[
{"level": "WARN", "message": "bar"},
{"level": "WARN", "message": "bar"}
]
I heard that inputs was designed to replace slurp, but when I tried
$ jq 'select(."level" != "INFO") | [inputs]' my_log_file.txt
that ignored my select and I had a list of every message again.
I can't use slurp, because it doesn't work with this command
If you use slurp, you'll need to wrap the select in a map() to make it work:
jq -s 'map(select(.level != "INFO"))' my_log_file.txt
So to get all the unique message where level != "INFO", you can use:
jq -s 'map(select(.level != "INFO").message) | unique[]' my_log_file.txt
That will output:
"bar"
"baz"
As you can test in this online demo.
The efficient way to make the selection while forming an array is to use inputs with the -n command-line option, along the lines of:
jq -n '[inputs | select(."level" != "INFO") ]' my_log_file.txt
So your query could be:
[inputs | select(."level" != "INFO") | .message] | unique
or to avoid the sort entailed by the call to unique and assuming all the .message values are strings:
INDEX(inputs | select(.level != "INFO"); .message)
| keys_unsorted
or even better:
INDEX(inputs | select(.level != "INFO") | .message; .)[]
If you want the array to be splatted, just append [].
I am trying to query json using jq in bash scripting
file customers.json contains the following
[
{
"_id": "5968dd23fc13ae04d9000001",
"product_name": "sildenafil citrate",
"supplier": "Wisozk Inc",
"quantity": 262,
"unit_cost": "$1047"
},
{
"_id": "5968dd23fc13ae04d9000002",
"product_name": "Mountain Juniperus ashei",
"supplier": "Keebler-Hilpert",
"quantity": 292,
"unit_cost": "$874"
},
{
"_id": "5968dd23fc13ae04d9000003",
"product_name": "Dextromathorphan HBr",
"supplier": "Schmitt-Weissnat",
"quantity": 211,
"unit_cost": "$2053"
}
]
when i run following bash script
key="supplier"
value="Wisozk Inc"
jq ".[] | select(.$key==$value)" customers.json
It throws the following error
jq: error: syntax error, unexpected IDENT, expecting ';' or ')' (Unix shell quoting issues?) at <top-level>, line 1:
.[] | select(.supplier == Wisozk Inc)
jq: 1 compile error
I think the space between "Wisozk" and "Inc" is the problem, what to do?
Do not let your shell variables to be interpolated in jq under double quotes. The error is because of spaces in the value field, the variable $value undergoes word-splitting by and the filter expression of jq gets two words Wisozk and Inc instead of "Wisozk Inc".
Just pass your variables to jq's context using the --arg field and let it deal with it.
jq --arg k supplier --arg v "Wisozk Inc" 'map(select(.[$k] == $v))' json
This question is in parts duplicate to both the below questions, but not as a whole.
passing arguments to jq filter
Passing bash variable to jq
I was checking the jq tutorial at https://programminghistorian.org/en/lessons/json-and-jq
It makes some json reshaping, extracting some data from a json file, found at https://programminghistorian.org/assets/jq_twitter.json
At some point it makes a group_by, grouping data with the same user, extracting some user data and adding its corresponding tweet ids with the command
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]}'
so far, so good... the response looks like this (just a part is extracted):
{
"user_id": 18270633,
"user_name": "ahhthatswhy",
"user_followers": 559,
"tweet_ids": [
501064204661850100
]
}
{
"user_id": 27202261,
"user_name": "Dushan41",
"user_followers": 1201,
"tweet_ids": [
619172281751711700,
619172321564098600
]
}
{
"user_id": 2500422674,
"user_name": "pecanEgba74318",
"user_followers": 17,
"tweet_ids": [
619172331592773600
]
}
But then I would like to add a {"multiple_tweets": true} to all the objects that have more than one tweet_ids.
If I plainly pipe, like this, it works fine:
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | (select(.tweet_ids | length > 1) .multiple_tweets = true)'
a part of the result:
{
"user_id": 1653718716,
"user_name": "OAnnie8",
"user_followers": 315,
"tweet_ids": [
501064215160172540
]
}
{
"user_id": 356854246,
"user_name": "DrJLMooreIII",
"user_followers": 4888,
"tweet_ids": [
501064202904404000,
501064231387947000
],
"multiple_tweets": true
}
{
"user_id": 117155917,
"user_name": "rebekahwsm",
"user_followers": 5069,
"tweet_ids": [
501064233186893800
]
}
But if (for whatever reason, in this example is not really needed, in fact I was doing it just to understand the update-assignment) I want to use the |= operator,
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I get the error ' jq: error (at :30259): Invalid path expression with result {"user_id":1330235048,"use... '
Now the thing that I really can't understand. If instead of using the operator |= directly, I pipe through the identity operator first, it works fine.
What is the reason of this behaviour? Why does |.|= behave differently than |= ?
Why does this change anything?
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | . |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I guess I'm still not understanding how the |= operator really works.
Thank you for your help.
JQ manual explains that behavior as follows:
The left-hand side can be any general path expression; see path().
Note that the left-hand side of |= refers to a value in .. Thus $var.foo |= . + 1 won't work as expected ($var.foo is not a valid or useful path expression in .); use $var | .foo |= . + 1 instead.
Since the underlying builtin (_modify) is implemented using setpath, getpath, and delpaths; the LHS of |= must be a valid path expression that can be represented as an array; in other words, path(LHS) must not fail. See below examples.
$ jq -n 'path(1)'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 |= . + 1'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 | path(.)'
[]
$ jq -n '1 | . |= . + 1'
2
I'm trying to use jq to filter my results when the value contains quote literals so my data looks like:
{"key": "site=\"abc\""}
I want to filter using contains (or some other method) for where site=abc but not site=abc123
current code that gets abc and abc123:
jq -c '.textPayload | select(contains("abc"))' test.json
I attempted to try to escape using \ but it looks like it doesn't work in the contains method?
Consider:
$ echo '{"key": "site=\"abc\""}' | jq 'select(.key | contains("\"abc\""))'
{
"key": "site=\"abc\""
}
$ echo '{"key": "site=\"abc\""}' | jq 'select(.key | index("\"abc\""))'
{
"key": "site=\"abc\""
}
$ echo '{"key": "site=\"abc\""}' | jq 'select(.key | test("\"abc\""))'
{
"key": "site=\"abc\""
}
So it's unclear what the difficulty is.