I'm trying to pass a variable to a jq delete function. It doesn't work with the variable, but does work if hard-coded. Could someone help me understand what I'm doing wrong.
Doesn't work:
echo '{"profiles": {"a":"1", "b":"2"}}' | jq --arg a '.a' '.profiles | del($a)'
Works:
echo '{"profiles": {"a":"1", "b":"2"}}' | jq '.profiles | del(.a)'
You can only pass raw strings (with the --arg option) or JSON-encoded strings (with the --argjson option) as arguments. You cannot pass jq code as a variable (and evaluate it inside jq).
You can, however, reference fields by their string representation. Thus, just use the field name as argument, and .[$var] to address it:
echo '{"profiles": {"a":"1", "b":"2"}}' | jq --arg a 'a' '.profiles | del(.[$a])'
{
"b": "2"
}
Related
Is there some way to use the name of an argument in the code?
For example, this command:
jq -n --arg name value '{($name): $name}'
has this output:
{
"value": "value"
}
Is it possible to get this output?:
{
"name": "value"
}
If you remove the ($name): JQ will use the name as the key fallback:
$ jq -n --arg name value '{$name}'
{
"name": "value"
}
$
From the jq Frequently Asked Questions:
Notable Differences between Versions
𝑸: In which version was the abbreviation {$x} for {"x": $x} introduced?
Version 1.5
From the Object Construction ({}) documentation:
The value can be any expression (although you may need to wrap it in parentheses if, for example, it contains colons), which gets applied to the {} expression's input (remember, all filters have an input and an output).
{foo: .bar}
will produce the JSON object {"foo": 42} if given the JSON object {"bar":42, "baz":43} as its input. You can use this to select particular fields of an object: if the input is an object with "user", "title", "id", and "content" fields and you just want "user" and "title", you can write
{user: .user, title: .title}
Because that is so common, there's a shortcut syntax for it:
{user, title}
Not directly. You already know the name you want to use, so either hard code it:
jq -n --arg name value '{name: $name}'
or use a second variable
jq -n --arg key name --arg name value '{($key): $name}'
(I forgot about #0stone0's approach; that's probably what you want to use.)
tl;dr
In the language of jq, why is
$ jq --compact-output reduce (1,2,3,4) as $i ([]; . + [$i])
[1,2,3,4]
not the same as
$ jq --compact-output (1,2,3,4) | reduce . as $i ([]; . + [$i])
[1]
[2]
[3]
[4]
Full question and discussion
I have a somewhat theoretical question in that I have figured out a way to get the transformation I want, but still I do not understand completely why my first attempt failed and I would like an explanation.
Interactive example at jqPlay
I have input
{
"data": {
"k1": "v1"
},
"item": {
"data": {
"k2": "v2"
}
},
"list": {
"item": {
"data": {
"k3": "v3",
"k4": "v4"
}
}
}
}
and I want to collect into a single array all of the values of all of the keys that are immediate children of a "data" key. So the output I want is
["v1","v2","v3","v4"]
I eventually figured out that this works
jq --compact-output '[.. | .data? | select(.) | to_entries | .[].value]'
My question is, why could I not get it to work with reduce? I originally tried
.. | .data? | select(.) | to_entries | .[].value | reduce . as $v ([]; . + [$v])
but that gave me
["v1"]
["v2"]
["v3"]
["v4"]
instead. My question is why? reduce is supposed to iterate over multiple values, but what kind of multiple values does it iterate over and what kind are treated as separate inputs to separate reduce statements?
I guess my fundamental confusion is when is . (dot) an expression with 4 results and when is it 4 expressions? Or if . is always a an expression with 1 result, how do you collect 4 results back into 1, which is what reduce is all about? Is the array operator the only way?
An expression of the form:
reduce STREAM as ...
reduces the given stream, whereas the compound expression:
STREAM | reduce . as ...
invokes reduce once for each item in the stream, and for each invocation, . is that item.
If the concept of streams is unclear in this context, you might be interested to read a stream-oriented introduction to jq that I wrote:
https://github.com/pkoppstein/jq/wiki/A-Stream-oriented-Introduction-to-jq
I'm trying to validate all versions in a versions.json file, and get as the output a json with only the invalid versions.
Here's a sample file:
{
"slamx": "16.4.0 ",
"sdbe": null,
"mimir": null,
"thoth": null,
"quasar": null,
"connectors": {
"s3": "16.0.17",
"azure": "6.0.17",
"url": "8.0.2",
"mongo": "7.0.15"
}
}
I can use the following jq script line to do what I want:
delpaths([paths(type == "string" and contains(" ") or type == "object" | not)])
| delpaths([paths(type == "object" and (to_entries | length == 0))])
And use it on a shell like this:
BAD_VERSIONS=$(jq 'delpaths([paths(type == "string" and contains(" ") or type == "object" | not)]) | delpaths([paths(type == "object" and (to_entries | length == 0))])' versions.json)
if [[ $BAD_VERSIONS != "{}" ]]; then
echo >&2 $'Bad versions detected in versions.json:\n'"$BAD_VERSIONS"
exit 1
fi
and get this as the output:
Bad versions detected in versions.json:
{
"slamx": "16.4.0 "
}
However, that's a very convoluted way of doing the filtering. Instead of just walking the paths tree and just saying "keep this, keep that", I need to create a list of things I do not want and remove them, twice.
Given all the path-handling builtins and recursive processing, I can't help but feel that there has to be a better way of doing this, something akin to select, but working recursively across the object, but the best I could do was this:
. as $input |
[path(recurse(.[]?)|select(strings|contains("16")))] as $paths |
reduce $paths[] as $x ({}; . | setpath($x; ($input | getpath($x))))
I don't like that for two reasons. First, I'm creating a new object instead of "editing" the old one. Second and foremost, it's full of variables, which points to a severe flow inversion issue, and adds to the complexity.
Any ideas?
Thanks to #jhnc's comment, I found a solution. The trick was using streams, which makes nesting irrelevant -- I can apply filters based solely on the value, and the objects will be recomposed given the key paths.
The first thing I tried did not work, however. This:
jq -c 'tostream|select(.[-1] | type=="string" and contains(" "))' versions.json
returns [["slamx"],"16.4.0 "], which is what I'm searching for. However, I could not fold it back into an object. For that to happen, the stream has to have the "close object" markers -- arrays with just one element, corresponding to the last key of the object being closed. So I changed it to this:
jq -c 'tostream|select((.[-1] | type=="string" and contains(" ")) or length==1)' versions.json
Breaking it down, .[-1] selects the last element of the array, which will be the value. Next, type=="string" and contains(" ") will select all values which are strings and contain spaces. The last part of the select, length==1, keeps all the "end" markers. Interestingly, it works even if the end marker does not correspond to the last key, so this might be brittle.
With that done, I can de-stream it:
jq -c 'fromstream(tostream|select((.[-1] | type=="string" and contains(" ")) or length==1))' versions.json
The jq expression is as follow:
fromstream(
tostream |
select(
(
.[-1] |
type=="string" and contains(" ")
) or
length==1
)
)
For objects, the test to_entries|length == 0 can be abbreviated to length==0.
If I understand the goal correctly, you could just use .., perhaps along the following lines:
..
| objects
| with_entries(
select(( .value|type == "string" and contains(" ")) or (.value|type == "object" and length==0)) )
| select(length>0)
paths
If you want the paths, then consider:
([], paths) as $p
| getpath($p)
| objects
| with_entries(
select(( .value|type == "string" and contains(" ")) or (.value|type == "object" and length==0)) )
| select(length>0) as $x
| {} | setpath($p; $x)
With your input modified so that s3 has a trailing blank, the above produces:
{"slamx":"16.4.0 "}
{"connectors":{"s3":"16.0.17 "}}
For instance, I might have something coming out of my jq command like this:
"some string"
"some thing"
"some ping"
...
Note that there is no outer object or array and no commas between items.
Or you might have something like:
["some string"
"some thing"
"some ping"]
["some wing"
"some bling"
"some fing"]
But again, no commas or outer object or array and no commas between them to indicate that this is JSON.
I keep thinking the answer is that it is called "raw", but I'm uncertain about this.
I'm specifically looking for a term to look for in the documentation that allows you to process the sorts of examples above, and I am at a loss as how to proceed.
To start with, the jq manual.yml describes the behavior of filters this way:
Some filters produce multiple results, for instance there's one that
produces all the elements of its input array. Piping that filter
into a second runs the second filter for each element of the
array. Generally, things that would be done with loops and iteration
in other languages are just done by gluing filters together in jq.
It's important to remember that every filter has an input and an
output. Even literals like "hello" or 42 are filters - they take an
input but always produce the same literal as output. Operations that
combine two filters, like addition, generally feed the same input to
both and combine the results. So, you can implement an averaging
filter as add / length - feeding the input array both to the add
filter and the length filter and then performing the division.
It's also important to keep in mind that the default behavior of jq is to run the filter you specify once for each JSON object. In the following example, jq runs the identity filter four times passing one value to it each time:
$ (echo 2;echo {}; echo []; echo 3) | jq .
2
{}
[]
3
What is happening here is similar to
$ jq -n '2, {}, [], 3 | .'
2
{}
[]
3
Since this isn't always what you want, the -s option can be used to tell jq to gather the separate values into an array and feed that to the filter:
$ (echo 2;echo {}; echo []; echo 3)| jq -s .
[
2,
{},
[],
3
]
which is similar to
$ jq -n '[2, {}, [], 3] | .'
[
2,
{},
[],
3
]
The jq manual.yml explains how the --raw-input/-R option can be included for even more control over input handing:
Don't parse the input as JSON. Instead, each line of text is passed to the filter as a string. If combined with --slurp,then the entire input is passed to the filter as a single long string.
You can see using the -s and -R options together in this example produces a different result:
$ (echo 2;echo {}; echo []; echo 3)| jq -s -R .
"2\n{}\n[]\n3\n"
I have a JSON array of Objects:
[{key1: value},{key2:value}, ...]
I would like to reduce these into the following structure:
{key1: value, key2: value, ...}
Is this possible to do with jq?
I was trying:
cat myjson.json | jq '.[] | {(.key): value}'
This doesn't quite work as it iterates over each datum rather than reducing it to one Object.
Note that jq has a builtin function called 'add' that the same thing that the first answer suggests, so you ought to be able to write:
jq add myjson.json
To expand on the other two answers a bit, you can "add" two objects together like this:
.[0] + .[1]
=> { "key1": "value", "key2": "value" }
You can use the generic reduce function to repeatedly apply a function between the first two items of a list, then between that result and the next item, and so on:
reduce .[] as $item ({}; . + $item)
We start with {}, add .[0], then add .[1] etc.
Finally, as a convenience, jq has an add function which is essentially an alias for exactly this function, so you can write the whole thing as:
add
Or, as a complete command line:
jq add myjson.json
I believe the following will work:
cat myjson.json | jq 'reduce .[] as $item ({}; . + $item)'
It takes each item in the array, and adds it to the sum of all the previous items.