I tried to use the if X and Y then A else B in jq, but there is some hidden context going on which gives errors. It is as if the condition in X changes a context or state that Y and A then use, but if I nest the if statements I do not get that behavior.
This works:
if .foo == "bar" then
if [.bazes[] | .bat=="blah"] | any then .add = "added" else . end
else . end
This does not work:
if .foo == "bar" and [.bazes[] | .bat=="blah"] | any then
.add = "added"
else . end
The query that fails does so because the associativity is not what you want. Adding parentheses to make your intentions clear is usually not a bad idea anyway:
if .foo == "bar"
and ([.bazes[] | .bat=="blah"] | any)
then .add = "added"
else . end
any/2
You could circumvent the issue, achieve a more efficient solution, and conform more closely to English usage by using any/2:
if .foo == "bar"
and any(.bazes[]; .bat=="blah")
then .add = "added"
else . end
Related
I have a huge JSON file with lots of stuff I don't care about, and I want to filter it down to only the few keys I care about, preserving the structure. I won't bother if the same key name might occur in different paths and I get both of them. I gleaned something very close from the answers to this question, it taught me how to delete all properties with certain values, like all null values:
del(..|nulls)
or, more powerfully
del(..|select(. == null))
I searched high and low if I could write a predicate over the name of a property when I am looking at a property. I come from XSLT where I could write something like this:
del(..|select(name|test("^(foo|bar)$")))
where name/1 would be the function that returns the property name or array index number where the current value comes from. But it seems that jq lacks the metadata on its values, so you can only write predicates about their value, and perhaps the type of their value (that's still just a feature of the value), but you cannot inspect the name, or path leading up to it?
I tried to use paths and leaf_paths and stuff like that, but I have no clue what that would do and tested it out to see how this path stuff works, but it seems to find child paths inside an object, not the path leading up to the present value.
So how could this be done, delete everything but a set of key values? I might have found a way here:
walk(
if type == "object" then
with_entries(
select( ( .key |test("^(foo|bar|...)$") )
and ( .value != "" )
and ( .value != null ) )
)
else
.
end
)
OK, this seems to work. But I still wonder it would be so much easier if we had a way of querying the current property name, array index, or path leading up to the present item being inspected with the simple recusion ..| form.
In analogy to your approach using .. and del, you could use paths and delpaths to operate on a stream of path arrays, and delete a given path if not all of its elements meet your conditions.
delpaths([paths | select(all(IN("foo", "bar") or type == "number") | not)])
For the condition I used IN("foo", "bar") but (type == "string" and test("^(foo|bar)$")) would work as well. To also retain array elements (which have numeric indices), I added or type == "number".
Unlike in XML, there's no concept of attributes in jq. You'll need to delete from objects.
To delete an element of an object, you need to use del( obj[ key ] ) (or use with_entries). You can get a stream of the keys of an object using keys[]/keys_unsorted[] and filter out the ones you don't want to delete.
Finally, you need to invert the result of test because you want to delete those that don't match.
After fixing these problems, we get the following:
INDEX( "foo", "bar" ) as $keep |
del(
.. | objects |
.[
keys_unsorted[] |
select( $keep[ . ] | not )
]
)
Demo on jqplay
Note that I substituted the regex match with a dictionary lookup. You could use test( "^(?:foo|bar)\\z" ) in lieu of $keep[ . ], but a dictionary lookup should be faster than a regex match. And it should be less error-prone too, considering you misused $ and (...) in lieu of \z and (?:...).
The above visits deleted branches for nothing. We can avoid that by using walk instead of ...
INDEX( "foo", "bar" ) as $keep |
walk(
if type == "object" then
del(
.[
keys_unsorted[] |
select( $keep[ . ] | not )
]
)
else
.
end
)
Demo on jqplay
Since I mentioned one could use with_entries instead of del, I'll demonstrate.
INDEX( "foo", "bar" ) as $keep |
walk(
if type == "object" then
with_entries( select( $keep[ .key ] ) )
else
.
end
)
Demo on jqplay
Here's a solution that uses a specialized variant of walk for efficiency (*). It retains objects all keys of which are removed; only trivial changes are needed if a blacklist or some other criterion (e.g., regexp-based) is given instead. WHITELIST should be a JSON array of the key names to be retained.
jq --argjson whitelist WHITELIST '
def retainKeys($array):
INDEX($array[]; .) as $keys
| def r:
if type == "object"
then with_entries( select($keys[.key]) )
| map_values( r )
elif type == "array" then map( r )
else .
end;
r;
retainKeys($whitelist)
' input.json
(*) Note for example:
the use of INDEX
the recursive function, r, has arity 0
for objects, the top-level deletion occurs first.
Here's a space-efficient, walk-free approach, tailored for the case of a WHITELIST. It uses the so-called "streaming" parser, so the invocation would look like this:
jq -n --stream --argjson whitelist WHITELIST -f program.jq input.json
where WHITELIST is a JSON array of the names of the keys to be deleted, and
where program.jq is a file containing the program:
# Input: an array
# Output: the longest head of the array that includes only numbers or items in the dictionary
def acceptable($dict):
last(label $out
| foreach .[] as $x ([];
if ($x|type == "number") or $dict[$x] then . + [$x]
else ., break $out
end));
INDEX( $whitelist[]; .) as $dict
| fromstream(inputs
| if length==2
then (.[0] | acceptable($dict)) as $p
| if ($p|length) == (.[0]|length) - 1 then .[0] = $p | .[1] = {}
elif ($p|length) < (.[0]|length) then empty
else .
end
else .
end )
Note: The reason this is relatively complicated is that it assumes that you want to retain objects all of whose keys have been removed, as illustrated in the following example. If that is not the case, then the required jq program is much simpler.
Example:
WHITELIST: '["items", "config", "spec", "setting2", "name"]'
input.json:
{
"items": [
{
"name": "issue1",
"spec": {
"config": {
"setting1": "abc",
"setting2": {
"name": "xyz"
}
},
"files": {
"name": "cde",
"path": "/home"
},
"program": {
"name": "apache"
}
}
},
{
"name": {
"etc": 0
}
}
]
}
Output:
{
"items": [
{
"name": "issue1",
"spec": {
"config": {
"setting2": {
"name": "xyz"
}
}
}
},
{
"name": {}
}
]
}
I am going to put my own tentative answer here.
The thing is, the solution I had already in my question, meaning I can select keys during forward navigation, but I cannot find out the path leading up to the present value.
I looked around in the source code of jq to see how come we cannot inquire the path leading up to the present value, so we could ask for the key string or array index of the present value. And indeed it looks like jq does not track the path while it walks through the input structure.
I think this is actually a huge opportunity forfeited that could be so easily kept track during the tree walk.
This is why I continue thinking that XML with XSLT and XPath is a much more robust data representation and tool chain than JSON. In fact, I find JSON harder to read even than XML. The benefit of the JSON being so close to javascript is really only relevant if - as I do in some cases - I read the JSON as a javascript source code assigning it to a variable, and then instrument it by changing the prototype of the anonymous JSON object so that I have methods to go with them. But changing the prototype is said to cause slowness. Though I don't think it does when setting it for otherwise anonymous JSON objects.
There is JsonPath that tries (by way of the name) to be something like what XPath is for XML. But it is a poor substitute and also has no way to navigate up the parent (or then sibling) axes.
So, in summary, while selecting by key in white or black lists is possible in principle, it is quite hard, because a pretty easy to have feature of a JSON navigation language is not specified and not implemented. Other useful features that could be easily achieved in jq is backward navigation to parent or ancestor of the present value. Currently, if you want to navigate back, you need to capture the ancestor you want to get back to as a variable. It is possible, but jq could be massively improved by keeping track of ancestors and paths.
I'm trying to validate all versions in a versions.json file, and get as the output a json with only the invalid versions.
Here's a sample file:
{
"slamx": "16.4.0 ",
"sdbe": null,
"mimir": null,
"thoth": null,
"quasar": null,
"connectors": {
"s3": "16.0.17",
"azure": "6.0.17",
"url": "8.0.2",
"mongo": "7.0.15"
}
}
I can use the following jq script line to do what I want:
delpaths([paths(type == "string" and contains(" ") or type == "object" | not)])
| delpaths([paths(type == "object" and (to_entries | length == 0))])
And use it on a shell like this:
BAD_VERSIONS=$(jq 'delpaths([paths(type == "string" and contains(" ") or type == "object" | not)]) | delpaths([paths(type == "object" and (to_entries | length == 0))])' versions.json)
if [[ $BAD_VERSIONS != "{}" ]]; then
echo >&2 $'Bad versions detected in versions.json:\n'"$BAD_VERSIONS"
exit 1
fi
and get this as the output:
Bad versions detected in versions.json:
{
"slamx": "16.4.0 "
}
However, that's a very convoluted way of doing the filtering. Instead of just walking the paths tree and just saying "keep this, keep that", I need to create a list of things I do not want and remove them, twice.
Given all the path-handling builtins and recursive processing, I can't help but feel that there has to be a better way of doing this, something akin to select, but working recursively across the object, but the best I could do was this:
. as $input |
[path(recurse(.[]?)|select(strings|contains("16")))] as $paths |
reduce $paths[] as $x ({}; . | setpath($x; ($input | getpath($x))))
I don't like that for two reasons. First, I'm creating a new object instead of "editing" the old one. Second and foremost, it's full of variables, which points to a severe flow inversion issue, and adds to the complexity.
Any ideas?
Thanks to #jhnc's comment, I found a solution. The trick was using streams, which makes nesting irrelevant -- I can apply filters based solely on the value, and the objects will be recomposed given the key paths.
The first thing I tried did not work, however. This:
jq -c 'tostream|select(.[-1] | type=="string" and contains(" "))' versions.json
returns [["slamx"],"16.4.0 "], which is what I'm searching for. However, I could not fold it back into an object. For that to happen, the stream has to have the "close object" markers -- arrays with just one element, corresponding to the last key of the object being closed. So I changed it to this:
jq -c 'tostream|select((.[-1] | type=="string" and contains(" ")) or length==1)' versions.json
Breaking it down, .[-1] selects the last element of the array, which will be the value. Next, type=="string" and contains(" ") will select all values which are strings and contain spaces. The last part of the select, length==1, keeps all the "end" markers. Interestingly, it works even if the end marker does not correspond to the last key, so this might be brittle.
With that done, I can de-stream it:
jq -c 'fromstream(tostream|select((.[-1] | type=="string" and contains(" ")) or length==1))' versions.json
The jq expression is as follow:
fromstream(
tostream |
select(
(
.[-1] |
type=="string" and contains(" ")
) or
length==1
)
)
For objects, the test to_entries|length == 0 can be abbreviated to length==0.
If I understand the goal correctly, you could just use .., perhaps along the following lines:
..
| objects
| with_entries(
select(( .value|type == "string" and contains(" ")) or (.value|type == "object" and length==0)) )
| select(length>0)
paths
If you want the paths, then consider:
([], paths) as $p
| getpath($p)
| objects
| with_entries(
select(( .value|type == "string" and contains(" ")) or (.value|type == "object" and length==0)) )
| select(length>0) as $x
| {} | setpath($p; $x)
With your input modified so that s3 has a trailing blank, the above produces:
{"slamx":"16.4.0 "}
{"connectors":{"s3":"16.0.17 "}}
This is my input JSON
{
"uuid":123,
"description": [
{
"car_name":"Toyota",
"saleDate":"23 May 2017"
}
]
}
Expected output:
{
"uuid":123,
"description": [
{
"name":"Toyota",
"saleDate":"23 May 2017"
}
]
}
Rule : .description[]|={name:.car_name,saleDate}
I am able to achieve the desired result with this rule. However, is there a way I can rename the 'car_name' property to 'name' and also not mention all the other properties ( in this case, 'saleDate') in the rule. I might end up having 50+ properties inside the object and I do not want to mention all of them in the rule.
There can be more than one object in the array.
To preserve the order of keys while being sure only to change the specific key name:
jq '.description[] |= with_entries(if .key == "car_name" then .key = "name" else . end)'
I just learned about += from an answer to another question. It works here too.
jq '.description[] |= (select(has("car_name")) += {"name":.car_name} | del(.car_name))'
The select function selects the objects that meet the condition. Then we add to, and delete from, those objects.
Does this do what you want?
jq '.description[]
|= . +
if has("car_name") then {"name":.car_name}
else null end
| del(.description[].car_name)'
The if-then block adds the object {name:.car_name} to any object in the description array that has a key with the name "car_name" and adds nothing (null) to any object in the array that does not have a key with that name.
The del function deletes key-value pairs at the path expression .description[].car_name
This seems to work while preserving the order.
jq '.description[] |= with_entries( .key |= gsub("car_name"; "name") )'
The jq manual explains what with_entries does.
Credit goes to an answer to a similar question by peak.
jq '.description[] |= with_entries(if .key == "car_name" then .key = "name" else . end)'
I want to pass my returned list into case but I'm getting the
"->" Error
Here is my code:
parse ( Element, [] ) -> false;
parse(Element,N) -> re:split(N,"([*+])",[{return,list}]),
parse ( Element, [ Item | ListTail ] ) ->
case ( Item == Element ) of
true -> true;
false -> listFind(Element, ListTail)
end.
I don't know how to pass the returned list into the case, Can anyone help?
If you want to test the result of re:split/3 in a case, you need to get rid of the extraneous function head stuck inside the second clause of parse/2. You also don't need the first clause of parse/2, as it's unnecessary and not used. Just look through the result of re:split/3 for Element, like this:
parse(Element, N) ->
Elements = re:split(N,"([*+])",[{return,list}]),
lists:member(Element, Elements).
If for example the argument N has the value "a*b+c*d+e", the re:split/3 call returns ["a","*","b","+","c","*","d","+","e"] (and let's assume that's really what you want). The call to lists:member/2 then searches that result for the value of Element, returning true if found or false if not found.
I've got a Dictionary like this:
a PluggableDictionary(
Rankable1->8.5
Rankable2->9.0
)
I need just an OrderedCollection with the Rankable objects in descending order:
a OrderedCollection(
Rankable2
Rankable1
)
I noticed it is easy to sort by keys, but I found it a bit more difficult to sort by values. What is the smalltalk way of doing this?
If you need one shot sorted collection in noncritical loop you might use something like this (uses pharo syntax to initialize example dictionary):
pd := PluggableDictionary newFromPairs: { 'a' . 2 . 'b' . 1 . 'c' . 3} .
(pd associations asSortedCollection: [:x :y | x value < y value])
collect: [:assoc | assoc key].
If you would need it more often, than you might consider introducing your own class that will keep this collection calculated.
If you're using VisualWorks, you can take advantage of SortFunction and Symbol>>value behavior to reduce all of that down to
(aDictionary associations sort: #value ascending) collect: #key
If you can use Grease (eg, when using Seaside), you can
probably use its GROrderedMultiMap. It is intended for small dictionaries with probably multiple values per key.
On a second note, probably you can swap key and value, and just send #asSortedCollection, like this:
(Dictionary newFrom: { 2 -> 'b' . 1-> 'a' })
asSortedCollection "--> a SortedCollection('a' 'b')"
(Tested in Squeak and Pharo)
Got it:
^ ((SortedCollection sortBlock:
[:association :otherAssociation | association value > otherAssociation value])
addAll: theDictionary associations;
yourself) collect: [:association | association key]