I have a huge JSON file with lots of stuff I don't care about, and I want to filter it down to only the few keys I care about, preserving the structure. I won't bother if the same key name might occur in different paths and I get both of them. I gleaned something very close from the answers to this question, it taught me how to delete all properties with certain values, like all null values:
del(..|nulls)
or, more powerfully
del(..|select(. == null))
I searched high and low if I could write a predicate over the name of a property when I am looking at a property. I come from XSLT where I could write something like this:
del(..|select(name|test("^(foo|bar)$")))
where name/1 would be the function that returns the property name or array index number where the current value comes from. But it seems that jq lacks the metadata on its values, so you can only write predicates about their value, and perhaps the type of their value (that's still just a feature of the value), but you cannot inspect the name, or path leading up to it?
I tried to use paths and leaf_paths and stuff like that, but I have no clue what that would do and tested it out to see how this path stuff works, but it seems to find child paths inside an object, not the path leading up to the present value.
So how could this be done, delete everything but a set of key values? I might have found a way here:
walk(
if type == "object" then
with_entries(
select( ( .key |test("^(foo|bar|...)$") )
and ( .value != "" )
and ( .value != null ) )
)
else
.
end
)
OK, this seems to work. But I still wonder it would be so much easier if we had a way of querying the current property name, array index, or path leading up to the present item being inspected with the simple recusion ..| form.
In analogy to your approach using .. and del, you could use paths and delpaths to operate on a stream of path arrays, and delete a given path if not all of its elements meet your conditions.
delpaths([paths | select(all(IN("foo", "bar") or type == "number") | not)])
For the condition I used IN("foo", "bar") but (type == "string" and test("^(foo|bar)$")) would work as well. To also retain array elements (which have numeric indices), I added or type == "number".
Unlike in XML, there's no concept of attributes in jq. You'll need to delete from objects.
To delete an element of an object, you need to use del( obj[ key ] ) (or use with_entries). You can get a stream of the keys of an object using keys[]/keys_unsorted[] and filter out the ones you don't want to delete.
Finally, you need to invert the result of test because you want to delete those that don't match.
After fixing these problems, we get the following:
INDEX( "foo", "bar" ) as $keep |
del(
.. | objects |
.[
keys_unsorted[] |
select( $keep[ . ] | not )
]
)
Demo on jqplay
Note that I substituted the regex match with a dictionary lookup. You could use test( "^(?:foo|bar)\\z" ) in lieu of $keep[ . ], but a dictionary lookup should be faster than a regex match. And it should be less error-prone too, considering you misused $ and (...) in lieu of \z and (?:...).
The above visits deleted branches for nothing. We can avoid that by using walk instead of ...
INDEX( "foo", "bar" ) as $keep |
walk(
if type == "object" then
del(
.[
keys_unsorted[] |
select( $keep[ . ] | not )
]
)
else
.
end
)
Demo on jqplay
Since I mentioned one could use with_entries instead of del, I'll demonstrate.
INDEX( "foo", "bar" ) as $keep |
walk(
if type == "object" then
with_entries( select( $keep[ .key ] ) )
else
.
end
)
Demo on jqplay
Here's a solution that uses a specialized variant of walk for efficiency (*). It retains objects all keys of which are removed; only trivial changes are needed if a blacklist or some other criterion (e.g., regexp-based) is given instead. WHITELIST should be a JSON array of the key names to be retained.
jq --argjson whitelist WHITELIST '
def retainKeys($array):
INDEX($array[]; .) as $keys
| def r:
if type == "object"
then with_entries( select($keys[.key]) )
| map_values( r )
elif type == "array" then map( r )
else .
end;
r;
retainKeys($whitelist)
' input.json
(*) Note for example:
the use of INDEX
the recursive function, r, has arity 0
for objects, the top-level deletion occurs first.
Here's a space-efficient, walk-free approach, tailored for the case of a WHITELIST. It uses the so-called "streaming" parser, so the invocation would look like this:
jq -n --stream --argjson whitelist WHITELIST -f program.jq input.json
where WHITELIST is a JSON array of the names of the keys to be deleted, and
where program.jq is a file containing the program:
# Input: an array
# Output: the longest head of the array that includes only numbers or items in the dictionary
def acceptable($dict):
last(label $out
| foreach .[] as $x ([];
if ($x|type == "number") or $dict[$x] then . + [$x]
else ., break $out
end));
INDEX( $whitelist[]; .) as $dict
| fromstream(inputs
| if length==2
then (.[0] | acceptable($dict)) as $p
| if ($p|length) == (.[0]|length) - 1 then .[0] = $p | .[1] = {}
elif ($p|length) < (.[0]|length) then empty
else .
end
else .
end )
Note: The reason this is relatively complicated is that it assumes that you want to retain objects all of whose keys have been removed, as illustrated in the following example. If that is not the case, then the required jq program is much simpler.
Example:
WHITELIST: '["items", "config", "spec", "setting2", "name"]'
input.json:
{
"items": [
{
"name": "issue1",
"spec": {
"config": {
"setting1": "abc",
"setting2": {
"name": "xyz"
}
},
"files": {
"name": "cde",
"path": "/home"
},
"program": {
"name": "apache"
}
}
},
{
"name": {
"etc": 0
}
}
]
}
Output:
{
"items": [
{
"name": "issue1",
"spec": {
"config": {
"setting2": {
"name": "xyz"
}
}
}
},
{
"name": {}
}
]
}
I am going to put my own tentative answer here.
The thing is, the solution I had already in my question, meaning I can select keys during forward navigation, but I cannot find out the path leading up to the present value.
I looked around in the source code of jq to see how come we cannot inquire the path leading up to the present value, so we could ask for the key string or array index of the present value. And indeed it looks like jq does not track the path while it walks through the input structure.
I think this is actually a huge opportunity forfeited that could be so easily kept track during the tree walk.
This is why I continue thinking that XML with XSLT and XPath is a much more robust data representation and tool chain than JSON. In fact, I find JSON harder to read even than XML. The benefit of the JSON being so close to javascript is really only relevant if - as I do in some cases - I read the JSON as a javascript source code assigning it to a variable, and then instrument it by changing the prototype of the anonymous JSON object so that I have methods to go with them. But changing the prototype is said to cause slowness. Though I don't think it does when setting it for otherwise anonymous JSON objects.
There is JsonPath that tries (by way of the name) to be something like what XPath is for XML. But it is a poor substitute and also has no way to navigate up the parent (or then sibling) axes.
So, in summary, while selecting by key in white or black lists is possible in principle, it is quite hard, because a pretty easy to have feature of a JSON navigation language is not specified and not implemented. Other useful features that could be easily achieved in jq is backward navigation to parent or ancestor of the present value. Currently, if you want to navigate back, you need to capture the ancestor you want to get back to as a variable. It is possible, but jq could be massively improved by keeping track of ancestors and paths.
I'm a jq newbie, and I try to transform a json (a Swagger spec). I want to add an element to the array value of the "parameter" keys:
{
...
"paths": {
"/great/endpoint1": {
"get": {
"parameters": [] <<--- add a value here
}
}
"/great/endpoint2": {
"post": {
"parameters": [] <<-- and here too here too etc.
....
The following jqplay almost works. It adds values to the right arrays, but it has the nasty side effect of also removing the "x-id" value from the root of the input json. It's probably because of a faulty if-condition. As the paths contain a varying string (the endpoint names), I don't know how to write a wildcard path expression to address those, which is why I have tried using walk instead:
https://jqplay.org/s/az56quLZa3
Since the sample data is incomplete, it's difficult to say exactly what you're looking for but it looks like you should be using parameters in the call to walk:
walk(if type=="object" and has("parameters")
then .parameters += [{"extra": "value"}]
else . end)
If you want to restrict the walk to the top-level paths, you would preface the above with: .paths |=
I have incoming param List<Somedata>.Somedata class contains id field.
My goal is to make HashMap<Somedata.id, Somedata> from this list.
Is next approach correct or there is a better/safer way to do that?
list
.filter { it.id != null }
.associateTo(HashMap(), {it.id!! to it})
Actually, I cannot understand, why should I use !! keyword in associateTo method, when above I filtered it with non-null values only.
Or maybe there is a good way to perform this with ?. or ?.let keywords?
You can do:
list.mapNotNull { e -> e.id?.let { it to e } }.toMap()
Breakdown:
The call to .let with the ?. safe call operator will make the result null if the element is null.
So the lambda passed to mapNotNull is of type (Somedata) -> Pair<IdType, Somedata>.
mapNotNull discards the null pairs, and toMap turns the resulting List<Pair<IdType, Somedata>> into a Map<IdType, Somedata>.
If you want to avoid the creation of an intermediate list to hold the pairs, you can turn the list into a lazy Sequence from the start:
list.asSequence().mapNotNull { e -> e.id?.let { it to e } }.toMap()
Alternatively, since you asked:
why should I use !! keyword in associateTo method, when above I filtered it with non-null values only.
this is because the list is still of type List<Somedata> - this says nothing about the nullability of the field itself. The compiler does not know that the id fields are still not null, by the time your associateTo call is executed.
I've got a global map in build.gradle (the config is written on Groovy).
I want to know whether it's possible to ask for already existing keys (probably, they doesn't exist yet, I understand) during the map declaration by the map literal way after those places where these keys have been declared.
ext.app = [
versions: [k1:'v1', k2: 'v2'],
dependencies: [d1:'dep1:$versions.k1', d2:'dep2:$versions.k2']
^^^^^^^^^^^ ^^^^^^^^^^^
]
Just to say "I define a new key with the value of the key which possibly may be in my map".
newKey: map(existingKey)
|
V
newKey: existingValue
Tried to clarify it simply. Any assistance would be appreciated.
Groovy can't autoreference its maps. with will work fine, though:
app = [:].with {
versions = [
k1: 'v1',
k2: 'v2'
]
dependencies = [
d1:"dep1:$versions.k1",
d2:"dep2:$versions.k2"
]
it
}
assert app.dependencies.d1 == "dep1:v1"
assert app.dependencies.d2 == "dep2:v2"
In a with block, all method calls are delegated to the calling object, thus, a $versions.k1 will be rerouted to the map being created.
Note the it at the end of the with block, otherwise the app variable won't contain the map, but instead will contain whatever that closure return.
I'm attempting to write a Meteor package which can be placed inside templates. So I first attempted to register a helper.
Template.registerHelper('testHelper', function(a, b) {
console.log(a);
console.log(b);
})
I've added the package inside /packages, and in my client template, when I added {{testHelper "hello" "meow"}}, the console logged hello and meow, which is what I expected.
When I added {{testHelper "hello"}}, I expected the console to log hello and null, since nothing was passed as the second parameter. But instead it returned hello and an object - Spacebars.kw {hash: Object}
What is this Spacebars.kw {hash: Object}? What can I do if I want it to return null instead?
Spacebars.kw contains a hash object that has a hash of input parameters.
Meteor has two methods to match up methods, one is direct matching which is where the parameters are directly input, e.g {{testHelper "variable1" "variable2" "variable3"}}, would match up as function(a,b,c) as variables 1-3 matching up to a,b and c respectively.
The second method of input is using a hash:
{{testHelper a="variable1" b="variable2" c="variable3"}}
This would give a single parameter to function(a) where a is a Spacebars.kw object.
The Spacebars.kw object would have a subobject called hash with a structure that matches:
{ "a" : "variable1",
"b" : "variable2",
"c" : "variable3" }
Meteor will attempt to match up the first param directly, but the subsequent parameters will be matched up as hashes incase the second input is empty such as in the case where you use {{testHelper 'hello'}} where b would be null, so it's given as the hash instead.
Its generically given as this, so if you get b as a Spacebars.kw object, you can assume there was no second input. The alternative is you could use the hash style declarations and then directly check if the hash value is null:
{{testHelper text="Hello"}}
{{testHelper text="Hello" othertext="Hellooo"}}
and the helper:
Template.registerHelper('testHelper', function(kw) {
console.log(kw.hash.text);
console.log(kw.hash.othertext);
});