I have a large dump of data in json that looks like:
[{
"recordList" : {
"record" : [{
"Production" : {
"creator" : {
"name" : "A"
}
}
},
{
"Production" : {}
},
{
"Production" : [{
"creator" : {
"name" : "B"
},
"creator" : {
"name" : "C"
}
}]
}]
}
}]
I need to check if there is at least one creator in a record or not. If there is I give a 1 else a 0 for that field in a CSV-file.
My code:
jq -r '.[].recordList.record[]|"\(if ((.Production.creator.name)? // (.Production[]?.creator.name)?) == null or ((.Production.creator.name)?|length // (.Production[]?.creator.name)?|length) == 0 then 0 else 1 end),"' file.json
The problem is that the field 'Production' is only an array when there are multiple creators.
The result I want to get in this case is:
1,
0,
1,
jq solution:
jq -r '.[].recordList.record[].Production
| "\(if ((type == "array" and .[0].creator.name !="")
or (type == "object" and .creator.name and .creator.name !=""))
then 1 else 0 end),"' file.json
The output:
1,
0,
1,
Simplified jq solution:
jq -r '.[].recordList.record[].Production
| ((type == "array" and .[0].creator.name) or .creator.name)
| if . then "1," else "0," end' file.json
Related
Suppose I have an object like:
{
"a": 1,
"b": 2,
"c": [
{
"d": 1,
"e": 2
},
{
"d": 2,
"e": 3
}
]
}
and I wish to extract only a set of keys (which are possibly nested), like .a and .c[].d, giving us the following output:
{
"a": 1,
"c": [
{
"d": 1
},
{
"d": 2
}
]
}
How would I go about doing that?
Here are two possible ways to do it:
Explicitly extracting the keys (like in this question), e.g.:
$ jq '{a, c: .c[] | {d}}' test.json
{
"a": 1,
"c": {
"d": 1
}
}
{
"a": 1,
"c": {
"d": 2
}
}
which works but can get ugly very quickly if you try to use it with long keys or deeply nested sub-objects.
Note that selecting paths that don't exist will result in null:
$ jq '{a, c: .c[] | {f}}' test.json
{
"a": 1,
"c": {
"f": null
}
}
{
"a": 1,
"c": {
"f": null
}
}
Implementing pick function to filter an object for specific keys:
def pick(paths):
. as $root |
reduce path(paths) as $path
({}; setpath($path; $root | getpath($path)));
Resulting in:
$ jq "$(cat query.jq)"' pick(.a, .c[].f, .c[].d)' test.json
{
"a": 1,
"c": [
{
"f": null,
"d": 1
},
{
"f": null,
"d": 2
}
]
}
If we'd like non-existent paths to be omitted instead of set to null, we can add haspath function to check if a path exists inside an object like so:
def haspath($path):
def h:
. as [$json, $p]
| (($p|length)==0) or
($json | (has($p[0]) and ( [getpath([$p[0]]), $p[1:] ] | h)));
[., $path] | h;
def pick(paths):
. as $root |
reduce path(paths) as $path
({}; if $root|haspath($path) then . + setpath($path; $root | getpath($path)) else . end);
Resulting in:
$ jq "$(cat query_haspath.jq)"' pick(.a, .c[].f, .c[].d)' test.json
{
"a": 1,
"c": [
{
"d": 1
},
{
"d": 2
}
]
}
Here is the jq I have, it just wants to build a new element and then append it to an array,
[.[] | . as { foo: $foo1, bar: $bar1} |
{
names: ([
$foo1 | range(0;length) as $i |
{ key: ($foo1[$i]) }
] + [{ key: $bar1 }])
} |
{
values: .names,
}
]
And suppose I have a json like this,
{
"foo":[
"key1",
"key2"
],
"bar": "key3"
}
This will generate a json file like this,
[
{
"values": [
{
"key": "key1"
},
{
"key": "key2"
},
{
"key": "key3"
}
]
}
]
But the element should be only appended when the $bar is not an empty string, can I do something like this?
[.[] | . as { foo1: $foo1, bar1: $bar1 if $bar != ""}
...
or do it when is appended,
names: ([
$foo1 | range(0;length) as $i |
{ key: ($foo1[$i]) }
] + [{ key: $bar1 }] | if $bar != "")
thanks in advance for any help!
You could just use select to filter out that case
jq '[{values: (.foo + [.bar | select(. != "")]) | map({key:.})}]'
If .bar == "key3", it prints
[
{
"values": [
{
"key": "key1"
},
{
"key": "key2"
},
{
"key": "key3"
}
]
}
]
If .bar == "", it prints
[
{
"values": [
{
"key": "key1"
},
{
"key": "key2"
}
]
}
]
If .bar does not exist, it will print
[
{
"values": [
{
"key": "key1"
},
{
"key": "key2"
},
{
"key": null
}
]
}
]
If in this last case you want to have the same result as with .bar == "", then change . != "" to values != "" in the filter to consider only values that are not null (or to strings != "" to only consider (non-empty) strings and disregard any other type).
Here my documents:
[
{
"id":"f3b8c257-9950-45e7-9e79-ace19ec8905e",
"identifier":[
{
"system":{
"value":"urn:oid:2.16.724.4.9.10.2"
},
"value":{
"value":"10839812"
}
}
]
},
{
"id":"f0a1e3ae-826f-4a03-b29e-10ef3bc86ea0",
"identifier":null
}
]
Currently, I'm aplying this jq filter:
map(
{
id,
dni: .identifier[] | (select(.system.value == "urn:oid:1.3.6.1.4.1.19126.3") | .value.value)
}
)
However, I'm getting this message:
jq: error (at practitioner-mongoexport.json:146715): Cannot iterate over null (null)
As you can guess, problem arises processing second object where .identifier: null.
I've tried with that:
map(
{
id,
dni: select(.identifier) | .identifier[] | (select(.system.value == "urn:oid:1.3.6.1.4.1.19126.3") | .value.value)
}
)
I've also tried with:
map(
select(.identifier) |
{
id,
dni: .identifier[] | (select(.system.value == "urn:oid:1.3.6.1.4.1.19126.3") | .value.value)
}
)
Then result is [].
I don't quite figure out what I'm doing wrong.
My desired output would be:
[
{
"id":"f3b8c257-9950-45e7-9e79-ace19ec8905e",
"dni": "10839812"
}
]
Any ideas?
You need to deal with the case if .identifier is null. Here, the ? operator may help.
without selection
jq 'map({id, dni: (.identifier[]?).value.value})'
Demo
with selection
jq 'map({id, dni: (.identifier[]? | select(.system.value == "urn:oid:2.16.724.4.9.10.2")).value.value})'
Demo
[
{
"id": "f3b8c257-9950-45e7-9e79-ace19ec8905e",
"dni": "10839812"
}
]
map(
select(.identifier?[]?.system.value == "urn:oid:2.16.724.4.9.10.2")
| { id, "dni": .identifier[].value.value }
)
Will generate:
[
{
"id": "f3b8c257-9950-45e7-9e79-ace19ec8905e",
"dni": "10839812"
}
]
The trick here is .identifier?[]?.system.value where the ? will ignore .identifier if it's null
JqPlay Demo
I have a JSON file that looks like this:
{
"InstanceId": "i-9KwoRGF6jbhYdZi823aE4qN",
"Tags": [
{
"Key": "blah",
"Value": "server-blah"
},
{
"Key": "environment",
"Value": "ops"
},
{
"Key": "server_role",
"Value": "appserver"
},
{
"Key": "Name",
"Value": "some_name"
},
{
"Key": "product",
"Value": "some_server"
}
]
}
{
...more objects like the above...
}
I need to display the InstanceId where "Key" == "environment" and "Value" == "ops".
I have jq-1.6.
If I say:
cat source.json | jq '
{ InstanceId, Tags } |
(.Tags[] | select( .Key == "environment" ))
'
I get some of what I want, but I cannot figure out how to include InstanceId in the output nor how to incorporate the "and" part of the select.
Here is a simple but efficient approach using any:
select( any(.Tags[]; .Key=="environment" and .Value == "ops") )
| .InstanceId
An alternative approach that avoids .Tags[]:
{"Key": "environment", "Value": "ops"} as $object
| select( .Tags | index($object) )
| .InstanceId
I'm not sure if this is the exact output you're looking for (comment if it isn't), but this will output the InstanceIds of JSON objects that contain a Tag with Key environment and Value ops.
jq 'select( .Tags[] | (.Key == "environment" and .Value == "ops")) | .InstanceId' < source.json
Here's my test data:
[
{
"id": "id-1",
"tags": {
"key": "name",
"value": "name-1"
}
},
{
"id": "id-2"
}
]
I'm trying to simplify the output, to show the 'name' field if present, and always show the id. For example, this script almost works:
~ $ cat testdata | jq '.[] | {id, name: .tags.value}'
{
"id": "id-1",
"name": "name-1"
}
{
"id": "id-2",
"name": null
}
When I try to add in a guard against .keys not existing and filter for the section of 'keys' I care about, here's what happens:
~ $ cat testdata | jq '.[] | {id, name: (select(.tags.key == "name") | .tags.value)}'
{
"id": "id-1",
"name": "name-1"
}
I assume {} is somehow ending up with a zero-length array instead of 'null'. What should I be using instead of |? What am I misunderstanding?
I ended up solving the problem using: [POSSIBLY_MATCHED_EXPRESSION][0], in this case:
cat testdata | jq '.[] | {id, name: ([select(.tags.key == "name") | .tags.value][0])}'
If I'm understanding correctly, if you wanted to include a name only if it existed, I'd do this:
map({id} + with_entries(select(.key == "tags") | .value))
Otherwise if you don't mind null names:
map({id, name: with_entries(select(.key == "tags") | .value) | .name})
Here's a more general solution if you have other "tags" so it's not hardcoded to only accept name values.
This assumes that any object value is actually a key/value pair.
map(with_entries(if .value | type == "object" then .value else . end))
Or if tags is the only dynamic property:
map(with_entries(if .key == "tags" then .value else . end))
If the goal is to produce:
{"id":"id-1","name":"name-1"}
{"id":"id-2"}
then the following three expressions are essentially equivalent solutions:
.[] | if .tags.key == "name" then {id, name: .tags.value} else {id} end
.[] | {id} + (if .tags.key == "name" then {name: .tags.value} else {} end)
.[] | (select(.tags.key == "name") | {id, name: .tags.value}) // {id}
You could just add
| if .name == null then del(.name) else . end
to the end of your filter to get rid of the .name key when its value is null.
With your test data, the following
.[]
| {id, name:.tags.value }
| if .name == null then del(.name) else . end
produces
{
"id": "id-1",
"name": "name-1"
}
{
"id": "id-2"
}