I have some json data that looks like:
{
"p": {
"d": {
"a" : {
"r": "foo",
"g": 1
},
"b": {
"r": "bar",
"g": 2
}
},
"c": {
"e": {
"r": "baz",
"g": 1
}
},
...
}
}
I want something like:
{
"d": [
"a",
"b"
],
"c": [
"e"
]
}
I can get the list of keys on the first level under "p" with jq '.p|keys', and the structure and keys on the second level with jq '.p|map(.|keys)', but I can't figure out how to combine it.
Use map_values instead of map to map the values of a JSON object while preserving the keys:
jq '.p | map_values(keys)'
On jq versions lower than 1.5, map_values is not defined: instead, you can use []|=:
jq '.p | . []|= keys'
In general
Top level keys:
curl -s https://crates.io/api/v1/crates/atty | jq '. |= keys'
[
"categories",
"crate",
"keywords",
"versions"
]
Two levels of keys:
curl -s https://crates.io/api/v1/crates/atty | jq '.| map_values(keys)'
{
"crate": [
"badges",
"categories",
"created_at",
"description",
"documentation",
"downloads",
"exact_match",
"homepage",
"id",
"keywords",
"links",
"max_version",
"name",
"newest_version",
"recent_downloads",
"repository",
"updated_at",
"versions"
],
"versions": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16
],
"keywords": [
0,
1,
2
],
"categories": []
}
Method versions
topLevelJsonKeys() {
curl -s $1 | jq '. |= keys'
# EXAMPLE:
# topLevelJsonKeys https://crates.io/api/v1/crates/atty
}
topLevelJsonKeys2() {
curl -s $1 | jq '.| map_values(keys)'
# EXAMPLE:
# topLevelJsonKeys2 https://crates.io/api/v1/crates/atty
}
Here is a solution which uses reduce and setpath
.p
| reduce keys[] as $k (
.
; setpath([$k]; .[$k] | keys)
)
Related
The simplest version of the input document I could come up with is
{
"references": [
{
"version": 5,
"id": "id1",
"objType": "A"
},
{
"version": 4,
"id": "id2",
"objType": "B",
"referencing": []
},
{
"version": 4,
"id": "id3",
"objType": "B",
"referencing": [
{
"version": 2,
"id": "id4",
"objType": "A"
},
{
"version": 3,
"id": "id5",
"objType": "B",
"referencing": []
}
]
}
]
}
Objects of type A have no referencing objects.
Objects of type B can be referenced by either type of object.
There are two outputs I need from this json:
Output #1 is the version info for objects of type A with the id value as a key with the value of version. A objects can be at the top level or at some arbitrary depth in the referencing arrays.
{
"references": {
"id1": {"version": 5},
"id4": {"version": 2}
}
}
The 2nd output is similar: the version info for objects of type B. The can be a chain of type B objects referencing other type B objects.
{
"references": {
"id2": {"version": 4},
"id3": {"version": 4},
"id5": {"version": 3}
}
}
Use recursive decsent operator and from_entries. You don't need to follow the "references" (at least not to produce the expected output in your question)
{
dependencies: [.. | select(.objType=="A")? | { key: .id, value: {version} }] | from_entries
},
{
dependencies: [.. | select(.objType=="B")? | { key: .id, value: {version} }] | from_entries
}
Output:
{
"dependencies": {
"id1": {
"version": 5
},
"id4": {
"version": 2
}
}
}
{
"dependencies": {
"id2": {
"version": 4
},
"id3": {
"version": 4
},
"id5": {
"version": 3
}
}
}
It's also possible to merge (add) objects instead of constructing them from their entries, which makes the code minimally shorter:
{
dependencies: [.. | select(.objType=="A")? | { (.id): {version} }] | add
}
You can use recurse to traverse the document, INDEX to create an object with IDs as keys, map_values to format their values using select to reduce according to your criteria.
jq --arg type A '
.references |= (
INDEX(.[] | recurse(.referencing[]?); .id)
| map_values(select(.objType == $type) | {version})
)
'
{
"references": {
"id1": {
"version": 5
},
"id4": {
"version": 2
}
}
}
Demo
This works for both questions, provide A or B to --arg type.
Note that this is using the error suppression operator ? when recursing down. If you want to restrict the traversal explicitly to .objType == "B", just prepend it in a select expression, i.e. replace recurse(.referencing[]?) with recurse(select(.objType == "B") | .referencing[]). Demo
Suppose I have an object like:
{
"a": 1,
"b": 2,
"c": [
{
"d": 1,
"e": 2
},
{
"d": 2,
"e": 3
}
]
}
and I wish to extract only a set of keys (which are possibly nested), like .a and .c[].d, giving us the following output:
{
"a": 1,
"c": [
{
"d": 1
},
{
"d": 2
}
]
}
How would I go about doing that?
Here are two possible ways to do it:
Explicitly extracting the keys (like in this question), e.g.:
$ jq '{a, c: .c[] | {d}}' test.json
{
"a": 1,
"c": {
"d": 1
}
}
{
"a": 1,
"c": {
"d": 2
}
}
which works but can get ugly very quickly if you try to use it with long keys or deeply nested sub-objects.
Note that selecting paths that don't exist will result in null:
$ jq '{a, c: .c[] | {f}}' test.json
{
"a": 1,
"c": {
"f": null
}
}
{
"a": 1,
"c": {
"f": null
}
}
Implementing pick function to filter an object for specific keys:
def pick(paths):
. as $root |
reduce path(paths) as $path
({}; setpath($path; $root | getpath($path)));
Resulting in:
$ jq "$(cat query.jq)"' pick(.a, .c[].f, .c[].d)' test.json
{
"a": 1,
"c": [
{
"f": null,
"d": 1
},
{
"f": null,
"d": 2
}
]
}
If we'd like non-existent paths to be omitted instead of set to null, we can add haspath function to check if a path exists inside an object like so:
def haspath($path):
def h:
. as [$json, $p]
| (($p|length)==0) or
($json | (has($p[0]) and ( [getpath([$p[0]]), $p[1:] ] | h)));
[., $path] | h;
def pick(paths):
. as $root |
reduce path(paths) as $path
({}; if $root|haspath($path) then . + setpath($path; $root | getpath($path)) else . end);
Resulting in:
$ jq "$(cat query_haspath.jq)"' pick(.a, .c[].f, .c[].d)' test.json
{
"a": 1,
"c": [
{
"d": 1
},
{
"d": 2
}
]
}
I have a dataset which looks like the following
{
"metadata":"d_meta_v_1.5.9",
"data": {
"a": {
"T": [
1652167964645,
1652168781684,
1652168781720
],
"V": [
1,
2,
3
]
},
"b": {
"T": [
1652167961657,
1652168781720,
1652168781818
],
"V": [
1,
3,
4
]
},
"c": {
"T": [
1652167960194,
1652168787377
],
"V": [
1,
3
]
}
}
}
I want to select the certain column and carry on the metadata also at the end. a part of this question is working in my perviou question here
How can I get my desired output ?
Metadata, Time, a, b
d_meta_v_1.5.9, <Time>, <value of _a>, < value of b>
d_meta_v_1.5.9, <Time>, <value of _a>, < value of b>
d_meta_v_1.5.9, <Time>, <value of _a>, < value of b>
let requested_columns = dynamic(["a","b"]);
datatable(doc:dynamic)
[
dynamic
(
{
"metadata":"d_meta_v_1.5.9",
"data": {
"a": {
"T": [
1652167964645,
1652168781684,
1652168781720
],
"V": [
1,
2,
3
]
},
"b": {
"T": [
1652167961657,
1652168781720,
1652168781818
],
"V": [
1,
3,
4
]
},
"c": {
"T": [
1652167960194,
1652168787377
],
"V": [
1,
3
]
}
}
}
)
]
| project metadata = doc.metadata, data = doc.data
| mv-expand data = data
| extend key = tostring(bag_keys(data)[0])
| where key in (requested_columns)
| mv-expand T = data[key].T to typeof(long), V = data[key].V to typeof(long)
| evaluate pivot(key, take_any(V), metadata, T)
| order by T asc
metadata
T
a
b
d_meta_v_1.5.9
1652167961657
1
d_meta_v_1.5.9
1652167964645
1
d_meta_v_1.5.9
1652168781684
2
d_meta_v_1.5.9
1652168781720
3
3
d_meta_v_1.5.9
1652168781818
4
Fiddle
This is an example input:
{
"key1": "value",
"key2": [
{"key3": 5, "key4": "value1"},
{"key3": null, "key4": "value2"},
{"key3": 9, "key4": "value3"}
]
}
Example output:
{
"key1": "value",
"value1": 5,
"value2": null,
"value3": 9
}
This was generated with the following Python code:
new = {x['key4']: x['key3'] for x in old}
new['key1'] = old['key1']
I attempted to do this with jq, but this is far as I got:
[.[] | {k1: .key1, k2: .key2 | map({ (.key4): .key3}) | add}]
Which gives me
{
"key1": "value",
"key2": {
"value1": 5,
"value2": null,
"value3": 9
}
}
I'd say that what's missing is how to "merge" keys in the object with the top-level. How can I do that?
With your input JSON fixed to take keys/values in double-quotes, your expression could be written as
[ {key1}, (.key2[] | { (.key4): .key3} ) ] | add
I want to parse terraform.tfstate (where openstack provider is used), to return instance name and it's internal + floating IP (if assigned).
First select what we are interested in:
jq -r '.modules?[]|.resources[]?|select(.type == "openstack_compute_floatingip_v2", .type == "openstack_compute_instance_v2")' < terraform.tfstate
For simplicity, pre-parsed example with the above part (one FIP and one instance):
{
"type": "openstack_compute_floatingip_v2",
"depends_on": [
"openstack_networking_router_interface_v2.management"
],
"primary": {
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"attributes": {
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"instance_id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"pool": "public",
"region": "RegionOne"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
{
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_floatingip_v2.management",
"openstack_compute_secgroup_v2.ssh_only",
"openstack_networking_network_v2.management"
],
"primary": {
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"attributes": {
"access_ip_v4": "10.10.10.5",
"access_ip_v6": "",
"all_metadata.%": "1",
"all_metadata.habitat": "sup",
"availability_zone": "nova",
"flavor_id": "eb36e84e-17c1-42ab-b359-4380f6f524ae",
"flavor_name": "m1.large",
"force_delete": "false",
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"image_id": "c574aeed-e47c-4fb7-9da0-75550b76ee56",
"image_name": "ubuntu-16.04",
"key_pair": "vault-etcd_test_tf",
"metadata.%": "1",
"metadata.habitat": "sup",
"name": "ctl01",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.10.10.5",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "",
"network.0.mac": "02:c6:61:f9:ee:7e",
"network.0.name": "management",
"network.0.port": "",
"network.0.uuid": "f2468669-e321-4eb4-9ede-003e362a8988",
"region": "RegionOne",
"security_groups.#": "1",
"security_groups.1845949017": "vault-etcd_test_ssh_only",
"stop_before_destroy": "false"
},
"meta": {
"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
"create": 1800000000000,
"delete": 1800000000000,
"update": 1800000000000
}
},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
Required is to take from "type": "openstack_compute_floatingip_v2" replace .primary.attributes.address and .fixed_ip and from corresponding .instance_id the .name.
So, sth like:
{"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"}
Well, I came with an idea while using walk, but miss how to actually assign the proper value from corresponding instance id:
jq -r "$(cat floating.jq)" terraform.tfstate
floating.jq:
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
.modules?[]|.resources[]?|select(.type ==
"openstack_compute_floatingip_v2", .type ==
"openstack_compute_instance_v2")|
.primary|walk( if type == "object" and .attributes.address then
.attributes.instance_id |= "REFERRED VALUE HERE") else . end)
Let's assume the two related objects are in a file named two.json. Then one way to merge the information from both objects is using the -s command-line option, e.g.
jq -s '
(.[0].primary.attributes | {address, fixed_ip})
+ {name: .[1].primary.attributes.name}' two.json
Output
With your example input, the output would be:
{
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"
}