I'm trying to use jq to process JSON from the VMware Ansible inventory module so I can produce a list of objects (VMs) that don't have ansible_host defined.
The closest I've been able to get is:
{
"config.cpuHotAddEnabled": true,
"config.cpuHotRemoveEnabled": false,
"config.hardware.numCPU": 1,
"config.instanceUuid": "500e4e98-50ec-a3a7-9d45-b0ac36c2d192",
"config.name": "tu-openldap-01",
"config.template": false,
"guest.guestId": "rhel6_64Guest",
"guest.guestState": "notRunning",
"guest.hostName": "tu-openldap-01",
"guest.ipAddress": null,
"name": "tu-openldap-01",
"runtime.maxMemoryUsage": 2048
}
{
"config.cpuHotAddEnabled": true,
"config.cpuHotRemoveEnabled": false,
"config.hardware.numCPU": 1,
"config.instanceUuid": "500efaa5-baac-163b-65c0-7ed2a19f1d7d",
"config.name": "tu1vcm7tst2001",
"config.template": false,
"guest.guestId": "rhel7_64Guest",
"guest.guestState": "running",
"guest.hostName": "rhel7-template",
"guest.ipAddress": null,
"name": "tu1vcm7tst2001",
"runtime.maxMemoryUsage": 4096
}
using the following:
jq '._meta.hostvars[] | select(.ansible_host | not)' prod-inventory_201905070920.json
This is almost where I want it, but the problem is how do I print these plus the key for the object itself?
If I do:
jq '._meta.hostvars | select(.ansible_host | not)' prod-inventory_201905070920.json
I get these:
"tw1pttest1001_420e92f4-453e-1267-4331-d6253d771882": {
"ansible_host": "<omitted>",
"config.cpuHotAddEnabled": true,
"config.cpuHotRemoveEnabled": false,
"config.hardware.numCPU": 2,
"config.instanceUuid": "500ef630-16c1-cb91-be9c-e9e667b551d9",
"config.name": "tw1pttest1001",
"config.template": false,
"guest.guestId": "windows9Server64Guest",
"guest.guestState": "running",
"guest.hostName": "<omitted>",
"guest.ipAddress": "<omitted>",
"name": "tw1pttest1001",
"runtime.maxMemoryUsage": 49152
},
"tw1swsrm1001_420e18d2-0c96-0df5-e6c7-1ff8fc070cdb": {
"ansible_host": "<omitted>",
"config.cpuHotAddEnabled": true,
"config.cpuHotRemoveEnabled": false,
"config.hardware.numCPU": 4,
"config.instanceUuid": "500e231d-1eda-4e66-3f4a-8c68392a70b5",
"config.name": "tw1swsrm1001",
"config.template": false,
"guest.guestId": "windows9Server64Guest",
"guest.guestState": "running",
"guest.hostName": "<omitted>",
"guest.ipAddress": "<omitted>",
"name": "tw1swsrm1001",
"runtime.maxMemoryUsage": 16384
},
Any suggestions? I feel like it's something simple that I'm missing.
Assuming you are searching for items found in the ._meta.hostvars object, you can filter the objects by key/value by using something like with_entries/1.
$ jq '._meta.hostvars | with_entries(select(.value.ansible_host | not))
' prod-inventory_201905070920.json
This will in effect take the hostvars object and only keep properties that match that condition (does not have an ansible_host property value).
Related
I am trying to sort following json response based on "startTime" and also want to filter based on "name" and fetch only "dataCenter" of matched record. Can you please help with jq function for doing it?
I tried something like this jq '.[]|= sort_by(.startTime)' but it doesnt return correct result.
[
{
"name": "JPCSKELT",
"dataCenter": "mvsADM",
"orderId": "G9HC8",
"scheduleTable": "FD33515",
"nodeGroup": null,
"controlmApp": "P/C-DEVELOPMENT-LRSP",
"groupName": "SCMTEST",
"assignmentGroup": "HOST_CONFIG_MGMT",
"owner": "PC00000",
"description": null,
"startTime": "2021-11-11 17:45:48.0",
"endTime": "2021-11-11 17:45:51.0",
"successCount": 1,
"failureCount": 0,
"dailyRunCount": 0,
"scriptName": "JPCSKELT"
},
{
"name": "JPCSKELT",
"dataCenter": "mvsADM",
"orderId": "FWX98",
"scheduleTable": "JPCS1005",
"nodeGroup": null,
"controlmApp": "P/C-DEVELOPMENT-LRSP",
"groupName": "SCMTEST",
"assignmentGroup": "HOST_CONFIG_MGMT",
"owner": "PC00000",
"description": null,
"startTime": "2021-07-13 10:49:47.0",
"endTime": "2021-07-13 10:49:49.0",
"successCount": 1,
"failureCount": 0,
"dailyRunCount": 0,
"scriptName": "JPCSKELT"
},
{
"name": "JPCSKELT",
"dataCenter": "mvsADM",
"orderId": "FWX98",
"scheduleTable": "JPCS1005",
"nodeGroup": null,
"controlmApp": "P/C-DEVELOPMENT-LRSP",
"groupName": "SCMTEST",
"assignmentGroup": "HOST_CONFIG_MGMT",
"owner": "PC00000",
"description": null,
"startTime": "2021-10-13 10:49:47.0",
"endTime": "2021-10-13 10:49:49.0",
"successCount": 1,
"failureCount": 0,
"dailyRunCount": 0,
"scriptName": "JPCSKELT"
}
]
You can use the following expression to sort the input -
sort_by(.startTime | sub("(?<time>.*)\\..*"; "\(.time)") | strptime("%Y-%m-%d %H:%M:%S") | mktime)
The sub("(?<time>.*)\\..*"; "\(.time)") expression removes the trailing decimal fraction.
I assume you can use the result from the above query to perform desired filtering.
Welcome. From what I'm guessing you're asking, you want to supply a value to filter the records on using the name property, sort the results by the startTime property and then just output the value of the dataCenter property for those records. How about this:
jq --arg name JPCSKELT '
map(select(.name==$name))|sort_by(.startTime)[].dataCenter
' data.json
Based on your sample data, this produces:
"mvsADM"
"mvsADM"
"mvsADM"
So I'm wondering if this is what you're really asking?
I have JSON coming from an external application, formatted like so:
{
"ticket_fields": [
{
"url": "https://example.com/1122334455.json",
"id": 1122334455,
"type": "tagger",
"custom_field_options": [
{
"id": 123456789,
"name": "I have a problem",
"raw_name": "I have a problem",
"value": "help_i_have_problem",
"default": false
},
{
"id": 456789123,
"name": "I have feedback",
"raw_name": "I have feedback",
"value": "help_i_have_feedback",
"default": false
},
]
}
{
"url": "https://example.com/6677889900.json",
"id": 6677889900,
"type": "tagger",
"custom_field_options": [
{
"id": 321654987,
"name": "United States,
"raw_name": "United States",
"value": "location_123_united_states",
"default": false
},
{
"id": 987456321,
"name": "Germany",
"raw_name": "Germany",
"value": "location_456_germany",
"default": false
}
]
}
]
}
The end goal is to be able to get the data into a TSV in the sense that each object in the custom_field_options array is grouped by the parent ID (ticket_fields.id), and then transposed such that each object would be represented on a single line, like so:
Ticket Field ID
Name
Value
1122334455
I have a problem
help_i_have_problem
1122334455
I have feedback
help_i_have_feedback
6677889900
United States
location_123_united_states
6677889900
Germany
location_456_germany
I have been able to export the data successfully to TSV already, but it reads per-line, and without preserving order, like so:
Using jq -r '.ticket_fields[] | select(.type=="tagger") | [.id, .custom_field_options[].name, .custom_field_options[].value] | #tsv'
Ticket Field ID
Name
Name
Value
Value
1122334455
I have a problem
I have feedback
help_i_have_problem
help_i_have_feedback
6677889900
United States
Germany
location_123_united_states
location_456_germany
Each of the custom_field_options arrays in production may consist of any number of objects (not limited to 2 each). But I seem to be stuck on how to appropriately group or map these objects to their parent ticket_fields.id and to transpose the data in a clean manner. The select(.type=="tagger") is mentioned in the query as there are multiple values for ticket_fields.type which need to be filtered out.
Based on another answer on here, I did try variants of jq -r '.ticket_fields[] | select(.type=="tagger") | map(.custom_field_options |= from_entries) | group_by(.custom_field_options.ticket_fields) | map(map( .custom_field_options |= to_entries))' without success. Any assistance would be greatly appreciated!
You need two nested iterations, one in each array. Save the value of .id in a variable to access it later.
jq -r '
.ticket_fields[] | select(.type=="tagger") | .id as $id
| .custom_field_options[] | [$id, .name, .value]
| #tsv
'
Assuming the below input, how can I detect the presence of duplicates in the replicas list? (replicas":[5,5,6]")
{"version":1,
"partitions":
[{"topic":"mytopic1","partition":3,"replicas":[4,5],"log_dirs":["any","any"]},
{"topic":"mytopic1","partition":1,"replicas":[5,5,6],"log_dirs":["any","any"]},
{"topic":"mytopic2","partition":2,"replicas":[6,5],"log_dirs":["any","any"]}]
}
This one will give you an array of just the partitions with duplicates in the replicas field:
jq '[.partitions[] | select((.replicas | length) != (.replicas | unique | length))]' input.json
Pretty-printed example output:
[
{
"topic": "mytopic1",
"partition": 1,
"replicas": [
5,
5,
6
],
"log_dirs": [
"any",
"any"
]
}
]
Docker Engine API returns container name with / appended
{
"Id": "8dfafdbc3a40",
"Names": [
"/boring_feynman"
],
"Image": "ubuntu:latest",
"ImageID": "d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82",
"Command": "echo 1",
"Created": 1367854155,
"State": "Exited",
"Status": "Exit 0",
"Ports": [{
"PrivatePort": 2222,
"PublicPort": 3333,
"Type": "tcp"
}],
"Labels": {
"com.example.vendor": "Acme",
"com.example.license": "GPL",
"com.example.version": "1.0"
},
"SizeRw": 12288,
"SizeRootFs": 0,
"HostConfig": {
"NetworkMode": "default"
},
"NetworkSettings": {
"Networks": {}
},
"Mounts": [{
"Name": "fac362...80535",
"Source": "/data",
"Destination": "/data",
"Driver": "local",
"Mode": "ro,Z",
"RW": false,
"Propagation": ""
}]
}
I want to remove the slash so the response can be used as a table in JQ:
jq -r '(["Names","Image"] | (., map(length*"-"))), (.[] | [.Names, .Image]) | #tsv'
Currently, when I run the above, I get:
jq: error (at <stdin>:1): array (["/boring_feynman"]) is not valid in a csv row
The problem is not because of / in the .Names field, but in your expression. For filters like #csv or #tsv to work, the values need to be in a scalar format and in an array. But your expression .Name in of type array.
So basically you are passing this result to the #tsv function
[
[
"/boring_feynman"
],
"ubuntu:latest"
]
instead of
[
"/boring_feynman",
"ubuntu:latest"
]
So modifying your filter, you can do below for the JSON in question.
jq -r '(["Names","Image"] | (., map(length*"-"))), ([.Names[], .Image]) | #tsv'
or if you still want to remove the /, use gsub() function
jq -r '(["Names","Image"] | (., map(length*"-"))), ([ (.Names[] | gsub("^/";"")), .Image]) | #tsv'
I'm using js tool to parse some JSONs/strings. My minimal example is the following command:
echo '"foo foo"' | jq 'match("(foo)"; "g")'
Which results in the following output:
{
"offset": 0,
"length": 3,
"string": "foo",
"captures": [
{
"offset": 0,
"length": 3,
"string": "foo",
"name": null
}
]
}
{
"offset": 4,
"length": 3,
"string": "foo",
"captures": [
{
"offset": 4,
"length": 3,
"string": "foo",
"name": null
}
]
}
I want my final output for this example to be:
"foo,foo"
But in this case I get two separate objects instead of an array or similar that I could call implode on. I guess either the API isn't made for my UC or my understanding of it is very wrong. Please, advise.
The following script takes the string value from each of the separate objects with .string, wraps them in an array [...] and then joins the members of the array with commas using join.
I modified the regex because you didn't actually need a capture group for the given use case, but if you wanted to access the capture groups you could do .captures[].string instead of .string.
echo '"foo foo"' | jq '[match("foo"; "g").string] | join(",")'