Remove slash character in JSON response using jq - jq

Docker Engine API returns container name with / appended
{
"Id": "8dfafdbc3a40",
"Names": [
"/boring_feynman"
],
"Image": "ubuntu:latest",
"ImageID": "d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82",
"Command": "echo 1",
"Created": 1367854155,
"State": "Exited",
"Status": "Exit 0",
"Ports": [{
"PrivatePort": 2222,
"PublicPort": 3333,
"Type": "tcp"
}],
"Labels": {
"com.example.vendor": "Acme",
"com.example.license": "GPL",
"com.example.version": "1.0"
},
"SizeRw": 12288,
"SizeRootFs": 0,
"HostConfig": {
"NetworkMode": "default"
},
"NetworkSettings": {
"Networks": {}
},
"Mounts": [{
"Name": "fac362...80535",
"Source": "/data",
"Destination": "/data",
"Driver": "local",
"Mode": "ro,Z",
"RW": false,
"Propagation": ""
}]
}
I want to remove the slash so the response can be used as a table in JQ:
jq -r '(["Names","Image"] | (., map(length*"-"))), (.[] | [.Names, .Image]) | #tsv'
Currently, when I run the above, I get:
jq: error (at <stdin>:1): array (["/boring_feynman"]) is not valid in a csv row

The problem is not because of / in the .Names field, but in your expression. For filters like #csv or #tsv to work, the values need to be in a scalar format and in an array. But your expression .Name in of type array.
So basically you are passing this result to the #tsv function
[
[
"/boring_feynman"
],
"ubuntu:latest"
]
instead of
[
"/boring_feynman",
"ubuntu:latest"
]
So modifying your filter, you can do below for the JSON in question.
jq -r '(["Names","Image"] | (., map(length*"-"))), ([.Names[], .Image]) | #tsv'
or if you still want to remove the /, use gsub() function
jq -r '(["Names","Image"] | (., map(length*"-"))), ([ (.Names[] | gsub("^/";"")), .Image]) | #tsv'

Related

Group nested array objects to parent key in JQ

I have JSON coming from an external application, formatted like so:
{
"ticket_fields": [
{
"url": "https://example.com/1122334455.json",
"id": 1122334455,
"type": "tagger",
"custom_field_options": [
{
"id": 123456789,
"name": "I have a problem",
"raw_name": "I have a problem",
"value": "help_i_have_problem",
"default": false
},
{
"id": 456789123,
"name": "I have feedback",
"raw_name": "I have feedback",
"value": "help_i_have_feedback",
"default": false
},
]
}
{
"url": "https://example.com/6677889900.json",
"id": 6677889900,
"type": "tagger",
"custom_field_options": [
{
"id": 321654987,
"name": "United States,
"raw_name": "United States",
"value": "location_123_united_states",
"default": false
},
{
"id": 987456321,
"name": "Germany",
"raw_name": "Germany",
"value": "location_456_germany",
"default": false
}
]
}
]
}
The end goal is to be able to get the data into a TSV in the sense that each object in the custom_field_options array is grouped by the parent ID (ticket_fields.id), and then transposed such that each object would be represented on a single line, like so:
Ticket Field ID
Name
Value
1122334455
I have a problem
help_i_have_problem
1122334455
I have feedback
help_i_have_feedback
6677889900
United States
location_123_united_states
6677889900
Germany
location_456_germany
I have been able to export the data successfully to TSV already, but it reads per-line, and without preserving order, like so:
Using jq -r '.ticket_fields[] | select(.type=="tagger") | [.id, .custom_field_options[].name, .custom_field_options[].value] | #tsv'
Ticket Field ID
Name
Name
Value
Value
1122334455
I have a problem
I have feedback
help_i_have_problem
help_i_have_feedback
6677889900
United States
Germany
location_123_united_states
location_456_germany
Each of the custom_field_options arrays in production may consist of any number of objects (not limited to 2 each). But I seem to be stuck on how to appropriately group or map these objects to their parent ticket_fields.id and to transpose the data in a clean manner. The select(.type=="tagger") is mentioned in the query as there are multiple values for ticket_fields.type which need to be filtered out.
Based on another answer on here, I did try variants of jq -r '.ticket_fields[] | select(.type=="tagger") | map(.custom_field_options |= from_entries) | group_by(.custom_field_options.ticket_fields) | map(map( .custom_field_options |= to_entries))' without success. Any assistance would be greatly appreciated!
You need two nested iterations, one in each array. Save the value of .id in a variable to access it later.
jq -r '
.ticket_fields[] | select(.type=="tagger") | .id as $id
| .custom_field_options[] | [$id, .name, .value]
| #tsv
'

Extract nested properties from an array of objects

I have the following JSON file :
{
"filter": [
{
"id": "id_1",
"criteria": {
"from": "mail#domain1.com",
"subject": "subject_1"
},
"action": {
"addLabelIds": [
"Label_id_1"
],
"removeLabelIds": [
"INBOX",
"SPAM"
]
}
},
{
"id": "id_2",
"criteria": {
"from": "mail#domain2.com",
"subject": "subject_1"
},
"action": {
"addLabelIds": [
"Label_id_2"
],
"removeLabelIds": [
"INBOX",
"SPAM"
]
}
}
]
}
And I would like to extract emails values : mail#domain1.com and mail#domain2.com
I have tried this command:
jq --raw-output '.filter[] | select(.criteria.from | test("mail"; "i")) | .id'
But does not work, I get this error :
jq: error (at <stdin>:1206): null (null) cannot be matched, as it is
not a string exit status 5
Another point : how to display the value of "id" key, where "from" key value = mail#domain1.com ?
So in my file id = id_1
Do you have an idea ?
If you only need to extract the emails from .criteria.from then this filter is enough as far as I can tell:
jq --raw-output '.filter[].criteria.from' file.json
If some objects don't have a criteria object then you can filter out nulls with:
jq --raw-output '.filter[].criteria.from | select(. != null)' file.json
If you want to keep the emails equal to "mail#domain1.com":
jq --raw-output '.filter[].criteria.from | select(. == "mail#domain1.com")' file.json
If you want to keep the emails that start with "mail#":
jq --raw-output '.filter[].criteria.from | select(. != null) | select(startswith("mail#"))' file.json
I would like to extract emails values
There is a wide spectrum of possible answers, with these
amongst the least specific with respect to where in the JSON the email addresses occur:
.. | objects | .from | select(type=="string")
.. | strings | select(test("#([a-z0-9]+[.])+[a-z]+$"))

Unable to retrieve key:values, getting error --> jq: error (at <stdin>:0): Cannot index number with string

Unable to extract key:value pairs, tried indexing the block.
<nd.com.citrix.netscaler.json" -X GET https://abcd.com/nitro/v1/config/lbpersistentsessions?args=vserver:puppet-vip.ta10.sd | jq '[.[] ] | .[3] | [.srcip]'
Getting the following error :
jq: error (at <stdin>:0): Cannot index array with string "srcip"
I need to extract the key:values as srcip and destip (see below)
<ion/vnd.com.citrix.netscaler.json" -X GET https://abcd.com/nitro/v1/config/lbpersistentsessions?args=vserver:somevip | jq '[.[] ] | .[3]' | more
[
{
"vserver": "somevip",
"type": "1",
"typestring": "SOURCEIP",
"srcip": “1.1.1.1",
"srcipv6": "::/0",
"destip": "2.2.2.2",
"destipv6": "::/0",
"flags": false,
"destport": 0,
"vservername": “somevip”,
"timeout": "0",
"referencecount": "0",
"persistenceparam": "1.1.1.1"
},
I had to use [.3] to index as the original output is :
<-Type:application/vnd.com.citrix.netscaler.json" -X GET https://abcd.com/nitro/v1/config/lbpersistentsessions?args=vserver:somevip | jq '[.[] ]' | more
[
0,
"Done",
"NONE",
[
{
"vserver": "somevip",
"type": "1",
"typestring": "SOURCEIP",
"srcip": “1.1.1.1”,
"srcipv6": "::/0",
"destip": "2.2.2.2",
"destipv6": "::/0",
"flags": false,
"destport": 0,
"vservername": "somevip",
"timeout": "0",
"referencecount": "0",
"persistenceparam": "1.1.1.1"
},
{
"vserver": "somevip",
"type": "1",
"typestring": "SOURCEIP",
"srcip": "3.3.3.3”,
"srcipv6": "::/0",
"destip": "4.4.4.4”,
"destipv6": "::/0",
"flags": false,
"destport": 0,
"vservername": "somevip",
"timeout": "0",
"referencecount": "0",
"persistenceparam": "1.1.1.1"
},
Also, tried this way and get the error :
<GET https://abcd.com/nitro/v1/config/lbpersistentsessions?args=vserver:somevip | jq -r '.[] | select(.vserver == "somevip") | .srcip'
jq: error (at <stdin>:0): Cannot index number with string "vserver"
After fixing some minor problems with the full JSON shown in the Q, the invocation:
jq '.[3][].srcip' input.json
yields:
"1.1.1.1"
"3.3.3.3"
Notes
.[3][].scrip is just a shortened form of: .[3] | .[] | .srcip
In your initial query, [.[]] effectively does nothing because the input is an array.

Use jq to combine two arrays of objects on a certain key

I am trying to use jq to solve this problem.
Suppose I have the following object
{
"listA": [
{
"id": "12345",
"code": "001"
}
]
"listB": [
{
"id": "12345",
"prop": "AABBCC"
}
]
}
In reality my two lists are longer, but the id isn't repeated within each list.
How may I combine the two lists into a single list where each item is an object with the non-id properties for the given id are collected into a single object?
For example, from the object above, I'd like the following:
{
"listC" : [
{
"id": "12345",
"code": "001",
"prop": "AABBCC"
}
]
}
A simple way would be to concatenate the arrays, group the elements by id and map each group into a single object using add;
jq '.listA+.listB | group_by(.id) | map(add)' test.json
If there may be more than two arrays you need to merge in the file, you could instead use flatten to concatenate all of them.
Test case below
# cat test.json
{
"listA": [
{ "id": "12345", "code": "001" },
{ "id": "12346", "code": "002" }
],
"listB": [
{ "id": "12345", "prop": "AABBCC" }
]
}
# jq 'flatten | group_by(.id) | map(add)' test.json
# or
# jq '.listA+.listB | group_by(.id) | map(add)' test.json
[
{
"id": "12345",
"code": "001",
"prop": "AABBCC"
},
{
"id": "12346",
"code": "002"
}
]
Using group_by entails a sort, which is unnecessary, so if efficiency is a concern, then an alternative approach such as the following should be considered:
INDEX(.listA[]; .id) as $one
| INDEX(.listB[]; .id) as $two
| reduce ($one|keys_unsorted[]) as $k ($two; .[$k] += $one[$k])
| {listC: [.[]] }

Jq getting output from nested hash

Hi I am trying to parse below hash with jq
{
"name": "a",
"data": [
{
"sensitive": false,
"type": "string",
"value": "mykeypair"
},
{
"sensitive": false,
"type": "int",
"value": 123
}
]
}
and get output like
a,string,mykeypair
a,int,123
I am able to get output like this
a,string,mykeypair
a,int,mykeypair
a,string,123
a,int,123
jq solution:
jq -r '.name as $n | .data[] | [$n, .type, .value] | #csv' file.json
The output:
"a","string","mykeypair"
"a","int",123
If it's mandatory to output unquoted values:
jq -r '.name as $n | .data[] | [$n, .type, "\(.value)"] | join(",")' file.json
The output:
a,string,mykeypair
a,int,123

Resources