Can I output the ip and source id only when source id is duplicate it should out put all ip in one array if no duplicate ip can be output with corresponding source id
{"ip":"192.134.5.31","access_key":"223434354656767","source_id":"2e74a68a-2fef-443544-815d-87"}
{"ip":"172.23.54.4","saccess_key":"223434354656767","source_id":"2e74a68a-2fef-443544-815d-87"}
{"ip":"182,555,44.44","access_key":"223434354656767","source_id":"2e74a68a-2fef-443544-815d-222"}
I dont care about access key here also if this access key can be done with Jq would be great
unique_by(.ip) |{ip.source_id[]}
.ip| select(.source_id[])
You should group_by to group all the matching source_id.
Then you can create the desired output, for example:
group_by(.source_id)[] | { ip: map(.ip), source_id: (first.source_id) }
Will output:
{
"ip": [
"182,555,44.44"
],
"source_id": "2e74a68a-2fef-443544-815d-222"
}
{
"ip": [
"192.134.5.31",
"172.23.54.4"
],
"source_id": "2e74a68a-2fef-443544-815d-87"
}
Since
We group on source_id
We create an object for each group, containing
An map() from all the lower ip's and the source_id taken from the first object
Use the --slurp option to combine those objects in to an array:
jq --slurp 'group_by(.source_id)[] | { ip: map(.ip), source_id: (first.source_id) }'
JqPlay Demo
I'm not sure what your expected output is, but if you are trying to group IPs by source id like such:
{
"2e74a68a-2fef-443544-815d-222": [
"182,555,44.44"
],
"2e74a68a-2fef-443544-815d-87": [
"192.134.5.31",
"172.23.54.4"
]
}
Then you can group by your source_id and then transform the output:
group_by(.source_id) | map({(.[0].source_id): map(.ip)}) | add
Or by using a custom function:
def group(k): group_by(k) | map({key:first|k, value:.}) | from_entries;
group(.source_id) | map_values(map(.ip))
or
def group(k;v): group_by(k) | map({key:first|k, value:map(v)}) | from_entries;
group(.source_id;.ip)
Related
I am trying to build an audit/compliance report from IAM identity center. We need a list of groups and the respective group members. At current count we have 1,500+ users and 700+ Groups across 120 accounts in AWS.
There isn't an API command to spit this data out, so I'm putting a few commands together to extract the groups to files in Cloudshell. Then I need to cross-reference and throw everything into a CSV for filtering in Excel for the auditors.
Retrieve UserName and UserID - store in UserID.json
aws identitystore list-users --identity-store-id d-123456789| jq '.Users[] | {Name: .UserName, ID:.UserId}' > UsersIds.json
Retrieve Groups and GroupIDs - store in GroupsID.json
aws identitystore list-groups --identity-store-id d-123456789| jq '.Groups[] | {GroupName: .DisplayName, ID:.GroupId}' > GroupsID.json
Retrieve list of All Users per Group - store in GroupMembers.json
result=$(aws identitystore list-groups --identity-store-id d-123456789| jq -r '.Groups[].GroupId')
for val in $result; do
aws identitystore list-group-memberships --identity-store-id d-123456789--group-id $val | jq -r '.GroupMemberships[] | \
{GroupID: .GroupId, Member:User.Id} ' >> GroupMembers.json
done
Example output from UserIds.json:
{
"Name": "first.last#example.com",
"ID": "123456789-9876543210-ABCD-4321-1234"
}
{
"Name": "last.first#example.com",
"ID": "12345678-4321-1234-2233-9876543210"
}
Example output from GroupsID.json:
{
"GroupName": "sso-aws-zone-role-CloudCoreOps",
"ID": "123456789-55668877-1234-5522-2255-987654321"
}
{
"GroupName": "sso-aws-zone-role-CloudCoreRO",
"ID": "1234567890-11224455-2255-5522-1343-9876543210"
}
Example Output from GroupsMembers.json:
{
"GroupID": "123456789-55668877-1234-5522-2255-987654321",
"Member": "123456789-9876543210-ABCD-4321-1234"
}
{
"GroupID": "1234567890-11224455-2255-5522-1343-9876543210",
"Member": "12345678-4321-1234-2233-9876543210"
}
Now I just need to correlate and I have read you can use JQ like SED. So, that means I should be able to replace the key values in GroupMembers.json. First is to replace the GroupID with the correct GroupName matched from the GroupsID.json file and the Member with the User Name that matches the ID from the UserID.json file.
I think this can be done in a loop, but I want need to learn not only how to do this, but the best way.
It should be doable with INDEX and JOIN in a two-level nesting:
jq --slurpfile users UserIds.json --slurpfile groups GroupsID.json '
JOIN($groups | INDEX(.ID);
JOIN($users | INDEX(.ID); .; .Member; add);
.GroupID; add) | {Name, GroupName}
' GroupsMembers.json
{
"Name": "first.last#example.com",
"GroupName": "sso-aws-zone-role-CloudCoreOps"
}
{
"Name": "last.first#example.com",
"GroupName": "sso-aws-zone-role-CloudCoreRO"
}
I'm trying to get this output the device name "test"
My filter is .[] | [.deviceName] and it's returning error: (at :7): Cannot index array with string "deviceName"
{
"test": [
{
"deviceName": "test",
"monitoringServer": "server1"
}
]
}
Presumably you meant:
jq '.test[] | [.deviceName]'
or perhaps:
jq '.[][] | [.deviceName]'
but without knowing your requirements, it's hard to say. That's one of the reasons why the http://stackoverflow.com/help/mcve guidelines were formulated.
I have a JSON object where the relevant parts are of the form
{
"_meta": {
"hostvars": {
"name_1": {
"ansible_host": "10.0.0.1"
},
"name_2": {
"ansible_host": "10.0.0.2"
},
"name_3": {
"ansible_host": "10.0.0.3"
}
}
},
...
"nodes": {
"hosts": [
"name_1",
"name_2"
]
}
}
(the output of ansible-inventory --list, for reference).
I would like to use jq to produce a list of IPs of the nodes hosts by looking up the names in ._meta.hostvars. In the example, the output should be:
10.0.0.1
10.0.0.2
Note that 10.0.0.3 should not be included because name_3 is not in the .nodes.hosts list. So just doing jq -r '._meta.hostvars[].ansible_host' doesn't work.
I've tried jq '.nodes.hosts[] | ._meta.hostvars[.].ansible_host' but that fails because ._meta doesn't scan from the root after the pipe.
You can store the root in a variable before changing the context:
jq -r '. as $root | .nodes.hosts[] | $root._meta.hostvars[.].ansible_host'
But a better solution is to just inline the "hosts" query:
jq -r '._meta.hostvars[.nodes.hosts[]].ansible_host'
I need help in correcting jq test cases syntax. Following is output file & trying to test ID list with command below. Gives error index to string type.
[[ $(echo $output| jq -r '.output.value[] | select(.identity).id_list') == *"id2"* ]]
output = {
"resource_output": {
"value": {
"identity": [
{
"id_list": [
"/subscriptions/---/id1",
"/subscriptions/---/id2",
"/subscriptions/--/id3"
],
"principal_id": "",
"tenant_id": "",
"type": "managed"
}
]
}
}
Your query does not match the sample JSON, and you have not indicated what output you are expecting, but the following variation of your query illustrates how to use select and test with your data along the lines suggested by your attempt:
echo "$output" |
jq -r '.resource_output.identity[].id_list[] | select(test("id2"))'
Output:
/subscriptions/---/id2
I have an array of objects of various types which reference one another with UUIDs (a terraform.tfstate file). I'd like to select one value from one such object based on the appearance of a different value in another object, where the two objects are related by one of those UUIDs.
By way of example, I can do this:
$ jq '.modules[].resources[]
| select(.type == "openstack_compute_instance_v2" and
.primary.attributes.name == "jumpbox").primary.id' terraform.tfstate
"5edfe2bf-94df-49d5-8118-3e91fb52946b"
$ jq '.modules[].resources[]
| select(.type =="openstack_compute_floatingip_associate_v2" and
.primary.attributes.instance_id == "5edfe2bf-94df-49d5-8118-3e91fb52946b").primary.attributes.floating_ip' terraform.tfstate
"10.120.241.21"
Giving me the external floating IP of the 'jumpbox' VM based on its name.
I'd like to make that all one jq call. Is that possible?
This would be easier to answer if you provided more sample data but
working backwards from your commands (with some reformatting)
$ jq '
.modules[].resources[]
| select(.type == "openstack_compute_instance_v2" and .primary.attributes.name == "jumpbox")
| .primary.id
' terraform.tfstate
"5edfe2bf-94df-49d5-8118-3e91fb52946b"
$ jq '
.modules[].resources[]
| select(.type =="openstack_compute_floatingip_associate_v2" and .primary.attributes.instance_id == "5edfe2bf-94df-49d5-8118-3e91fb52946b")
| .primary.attributes.floating_ip
' terraform.tfstate
"10.120.241.21"
we can infer you have data which looks like
{
"modules": [
{
"resources": [
{
"type": "openstack_compute_instance_v2",
"primary": {
"id": "5edfe2bf-94df-49d5-8118-3e91fb52946b",
"attributes": {
"name": "jumpbox"
}
}
},
{
"type": "openstack_compute_floatingip_associate_v2",
"primary": {
"attributes": {
"instance_id": "5edfe2bf-94df-49d5-8118-3e91fb52946b",
"floating_ip": "10.120.241.21"
}
}
}
]
}
]
}
The following filter demonstrates a solution using functions, variables and parenthesis ():
def get_primary_id($name):
select(.type == "openstack_compute_instance_v2"
and .primary.attributes.name == $name)
| .primary.id
;
def get_floating_ip($id):
select(.type =="openstack_compute_floatingip_associate_v2"
and .primary.attributes.instance_id == $id)
| .primary.attributes.floating_ip
;
.modules[]
| ( .resources[] | get_primary_id("jumpbox") ) as $id
| ( .resources[] | get_floating_ip($id) ) as $fip
| ($id, $fip)
if this filter is in filter.jq and data.json contains the sample data above
then
$ jq -M -f filter.jq data.json
produces the output:
"5edfe2bf-94df-49d5-8118-3e91fb52946b"
"10.120.241.21"