Escaping period character when enumerating JSON object - azure-data-explorer

I'm trying to extract some information using Azure Data Explorer from the Qualys vulnerability scanner logs.
I have got the below KQL, and it all works fine except for line 9. I'm trying to extract information from the returned json object, but the key name has a period in it i.e. 2.0 and I can't work out how to escape it so that I can still use the JSON dot notation.
securityresources | where type =~ "microsoft.security/assessments/subassessments"
| extend assessmentKey=extract(#"(?i)providers/Microsoft.Security/assessments/([^/]*)", 1, id), QID=tostring(properties.id), parentResourceId= extract("(.+)/providers/Microsoft.Security", 1, id)
| extend resourceId = tostring(properties.resourceDetails.id)
| where properties.additionalData.assessedResourceType =~ "ServerVulnerability" or properties.additionalData.assessedResourceType =~ "GeneralVulnerability"
| where properties.additionalData.source =~ "Built-in Qualys vulnerability assessment"
| extend vulnerabilityName=tostring(properties.displayName),
vulnerabilityType = tostring(properties.additionalData.assessedResourceType),
virtualMachineName=split(properties.resourceDetails.id, "/")[-1],
cvss2Score = tostring(properties.additionalData.cvss.\(2.0).base)
| project QID, vulnerabilityName, virtualMachineName, vulnerabilityType, cvss2Score
JSON structure:
{
"2.0": {
"base": 7.6
},
"3.0": {
"base": 7.5
}
}

please see: Dynamic object accessors.
for example:
print properties = dynamic({
"additionalData":{
"cvss":{
"2.0": {
"base": 7.6
},
"3.0": {
"base": 7.5
}
}
}
})
| project base = todouble(properties.additionalData.cvss['2.0'].base)
base
7.6

Related

Jq parse duplicate json file using jqplay.org

Can I output the ip and source id only when source id is duplicate it should out put all ip in one array if no duplicate ip can be output with corresponding source id
{"ip":"192.134.5.31","access_key":"223434354656767","source_id":"2e74a68a-2fef-443544-815d-87"}
{"ip":"172.23.54.4","saccess_key":"223434354656767","source_id":"2e74a68a-2fef-443544-815d-87"}
{"ip":"182,555,44.44","access_key":"223434354656767","source_id":"2e74a68a-2fef-443544-815d-222"}
I dont care about access key here also if this access key can be done with Jq would be great
unique_by(.ip) |{ip.source_id[]}
.ip| select(.source_id[])
You should group_by to group all the matching source_id.
Then you can create the desired output, for example:
group_by(.source_id)[] | { ip: map(.ip), source_id: (first.source_id) }
Will output:
{
"ip": [
"182,555,44.44"
],
"source_id": "2e74a68a-2fef-443544-815d-222"
}
{
"ip": [
"192.134.5.31",
"172.23.54.4"
],
"source_id": "2e74a68a-2fef-443544-815d-87"
}
Since
We group on source_id
We create an object for each group, containing
An map() from all the lower ip's and the source_id taken from the first object
Use the --slurp option to combine those objects in to an array:
jq --slurp 'group_by(.source_id)[] | { ip: map(.ip), source_id: (first.source_id) }'
JqPlay Demo
I'm not sure what your expected output is, but if you are trying to group IPs by source id like such:
{
"2e74a68a-2fef-443544-815d-222": [
"182,555,44.44"
],
"2e74a68a-2fef-443544-815d-87": [
"192.134.5.31",
"172.23.54.4"
]
}
Then you can group by your source_id and then transform the output:
group_by(.source_id) | map({(.[0].source_id): map(.ip)}) | add
Or by using a custom function:
def group(k): group_by(k) | map({key:first|k, value:.}) | from_entries;
group(.source_id) | map_values(map(.ip))
or
def group(k;v): group_by(k) | map({key:first|k, value:map(v)}) | from_entries;
group(.source_id;.ip)

How to extrapolate values in one AWS CLI output with values from two separate CLI outputs as input files?

I am trying to build an audit/compliance report from IAM identity center. We need a list of groups and the respective group members. At current count we have 1,500+ users and 700+ Groups across 120 accounts in AWS.
There isn't an API command to spit this data out, so I'm putting a few commands together to extract the groups to files in Cloudshell. Then I need to cross-reference and throw everything into a CSV for filtering in Excel for the auditors.
Retrieve UserName and UserID - store in UserID.json
aws identitystore list-users --identity-store-id d-123456789| jq '.Users[] | {Name: .UserName, ID:.UserId}' > UsersIds.json
Retrieve Groups and GroupIDs - store in GroupsID.json
aws identitystore list-groups --identity-store-id d-123456789| jq '.Groups[] | {GroupName: .DisplayName, ID:.GroupId}' > GroupsID.json
Retrieve list of All Users per Group - store in GroupMembers.json
result=$(aws identitystore list-groups --identity-store-id d-123456789| jq -r '.Groups[].GroupId')
for val in $result; do
aws identitystore list-group-memberships --identity-store-id d-123456789--group-id $val | jq -r '.GroupMemberships[] | \
{GroupID: .GroupId, Member:User.Id} ' >> GroupMembers.json
done
Example output from UserIds.json:
{
"Name": "first.last#example.com",
"ID": "123456789-9876543210-ABCD-4321-1234"
}
{
"Name": "last.first#example.com",
"ID": "12345678-4321-1234-2233-9876543210"
}
Example output from GroupsID.json:
{
"GroupName": "sso-aws-zone-role-CloudCoreOps",
"ID": "123456789-55668877-1234-5522-2255-987654321"
}
{
"GroupName": "sso-aws-zone-role-CloudCoreRO",
"ID": "1234567890-11224455-2255-5522-1343-9876543210"
}
Example Output from GroupsMembers.json:
{
"GroupID": "123456789-55668877-1234-5522-2255-987654321",
"Member": "123456789-9876543210-ABCD-4321-1234"
}
{
"GroupID": "1234567890-11224455-2255-5522-1343-9876543210",
"Member": "12345678-4321-1234-2233-9876543210"
}
Now I just need to correlate and I have read you can use JQ like SED. So, that means I should be able to replace the key values in GroupMembers.json. First is to replace the GroupID with the correct GroupName matched from the GroupsID.json file and the Member with the User Name that matches the ID from the UserID.json file.
I think this can be done in a loop, but I want need to learn not only how to do this, but the best way.
It should be doable with INDEX and JOIN in a two-level nesting:
jq --slurpfile users UserIds.json --slurpfile groups GroupsID.json '
JOIN($groups | INDEX(.ID);
JOIN($users | INDEX(.ID); .; .Member; add);
.GroupID; add) | {Name, GroupName}
' GroupsMembers.json
{
"Name": "first.last#example.com",
"GroupName": "sso-aws-zone-role-CloudCoreOps"
}
{
"Name": "last.first#example.com",
"GroupName": "sso-aws-zone-role-CloudCoreRO"
}

Print the key and a subset of fields if a field is not a specific value

I am new to jq and can't seem to quite get the syntax right for what I want to do. I am executing a command and piping its JSON output into jq. The structure looks like this:
{
"timestamp": 1658186185,
"nodes": {
"x3006c0s13b1n0": {
"Mom": "x3006c0s13b1n0.hsn.cm",
"Port": 15002,
"state": "free",
"pcpus": 64,
"resources_available": {
"arch": "linux",
"gputype": "A100",
"host": "x3006c0s13b1n0",
"mem": "527672488kb",
"ncpus": 64,
"ngpus": 4,
"system": "polaris",
"tier0": "x3006-g1",
"tier1": "g1",
"vnode": "x3006c0s13b1n0"
},
"resources_assigned": {},
"comment": "CHC- Offlined due to node health check failure",
"resv_enable": "True",
"sharing": "default_shared",
"license": "l",
"last_state_change_time": 1658175652,
"last_used_time": 1658175652
},
And so on with a record for each node. In psuedocode, what I want to do is this:
if state is not free then display nodename : {comment = "Why is the node down"}
The nodename is the key, but could be extracted from a field inside the record. However, for future reference, I would like to understand how to get the key. I figured out (I think) that you can't use == on strings, but instead have to use the regex functions.
This gives me the if state is not free part:
<stdin> | jq '.nodes[] | .state | test("free") | not'
This gives me an object with the Mom (which includes the key) and the comment:
jq '.nodes[] | {Mom: .Mom, comment: .comment}'
The question is how do I put all that together? And as for the keys, this gives me a list of the keys: jq '.nodes | keys' but that uses the non-array version of nodes.
One way without touching the keys would be to only select those array items that match the condition, and map the remaining items' value to the comment itself using map_values:
jq '.nodes | map_values(select(.state != "free").comment)'
{
"x3006c0s13b1n0": "CHC- Offlined due to node health check failure"
}
Keeping the whole comments object, which is closer to your desired output, would be similar:
jq '.nodes | map_values(select(.state != "free") | {comment})'
{
"x3006c0s13b1n0": {
"comment": "CHC- Offlined due to node health check failure"
}
}
Accessing the keys directly is still possible though. You may want to have a look at keys, keys_unsorted or to_entries.

Trying to get the correct output from JQ

I'm trying to get this output the device name "test"
My filter is .[] | [.deviceName] and it's returning error: (at :7): Cannot index array with string "deviceName"
{
"test": [
{
"deviceName": "test",
"monitoringServer": "server1"
}
]
}
Presumably you meant:
jq '.test[] | [.deviceName]'
or perhaps:
jq '.[][] | [.deviceName]'
but without knowing your requirements, it's hard to say. That's one of the reasons why the http://stackoverflow.com/help/mcve guidelines were formulated.

Using jq to suffix a string to a field in an object not containing it

I have a json as follows:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"batchPools": {
"value": [
{
"networkConfiguration": {
"subnetId": "/subscriptions/xxxx/resourceGroups/xxx/providers/Microsoft.Network/virtualNetworks/xxxxx/subnets/sample-name-batch",
"subnetAddressPrefix": ""
}
},
{
"networkConfiguration": {
"subnetId": "/subscriptions/xxxx/resourceGroups/xxx/providers/Microsoft.Network/virtualNetworks/xxxxx/subnets/sample-name",
"subnetAddressPrefix": ""
}
}
]
}
}
}
I need to check if any of the networkConfiguration.subnetID under the array value contains the string batch. If yes then nothing needs to be done. Else, append -batch to the existing value. In this case, only the second networkConfiguration.subnetID of the array should be updated.
I tried the following:
(.parameters.batchPools.value[] | select(.networkConfiguration.subnetId | contains("-batch") | not) | .networkConfiguration.subnetId) |= (.networkConfiguration.subnetId+"-batch")
I get the following error:
jq: error (at <stdin>:38): Cannot index string with string "networkConfiguration" exit status 5
I tried this:
(.parameters.batchPools.value[] | select(.networkConfiguration.subnetId | contains("batch") | not) | .networkConfiguration.subnetId) |= "someValue"
This worked fine and replaced the entire subnetId with someValue. I am not able to figure out why the previous command is not working out. Please help.
You don't need to use the whole path of subnetId again. You've already selected that node from the pipeline before. Just use the += append operator to suffix the required string
( .parameters.batchPools.value[] |
select(.networkConfiguration.subnetId | contains("batch") | not) |
.networkConfiguration.subnetId ) += "-batch"
jqplay - Online demo

Resources