How do I iterate through array in Kusto? - azure-data-explorer

I need to iterate through managed data disks in Azure Resource Graph Explorer (https://preview.portal.azure.com/). My query is below but it returns JSON array, I need to extract name of disk and type of storage account which is being used (sample JSON return is below). So I'd like to see on screen grouping by machine name, disk name and then storage account type. My current query is below but obviously it does not work due to return of JSON
where type =~ 'Microsoft.Compute/virtualmachines' |
extend disks = properties.storageProfile.dataDisks |
project name, disks
Same JSON output
[
{
"name": "COMP02_DDisk1",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk1",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 0,
"writeAcceleratorEnabled": false
},
{
"name": "COMP02_DDisk2",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk2",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 1,
"writeAcceleratorEnabled": false
}
]

in such cases, it's usually helpful to use mv-expand to expand the array and then apply the dynamic-property accessors foreach record.
https://learn.microsoft.com/en-us/azure/kusto/query/mvexpandoperator
example:
print d = dynamic([
{
"name": "COMP02_DDisk1",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk1",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 0,
"writeAcceleratorEnabled": false
},
{
"name": "COMP02_DDisk2",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk2",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 1,
"writeAcceleratorEnabled": false
}
])
| mv-expand d
| project d.name, d.managedDisk.storageAccountType
which will output:
| d_name | d_managedDisk_storageAccountType |
|---------------|----------------------------------|
| COMP02_DDisk1 | Premium_LRS |
| COMP02_DDisk2 | Premium_LRS |

Hope you are doing good .
You can try this way also, First i found networksecuritygroups from entire collection and later filtered defaultSecurityRules which is again an array.
After collecting it using mvexpand in local variable rules you should be able to fetch the desired after applying.
where type =~ "microsoft.network/networksecuritygroups"
| mvexpand rules = properties.defaultSecurityRules
| where rules.properties.destinationAddressPrefix =~ "*"
You can also refer given below link and wish it will help you also.

Related

jq array filter for nested array elements

I am trying to add a new user in below json which matches group NP01-RW. i am able to do without NP01-RW but not able to select users under NP01-RW and return updated json.
{
"id": 181,
"guid": "c9b7dbde-63de-42cc-9840-1b4a06e13364",
"isEnabled": true,
"version": 17,
"service": "Np-Hue",
"name": "DATASCIENCE-CUROPT-RO",
"policyType": 0,
"policyPriority": 0,
"isAuditEnabled": true,
"resources": {
"database": {
"values": [
"hive_cur_acct_1dev",
"hive_cur_acct_1eng",
"hive_cur_acct_1rwy",
"hive_cur_acct_1stg",
"hive_opt_acct_1dev",
"hive_opt_acct_1eng",
"hive_opt_acct_1stg",
"hive_opt_acct_1rwy"
],
"isExcludes": false,
"isRecursive": false
},
"column": {
"values": [
"*"
],
"isExcludes": false,
"isRecursive": false
},
"table": {
"values": [
"*"
],
"isExcludes": false,
"isRecursive": false
}
},
"policyItems": [
{
"accesses": [
{
"type": "select",
"isAllowed": true
},
{
"type": "update",
"isAllowed": true
},
{
"type": "create",
"isAllowed": true
},
{
"type": "drop",
"isAllowed": true
},
{
"type": "alter",
"isAllowed": true
},
{
"type": "index",
"isAllowed": true
},
{
"type": "lock",
"isAllowed": true
},
{
"type": "all",
"isAllowed": true
},
{
"type": "read",
"isAllowed": true
},
{
"type": "write",
"isAllowed": true
}
],
"users": [
"user1",
"user2",
"user3"
],
"groups": [
"NP01-RW"
],
"conditions": [],
"delegateAdmin": false
},
{
"accesses": [
{
"type": "select",
"isAllowed": true
}
],
"users": [
"user1"
],
"groups": [
"NP01-RO"
],
"conditions": [],
"delegateAdmin": false
}
],
"denyPolicyItems": [],
"allowExceptions": [],
"denyExceptions": [],
"dataMaskPolicyItems": [],
"rowFilterPolicyItems": [],
"options": {},
"validitySchedules": [],
"policyLabels": [
"DATASCIENCE-CurOpt-RO_NP01"
]
}
below is what i have tried but it returns part of the JSON matching NP01-RW and not full JSON
jq --arg username "$sync_userName" '.policyItems[] | select(.groups[] | IN("NP01-RO")).users += [$username]' > ${sync_policyName}.json
Operator precedence in jq is not always intuitive. Your program is parsed as:
.policyItems[] | (select(.groups[] | IN("NP01-RO")).users += [$username])
Which first streams all policyItems and only then changes them, leaving you with policyItems only in the output.
You need to make sure that the stream selects the correct values, which you can then assign:
(.policyItems[] | select(.groups[] | IN("NP01-RO")).users) += [$username]
This will do the assignment, but still return the full input (.).

Extracting values from JSON column using KQL (Azure Data Explorer)

Can you please tell me how to extract values of category, enabled and categoryGroup from the below JSON column in KQL(Azure Data Explorer).
Below JSON value is exactly what I see in the column called "Logs". I see that the Column Logs is defined as string datatype in table
AzLogsCoverage
| extend Logs = case(isnull(Logs) or isempty(Logs), 'N/A', Logs)
| where Logs <> 'N/A'
| project Logs
| extend LogsCategory = parse_json(Logs).category
[
{
"category": "Administrative",
"enabled": true,
"categoryGroup": null
},
{
"category": "Security",
"enabled": false,
"categoryGroup": null
},
{
"category": "ServiceHealth",
"enabled": false,
"categoryGroup": null
},
{
"category": "Alert",
"enabled": false,
"categoryGroup": null
},
{
"category": "Recommendation",
"enabled": false,
"categoryGroup": null
},
{
"category": "Policy",
"enabled": false,
"categoryGroup": null
},
{
"category": "Autoscale",
"enabled": false,
"categoryGroup": null
},
{
"category": "ResourceHealth",
"enabled": false,
"categoryGroup": null
}
]
if the input is of type string, you first need to invoke parse_json() on it, to make it of type dynamic.
Then, you can use mv-expand/mv-apply to expand elements in the array, and then you can explicitly project properties of interest for each element.
for example:
print input = ```[
{
"category": "Administrative",
"enabled": true,
"categoryGroup": null
},
{
"category": "Security",
"enabled": false,
"categoryGroup": null
},
{
"category": "ServiceHealth",
"enabled": false,
"categoryGroup": null
},
{
"category": "Alert",
"enabled": false,
"categoryGroup": null
},
{
"category": "Recommendation",
"enabled": false,
"categoryGroup": null
},
{
"category": "Policy",
"enabled": false,
"categoryGroup": null
},
{
"category": "Autoscale",
"enabled": false,
"categoryGroup": null
},
{
"category": "ResourceHealth",
"enabled": false,
"categoryGroup": null
}
]```
| extend d = parse_json(input)
| mv-apply d on (
project Category = tostring(d.category),
Enabled = tobool(d.enabled),
CategoryGroup = tostring(d.categoryGroup)
)
| project-away input
Category
Enabled
CategoryGroup
Administrative
True
Security
False
ServiceHealth
False
Alert
False
Recommendation
False
Policy
False
Autoscale
False
ResourceHealth
False
AzLogsCoverage
| extend Logs = case(isnull(Logs) or isempty(Logs), 'N/A', Logs)
| extend Metrics = case(isnull(Metrics) or isempty(Metrics), 'N/A', Metrics)
| where Logs <> 'N/A'
| extend LogsDynamic = todynamic(Logs)
| extend MetricsDynamics = todynamic(Metrics)
| mv-expand LogsDynamic, MetricsDynamics
| project SubscriptionId, ResourceId, ResourceName, ResourceType, DiagnosticSettingStatus, DiagnosticSettingId, DiagnosticSettingName, DiagnosticSettingType, LAworkspaceId, LAworkspaceRetentionPeriod,
LogsDynamic.category,LogsDynamic.enabled,LogsDynamic.categoryGroup, MetricsDynamics.category, MetricsDynamics.categoryGroup, MetricsDynamics.enabled, MetricsDynamics.retentionPolicy.enabled , MetricsDynamics.retentionPolicy.days

Parsing on jq with_entries

I have the following data
Command output:
| jq '.rrsets[]'
{
"comments": [],
"name": "name1.",
"records": [
{
"content": "10.10.10.10",
"disabled": false
}
],
"ttl": 60,
"type": "A"
}
{
"comments": [],
"name": "name2.",
"records": [
{
"content": "20.20.20.20",
"disabled": false
}
],
"ttl": 60,
"type": "CNAME"
}
I want to get names where type is A.
Help, tell me how to do this?
| jq '.rrsets[] | with_entries(select(.key== "name", .value == "A"))'
{
"name": "name1."
}
{
"name": "name2.",
"type": "A"
}
Displays all the lines, but I only need where type = A
I'm not sure if this is what you are looking for, but wouldn't simply ... | select(.type == "A") do the trick?
... | jq '.rrsets[] | select(.type == "A")'
{
"comments": [],
"name": "name1.",
"records": [
{
"content": "10.10.10.10",
"disabled": false
}
],
"ttl": 60,
"type": "A"
}
Demo
And then just get the .name if you want only that (using -r to get rid of the JSON formatting):
... | jq -r '.rrsets[] | select(.type == "A").name'
name1.
Demo

How to iterate through the array and find sum of all products for all customers using SQL in Azure Cosmos DB -Core SQL API?

{
"Name": "Naveen",
"Age": 28,
"IS_MEMBER": false,
"Products":[
{
"id": 1,
"NAME": "XXXX"
"COST": 5.99
},
{
"id": 2,
"NAME": "XXXX"
"COST": 4.99
}
]
}
What you need is,
SELECT SUM(a.COST) AS SumTotal
FROM a in c.Products
This can be achieved using "Join".
SELECT sum(t.Cost) as SumTotal FROM c join t in c.Products

JQ Cross reference or how to replace one value with another part of the input

I want to parse terraform.tfstate (where openstack provider is used), to return instance name and it's internal + floating IP (if assigned).
First select what we are interested in:
jq -r '.modules?[]|.resources[]?|select(.type == "openstack_compute_floatingip_v2", .type == "openstack_compute_instance_v2")' < terraform.tfstate
For simplicity, pre-parsed example with the above part (one FIP and one instance):
{
"type": "openstack_compute_floatingip_v2",
"depends_on": [
"openstack_networking_router_interface_v2.management"
],
"primary": {
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"attributes": {
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"instance_id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"pool": "public",
"region": "RegionOne"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
{
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_floatingip_v2.management",
"openstack_compute_secgroup_v2.ssh_only",
"openstack_networking_network_v2.management"
],
"primary": {
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"attributes": {
"access_ip_v4": "10.10.10.5",
"access_ip_v6": "",
"all_metadata.%": "1",
"all_metadata.habitat": "sup",
"availability_zone": "nova",
"flavor_id": "eb36e84e-17c1-42ab-b359-4380f6f524ae",
"flavor_name": "m1.large",
"force_delete": "false",
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"image_id": "c574aeed-e47c-4fb7-9da0-75550b76ee56",
"image_name": "ubuntu-16.04",
"key_pair": "vault-etcd_test_tf",
"metadata.%": "1",
"metadata.habitat": "sup",
"name": "ctl01",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.10.10.5",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "",
"network.0.mac": "02:c6:61:f9:ee:7e",
"network.0.name": "management",
"network.0.port": "",
"network.0.uuid": "f2468669-e321-4eb4-9ede-003e362a8988",
"region": "RegionOne",
"security_groups.#": "1",
"security_groups.1845949017": "vault-etcd_test_ssh_only",
"stop_before_destroy": "false"
},
"meta": {
"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
"create": 1800000000000,
"delete": 1800000000000,
"update": 1800000000000
}
},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
Required is to take from "type": "openstack_compute_floatingip_v2" replace .primary.attributes.address and .fixed_ip and from corresponding .instance_id the .name.
So, sth like:
{"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"}
Well, I came with an idea while using walk, but miss how to actually assign the proper value from corresponding instance id:
jq -r "$(cat floating.jq)" terraform.tfstate
floating.jq:
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
.modules?[]|.resources[]?|select(.type ==
"openstack_compute_floatingip_v2", .type ==
"openstack_compute_instance_v2")|
.primary|walk( if type == "object" and .attributes.address then
.attributes.instance_id |= "REFERRED VALUE HERE") else . end)
Let's assume the two related objects are in a file named two.json. Then one way to merge the information from both objects is using the -s command-line option, e.g.
jq -s '
(.[0].primary.attributes | {address, fixed_ip})
+ {name: .[1].primary.attributes.name}' two.json
Output
With your example input, the output would be:
{
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"
}

Resources