Extracting values from JSON column using KQL (Azure Data Explorer) - azure-data-explorer

Can you please tell me how to extract values of category, enabled and categoryGroup from the below JSON column in KQL(Azure Data Explorer).
Below JSON value is exactly what I see in the column called "Logs". I see that the Column Logs is defined as string datatype in table
AzLogsCoverage
| extend Logs = case(isnull(Logs) or isempty(Logs), 'N/A', Logs)
| where Logs <> 'N/A'
| project Logs
| extend LogsCategory = parse_json(Logs).category
[
{
"category": "Administrative",
"enabled": true,
"categoryGroup": null
},
{
"category": "Security",
"enabled": false,
"categoryGroup": null
},
{
"category": "ServiceHealth",
"enabled": false,
"categoryGroup": null
},
{
"category": "Alert",
"enabled": false,
"categoryGroup": null
},
{
"category": "Recommendation",
"enabled": false,
"categoryGroup": null
},
{
"category": "Policy",
"enabled": false,
"categoryGroup": null
},
{
"category": "Autoscale",
"enabled": false,
"categoryGroup": null
},
{
"category": "ResourceHealth",
"enabled": false,
"categoryGroup": null
}
]

if the input is of type string, you first need to invoke parse_json() on it, to make it of type dynamic.
Then, you can use mv-expand/mv-apply to expand elements in the array, and then you can explicitly project properties of interest for each element.
for example:
print input = ```[
{
"category": "Administrative",
"enabled": true,
"categoryGroup": null
},
{
"category": "Security",
"enabled": false,
"categoryGroup": null
},
{
"category": "ServiceHealth",
"enabled": false,
"categoryGroup": null
},
{
"category": "Alert",
"enabled": false,
"categoryGroup": null
},
{
"category": "Recommendation",
"enabled": false,
"categoryGroup": null
},
{
"category": "Policy",
"enabled": false,
"categoryGroup": null
},
{
"category": "Autoscale",
"enabled": false,
"categoryGroup": null
},
{
"category": "ResourceHealth",
"enabled": false,
"categoryGroup": null
}
]```
| extend d = parse_json(input)
| mv-apply d on (
project Category = tostring(d.category),
Enabled = tobool(d.enabled),
CategoryGroup = tostring(d.categoryGroup)
)
| project-away input
Category
Enabled
CategoryGroup
Administrative
True
Security
False
ServiceHealth
False
Alert
False
Recommendation
False
Policy
False
Autoscale
False
ResourceHealth
False

AzLogsCoverage
| extend Logs = case(isnull(Logs) or isempty(Logs), 'N/A', Logs)
| extend Metrics = case(isnull(Metrics) or isempty(Metrics), 'N/A', Metrics)
| where Logs <> 'N/A'
| extend LogsDynamic = todynamic(Logs)
| extend MetricsDynamics = todynamic(Metrics)
| mv-expand LogsDynamic, MetricsDynamics
| project SubscriptionId, ResourceId, ResourceName, ResourceType, DiagnosticSettingStatus, DiagnosticSettingId, DiagnosticSettingName, DiagnosticSettingType, LAworkspaceId, LAworkspaceRetentionPeriod,
LogsDynamic.category,LogsDynamic.enabled,LogsDynamic.categoryGroup, MetricsDynamics.category, MetricsDynamics.categoryGroup, MetricsDynamics.enabled, MetricsDynamics.retentionPolicy.enabled , MetricsDynamics.retentionPolicy.days

Related

Project Nested values in KQL from target resources

I am new to KQL, struggling to project values inside target resources but not able to retrieve the required values can you please help me below.
AuditLogs
| where ActivityDisplayName contains "Update user" and InitiatedBy contains "Testuser"
| where TargetResources contains "Id=xxxxxxxx"
wanted to project id and userprinciplename which are coming inside the targetresources, attached screenshot below link for reference
https://ibb.co/DpzZkks
Sample Json
[{
"id": "xyz",
"displayName": null,
"type": "User",
"userPrincipalName": "ABC",
"modifiedProperties": [{
"displayName": "xxxxxx",
"oldValue": "xxxxx",
"newValue": "yyyyy"
},
{
"displayName": "xxxxx",
"oldValue": "[{xxxx}]",
"newValue": "[{xxxx}]"
},
{
"displayName": "Included Updated Properties",
"oldValue": null,
"newValue": "xxxxx"
},
{
"displayName": "TargetId.UserType",
"oldValue": null,
"newValue": "\"Member\""
}
],
"administrativeUnits": []
}]
if TargetResources is of type string, you can use parse_json() to make it of type dynamic, then access its properties as follows:
| extend TargetResources = parse_json(TargetResources)
| project A = tostring(TargetResources.PropertyA),
B = tolong(TargetResources.PropertyB),
C = TargetResources.PropertyC
if TargetResources is already of type dynamic, you can simply access its properties as shown above, without having to invoke parse_json().
relevant documentation topics:
the dynamic data type
the parse_json() function
for example:
print input = '{"key1":1, "key2":"value2", "key3":{"key4":4.4}}'
| extend input = parse_json(input)
| project v1 = tolong(input.key1),
v2 = tostring(input.key2),
v4 = todouble(input.key3.key4)
v1
v2
v4
1
value2
4.4
UPDATE, based on the provided data sample
when dealing with arrays, you can use mv-expand or mv-apply to expand elements in the arrays for further processing
examples:
examples:
print TargetResources = dynamic([{
"id": "xyz",
"displayName": null,
"type": "User",
"userPrincipalName": "ABC",
"modifiedProperties": [{
"displayName": "xxxxxx",
"oldValue": "xxxxx",
"newValue": "yyyyy"
},
{
"displayName": "xxxxx",
"oldValue": "[{xxxx}]",
"newValue": "[{xxxx}]"
},
{
"displayName": "Included Updated Properties",
"oldValue": null,
"newValue": "xxxxx"
},
{
"displayName": "TargetId.UserType",
"oldValue": null,
"newValue": "\"Member\""
}
],
"administrativeUnits": []
}])
| mv-expand TargetResources
| project TargetResources.id, TargetResources.userPrincipalName
TargetResources_id
TargetResources_userPrincipalName
xyz
ABC
print TargetResources = dynamic([{
"id": "xyz",
"displayName": null,
"type": "User",
"userPrincipalName": "ABC",
"modifiedProperties": [{
"displayName": "xxxxxx",
"oldValue": "xxxxx",
"newValue": "yyyyy"
},
{
"displayName": "xxxxx",
"oldValue": "[{xxxx}]",
"newValue": "[{xxxx}]"
},
{
"displayName": "Included Updated Properties",
"oldValue": null,
"newValue": "xxxxx"
},
{
"displayName": "TargetId.UserType",
"oldValue": null,
"newValue": "\"Member\""
}
],
"administrativeUnits": []
}])
| mv-expand TargetResources
| mv-expand TargetResources.modifiedProperties
| project TargetResources_modifiedProperties
TargetResources_modifiedProperties
{ "displayName": "xxxxxx", "oldValue": "xxxxx", "newValue": "yyyyy"}
{ "displayName": "xxxxx", "oldValue": "[{xxxx}]", "newValue": "[{xxxx}]"}
{ "displayName": "Included Updated Properties", "oldValue": null, "newValue": "xxxxx"}
{ "displayName": "TargetId.UserType", "oldValue": null, "newValue": ""Member""}

jq array filter for nested array elements

I am trying to add a new user in below json which matches group NP01-RW. i am able to do without NP01-RW but not able to select users under NP01-RW and return updated json.
{
"id": 181,
"guid": "c9b7dbde-63de-42cc-9840-1b4a06e13364",
"isEnabled": true,
"version": 17,
"service": "Np-Hue",
"name": "DATASCIENCE-CUROPT-RO",
"policyType": 0,
"policyPriority": 0,
"isAuditEnabled": true,
"resources": {
"database": {
"values": [
"hive_cur_acct_1dev",
"hive_cur_acct_1eng",
"hive_cur_acct_1rwy",
"hive_cur_acct_1stg",
"hive_opt_acct_1dev",
"hive_opt_acct_1eng",
"hive_opt_acct_1stg",
"hive_opt_acct_1rwy"
],
"isExcludes": false,
"isRecursive": false
},
"column": {
"values": [
"*"
],
"isExcludes": false,
"isRecursive": false
},
"table": {
"values": [
"*"
],
"isExcludes": false,
"isRecursive": false
}
},
"policyItems": [
{
"accesses": [
{
"type": "select",
"isAllowed": true
},
{
"type": "update",
"isAllowed": true
},
{
"type": "create",
"isAllowed": true
},
{
"type": "drop",
"isAllowed": true
},
{
"type": "alter",
"isAllowed": true
},
{
"type": "index",
"isAllowed": true
},
{
"type": "lock",
"isAllowed": true
},
{
"type": "all",
"isAllowed": true
},
{
"type": "read",
"isAllowed": true
},
{
"type": "write",
"isAllowed": true
}
],
"users": [
"user1",
"user2",
"user3"
],
"groups": [
"NP01-RW"
],
"conditions": [],
"delegateAdmin": false
},
{
"accesses": [
{
"type": "select",
"isAllowed": true
}
],
"users": [
"user1"
],
"groups": [
"NP01-RO"
],
"conditions": [],
"delegateAdmin": false
}
],
"denyPolicyItems": [],
"allowExceptions": [],
"denyExceptions": [],
"dataMaskPolicyItems": [],
"rowFilterPolicyItems": [],
"options": {},
"validitySchedules": [],
"policyLabels": [
"DATASCIENCE-CurOpt-RO_NP01"
]
}
below is what i have tried but it returns part of the JSON matching NP01-RW and not full JSON
jq --arg username "$sync_userName" '.policyItems[] | select(.groups[] | IN("NP01-RO")).users += [$username]' > ${sync_policyName}.json
Operator precedence in jq is not always intuitive. Your program is parsed as:
.policyItems[] | (select(.groups[] | IN("NP01-RO")).users += [$username])
Which first streams all policyItems and only then changes them, leaving you with policyItems only in the output.
You need to make sure that the stream selects the correct values, which you can then assign:
(.policyItems[] | select(.groups[] | IN("NP01-RO")).users) += [$username]
This will do the assignment, but still return the full input (.).

Parsing json file with jq from HPE iLO

I have a json file pulled from an HPE iLO interface with the snmp configuration. It looks like:
[
{
"Comments": {
"BIOSDate": "01/23/2021",
"BIOSFamily": "U30",
"Manufacturer": "HPE",
"Model": "ProLiant DL380 Gen10",
"SerialNumber": "5UNESX378",
"iLOVersion": "iLO 5 v2.65"
}
},
{
"#HpeiLOSnmpService.v2_3_0.HpeiLOSnmpService": {
"/redfish/v1/Managers/1/SnmpService/": {
"#odata.context": "/redfish/v1/$metadata#HpeiLOSnmpService.HpeIloSnmpService",
"#odata.id": "/redfish/v1/Managers/1/SnmpService",
"Actions": {
"#HpeIloSnmpService.SendSNMPTestAlert": {
"target": "/redfish/v1/Managers/1/SnmpService/Actions/HpeILOSnmpService.SendSNMPTestAlert/"
}
},
"AlertDestinationAssociations": [
{
"SNMPAlertProtocol": "SNMPv3Trap",
"SecurityName": null
}
],
"AlertDestinations": [
"1.2.3.4",
"5.6.7.8",
null,
null
],
"AlertsEnabled": true,
"Name": "SnmpService"
},
"PeriodicHSATrapConfig": "Disabled",
"ReadCommunities": [
"",
"",
""
],
"Role": "",
"RoleDetail": "",
"SNMPAlertDestinations": {
"#odata.id": "/redfish/v1/Managers/1/SnmpService/SNMPAlertDestinations/"
},
"SNMPUsers": {
"#odata.id": "/redfish/v1/Managers/1/SnmpService/SNMPUsers/"
},
"SNMPv1Enabled": false,
"SNMPv3EngineID": "0x8920000000E3028329E002033",
"SNMPv3InformRetryAttempt": 2,
"SNMPv3InformRetryIntervalSeconds": 15,
"Status": {
"State": "Enabled"
},
"TrapCommunities": [
"",
"",
"",
"",
"",
"",
""
],
"TrapSourceHostname": "Manager",
"Users": [
{
"AuthProtocol": "MD5",
"PrivacyProtocol": "DES",
"SecurityName": "",
"UserEngineID": null
},
{
"AuthProtocol": "MD5",
"PrivacyProtocol": "DES",
"SecurityName": "",
"UserEngineID": null
},
{
"AuthProtocol": "SHA",
"PrivacyProtocol": "AES",
"SecurityName": "oneview_4849283d97929392",
"UserEngineID": null
},
{
"AuthProtocol": "MD5",
"PrivacyProtocol": "DES",
"SecurityName": "",
"UserEngineID": null
}
]
}
}
]
I want to select an element in the Users array that has SecurityName set to "" and change that element. I don't need the Comments portion. So, I try to select the section starting with #HpeiLOSnmpService.v2_3_0.HpeiLOSnmpService with:
jq -r '.[] | .#HpeiLOSnmpService.v2_3_0.HpeiLOSnmpService' snmp.json
but it gives me everything without the enclosing array. Anyone have a suggestion?
Thanks!
# starts a comment and your jq program degenerates to .[]|. which is identical to the program .[] (|. is a no-op/the identity filter). This program will simply select all values from the input.
You must quote certain characters, such as #, when they are part of propery names. The following will work with your JSON file:
jq -r '.[] | ."#HpeiLOSnmpService".v2_3_0.HpeiLOSnmpService'
Thanks. Using the quotes around the key with a '#' works. Ultimately, selecting the SecurityName that was unset was done with:
jq -r '.Users[] | select (.SecurityName == "") | {"AuthProtocol":.AuthProtocol,"PrivacyProtocol":.PrivacyProtocol,"SecurityName":.SecurityName,"UserEngineID":.UserEngineID}'

Parsing on jq with_entries

I have the following data
Command output:
| jq '.rrsets[]'
{
"comments": [],
"name": "name1.",
"records": [
{
"content": "10.10.10.10",
"disabled": false
}
],
"ttl": 60,
"type": "A"
}
{
"comments": [],
"name": "name2.",
"records": [
{
"content": "20.20.20.20",
"disabled": false
}
],
"ttl": 60,
"type": "CNAME"
}
I want to get names where type is A.
Help, tell me how to do this?
| jq '.rrsets[] | with_entries(select(.key== "name", .value == "A"))'
{
"name": "name1."
}
{
"name": "name2.",
"type": "A"
}
Displays all the lines, but I only need where type = A
I'm not sure if this is what you are looking for, but wouldn't simply ... | select(.type == "A") do the trick?
... | jq '.rrsets[] | select(.type == "A")'
{
"comments": [],
"name": "name1.",
"records": [
{
"content": "10.10.10.10",
"disabled": false
}
],
"ttl": 60,
"type": "A"
}
Demo
And then just get the .name if you want only that (using -r to get rid of the JSON formatting):
... | jq -r '.rrsets[] | select(.type == "A").name'
name1.
Demo

How do I iterate through array in Kusto?

I need to iterate through managed data disks in Azure Resource Graph Explorer (https://preview.portal.azure.com/). My query is below but it returns JSON array, I need to extract name of disk and type of storage account which is being used (sample JSON return is below). So I'd like to see on screen grouping by machine name, disk name and then storage account type. My current query is below but obviously it does not work due to return of JSON
where type =~ 'Microsoft.Compute/virtualmachines' |
extend disks = properties.storageProfile.dataDisks |
project name, disks
Same JSON output
[
{
"name": "COMP02_DDisk1",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk1",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 0,
"writeAcceleratorEnabled": false
},
{
"name": "COMP02_DDisk2",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk2",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 1,
"writeAcceleratorEnabled": false
}
]
in such cases, it's usually helpful to use mv-expand to expand the array and then apply the dynamic-property accessors foreach record.
https://learn.microsoft.com/en-us/azure/kusto/query/mvexpandoperator
example:
print d = dynamic([
{
"name": "COMP02_DDisk1",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk1",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 0,
"writeAcceleratorEnabled": false
},
{
"name": "COMP02_DDisk2",
"createOption": "Attach",
"diskSizeGB": 400,
"managedDisk": {
"id": "/subscriptions/5f5c5be9-77d4db790171/resourceGroups/BRAZILSOUTHDB/providers/Microsoft.Compute/disks/COMP02_DDisk2",
"storageAccountType": "Premium_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 1,
"writeAcceleratorEnabled": false
}
])
| mv-expand d
| project d.name, d.managedDisk.storageAccountType
which will output:
| d_name | d_managedDisk_storageAccountType |
|---------------|----------------------------------|
| COMP02_DDisk1 | Premium_LRS |
| COMP02_DDisk2 | Premium_LRS |
Hope you are doing good .
You can try this way also, First i found networksecuritygroups from entire collection and later filtered defaultSecurityRules which is again an array.
After collecting it using mvexpand in local variable rules you should be able to fetch the desired after applying.
where type =~ "microsoft.network/networksecuritygroups"
| mvexpand rules = properties.defaultSecurityRules
| where rules.properties.destinationAddressPrefix =~ "*"
You can also refer given below link and wish it will help you also.

Resources