I'm trying to extract some info from the Defender for Cloud Qualys scanner through Azure Data Explorer.
What I want to do is for each row, I want to do a foreach for each CVE and then display each one with the server name and QID e.g.
"cve": [
{
"title": "CVE-2022-21123",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21123"
},
{
"title": "CVE-2022-21125",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21125"
},
{
"title": "CVE-2022-21127",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21127"
},
{
"title": "CVE-2022-21166",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21166"
}
]
The table should look something like this:
virtualMachineName,cvetitle,QID
computer1,CVE-2022-21123,48585
computer1,CVE-2022-21125,48585
computer1,CVE-2022-21127,48585
computer1,CVE-2022-21166,48585
computer2,CVE-2022-21125,48585
computer2,CVE-2010-38244,39459
computer3,CVE-2009-83492,39459
On line 8, it grabs the first CVE item but how do I grab the rest and then for each one create another row?
securityresources
| where type =~ "microsoft.security/assessments/subassessments"
| extend QID=tostring(properties.id)
| where properties.additionalData.source =~ "Built-in Qualys vulnerability assessment"
| extend vulnerabilityName=tostring(properties.displayName),
vulnerabilityType = tostring(properties.additionalData.assessedResourceType),
virtualMachineName=split(properties.resourceDetails.id, "/")[-1],
allCves = properties.cve
| extend cvetitle = properties['additionalData']['cve'][0]['title']
| project QID, vulnerabilityName, vulnerabilityType, virtualMachineName, cvetitle, allCves
you can use the mv-apply operator.
for example:
datatable(virtualMachineName:string, QID:long, payload:dynamic)[
"computer1", 48585, dynamic({"cve": [
{
"title": "CVE-2022-21123",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21123"
},
{
"title": "CVE-2022-21125",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21125"
},
{
"title": "CVE-2022-21127",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21127"
},
{
"title": "CVE-2022-21166",
"link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21166"
}
]})
]
| mv-apply cve = payload.cve on ( project cvetitle = tostring(cve['title']))
| project-away payload
virtualMachineName
QID
cvetitle
computer1
48585
CVE-2022-21123
computer1
48585
CVE-2022-21125
computer1
48585
CVE-2022-21127
computer1
48585
CVE-2022-21166
Related
I have two json files (these are from AWS). One is returned from amazon following a server state change (state.json), the other has details of the instance including a specific tag (The tag is called "Name" and the value has the hostname of the server) - file is called instance.json in my example. I'm trying to write some jq which uses the instanceID retrieved from the state to query the instance details document (with the same instanceID key). I think if I could get the hostname tag added to the state document in the right place, that would be ideal...
This would normally be something I'd possibly be capable of, but the keys are at different depths in the json and I can't figure out how to retrieve different depth/matching keys. (If you're familiar with aws you'll know the server state is also in the instance, however I'm changing the state and don't wish to make 3 amazon calls).
Some sample json below:
instance.json (this is a huge file, I've edited out all the useless bits and maintained the structure):
{
"Reservations": [
{
"Instances": [
{
"InstanceId": "i-1",
"Tags": [
{
"Value": "hostname1",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-1",
"Groups": []
},
{
"Instances": [
{
"InstanceId": "i-2",
"Tags": [
{
"Value": "hostname2",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-1",
"Groups": []
},
{
"Instances": [
{
"InstanceId": "i-3",
"Tags": [
{
"Value": "hostname3",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-1",
"Groups": []
}
]
}
state.json:
{
"StoppingInstances": [
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-1",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-2",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-3",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}
Desirable output (if possible):
{
"StoppingInstances": [
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-1",
"Hostname": "hostname1",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-2",
"Hostname": "hostname2",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-3",
"Hostname": "hostname3",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}
Here's a straightforward approach that just uses INDEX. It assumes an invocation of the following form, though different variations are of course possible:
jq -n --argfile state state.json --argfile instance instance.json -f program.jq
where program.jq contains:
INDEX($instance.Reservations[].Instances[]; .InstanceId)
| map_values(.Tags|from_entries.Name) as $dict
| $state
| .StoppingInstances |= map(. + {Hostname: $dict[.InstanceId]})
If you're not sure where .InstanceId is located in instance.json, you could modify the above as follows:
INDEX($instance | .. | objects | select(has("Instances")) | .Instances[];
.InstanceId)
| map_values(.Tags|from_entries.Name) as $dict
| $state
| .StoppingInstances |= map(. + {Hostname: $dict[.InstanceId]})
To query specific tags in instance.json based on ids shared with state.json, you can iterate over the targets by comparing their common ids using a variable and the select function.
jq -r --argfile state state.json --arg tag "Name" '
$state.StoppingInstances[].InstanceId as $id
| .Reservations[].Instances[]
| select(.InstanceId == $id).Tags[]
| select(.Key == $tag).Value
' instance.json
hostname1
hostname2
hostname3
To join records from instance.json with items in state.json, you could use the INDEX and JOIN builtins:
jq --argfile instance instance.json '
.StoppingInstances |= JOIN(
INDEX($instance.Reservations[].Instances[]; .InstanceId); .InstanceId
)
' state.json
{
"StoppingInstances": [
[
{"CurrentState":{"Code":80,"Name":"stopped"},"InstanceId":"i-1","PreviousState":{"Code":80,"Name":"stopped"}},
{"InstanceId":"i-1","Tags":[{"Value":"hostname1","Key":"Name"}],"AmiLaunchIndex":0}
],
[
{"CurrentState":{"Code":80,"Name":"stopped"},"InstanceId":"i-2","PreviousState":{"Code":80,"Name":"stopped"}},
{"InstanceId":"i-2","Tags":[{"Value":"hostname2","Key":"Name"}],"AmiLaunchIndex":0}
],
[
{"CurrentState":{"Code":80,"Name":"stopped"},"InstanceId":"i-3","PreviousState":{"Code":80,"Name":"stopped"}},
{"InstanceId":"i-3","Tags":[{"Value":"hostname3","Key":"Name"}],"AmiLaunchIndex":0}
]
]
}
Here my document:
[
{
"id": "9f0e27fe-3b8f-4857-8e1d-e57e7a3f4c31",
"identifier": [
{
"system": {
"value": "urn:oid:1.3.6.1.4.1.19126.3"
},
"value": {
"value": "Y3454867M"
}
},
{
"system": {
"value": "urn:oid:2.16.724.4.9.10.2"
},
"value": {
"value": "108505134"
}
}
]
}
]
I need to pick only .identifier[where .system.value == "urn:oid:1.3.6.1.4.1.19126.3"] and project .identifier.value.value.
Desired output:
[
{
"id": "9f0e27fe-3b8f-4857-8e1d-e57e7a3f4c31",
"identifier": "Y3454867M"
}
]
I've been playing with map and select but I don't quite figure out what's the right way to get it.
Any ideas?
This approach uses first to get the first result, in case there is more than one array item matching the criteria.
jq --arg v "urn:oid:1.3.6.1.4.1.19126.3" '
map(.identifier |= first(.[] | select(.system.value == $v).value.value))
'
[
{
"id": "9f0e27fe-3b8f-4857-8e1d-e57e7a3f4c31",
"identifier": "Y3454867M"
}
]
Demo
Right on the money with the good ol' select tool, since you need data from an arbitrary index. I fumbled a bit before I unwrapped the inner array that gets piped to my select.
jq -r '.[] | [{id: .id, identifier: .identifier | .[] | select(.system.value | contains("urn:oid:1.3.6.1.4.1.19126.3")) | .value.value }]'
Still new to jq myself, so any feedback is welcome.
are there have any way to achieve simply as expected?
it using combine with permutate in default.
The Following Case
source
{
"title":["title1","title2"],
"link":["http://testapi.cn","http://testapi.org"]
}
expression [{title:.title[],link:.link[]}]
console
[
{
"title": "title1",
"link": "http://testapi.cn"
},
{
"title": "title1",
"link": "http://testapi.org"
},
{
"title": "title2",
"link": "http://testapi.cn"
},
{
"title": "title2",
"link": "http://testapi.org"
}
]
expected
[
{
"title": "title1",
"link": "http://testapi.cn"
},
{
"title": "title2",
"link": "http://testapi.org"
}
]
Use transpose to generate an array of arrays with one element of each input array.
[.title,.link] | transpose | map({title:.[0],link:.[1]})
I am trying to extract kubeconfig data with jq.
kubectl config view --raw -o json | jq ...
There is a json produced of this kind:
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [
{
"name": "some-name",
"cluster": {
"server": "https://some-url",
"certificate-authority-data": "some-cert"
}
},
{
"name": "another-name",
"cluster": {
"server": "https://another-url",
"certificate-authority-data": "another-cert"
}
}
],
"users": [
{
"name": "some-name",
"user": {
"username": "some-user",
"password": "some-password"
}
},
{
"name": "another-name",
"user": {
"username": "another-user",
"password": "another-password"
}
}
],
"contexts": [],
"current-context": "some-context"
}
Question #1:
For a given name ,"some-name", I'd like to extract json:
{
url: "https://some-url",
cert: "some-cert",
username: "some-user",
password: "some-password"
}
Question #2:
"users" sub-section can have other format
"users": [
{
"name": "...",
"user": {
"exec": {
...
}
Where .user.username or .user.password or both can be missing
In this case overall query should return "{}", even though, "clusters" query/branch has result
Question 3, as a follow up to Jeff Mercado answer:
I want to get all clusters, joined (grouped by) name:
Looking at the manual, https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions ,
section "Multiplication, division, modulo: *, /, and %", example:
jq '{"k": {"a": 1, "b": 2}} * {"k": {"a": 0,"c": 3}}' => {"k": {"a": 0, "b": 2, "c": 3}}'
gives presumably right result, assuming "k" is value of "name". So, grouping by "k", merging (*) the results.
I produced following query:
echo "${json}" | jq -r '(.clusters[] | {(.name): {url: .cluster.server, cert: .cluster["certificate-authority-data"]}}) * (.users[] | {(.name): {user: .user.username, password: .user.password}})'
First part returns {"name": {url: cert}}, second part is {"name": {username, password}}
However, result is not merge as in jq Manual, but something else ... product ?
{
"some-name": {
"url": "https://some-url",
"cert": "some-cert",
"user": "some-user",
"password": "some-password"
}
}
{
"another-name": {
"url": "https://another-url",
"cert": "another-cert"
},
"some-name": {
"user": "some-user",
"password": "some-password"
}
}
{
"some-name": {
"url": "https://some-url",
"cert": "some-cert"
},
"another-name": {
"user": "another-user",
"password": "another-password"
}
}
{
"another-name": {
"url": "https://another-url",
"cert": "another-cert",
"user": "another-user",
"password": "another-password"
}
}
Why/what is it ? Kind of following idea of the product ('*') but not of the jq tutorial as I (most likely, incorrectly) understand it
Experimentation:
I have 2 queries now producing partial result.
Let's grab original json (above) in and parse:
read -d '' json << EOF
...
EOF
queries:
echo "${json}" | jq -r '.clusters[] | select(.name=="some-name") | .cluster | {url: .server, cert: .["certificate-authority-data"]}' &&\
echo "${json}" | jq -r '.users[] | select(.name=="some-name") | .user | {user: .username, password: .password}'
Will produce the split output:
{
"url": "https://some-url",
"cert": "some-cert"
}
{
"user": "some-user",
"password": "some-password"
}
Or, with key added for further merge:
echo "${json}" | jq -r '.clusters[] | select(.name=="some-name") | {name: .name, url: .cluster.server, cert: .cluster["certificate-authority-data"]}' &&\
echo "${json}" | jq -r '.users[] | select(.name=="some-name") | {name: .name, user: .user.username, password: .user.password}'
Will produce:
{
"name": "some-name",
"url": "https://some-url",
"cert": "some-cert"
}
{
"name": "some-name",
"user": "some-user",
"password": "some-password"
}
"name" is not needed but can be used as a join operation
So you already know how to get the cluster and user by name separately, first step is to select them both within a single filter:
(.clusters[] | select(.name == $name).cluster), (.users[] | select(.name == $name).user)
This will yield two separate objects, the cluster, then the user. But we want to merge them. There's plenty of ways to do this. You could add them (+) directly or merge them (*) but no real difference there. You'll just want to remap the properties to the names you wanted where needed.
(.clusters[] | select(.name == $name).cluster | {url: .server, cert: ."certificate-authority-data"})
+
(.users[] | select(.name == $name).user | {username, password})
Pass the name in as a parameter to your filter;
$ kubectl config view --raw -o json | jq --arg name some-name '
(.clusters[] | select(.name == $name).cluster | {url: .server, cert: ."certificate-authority-data"})
+
(.users[] | select(.name == $name).user | {username, password})
'
For the second part of your question, if it turns out the mapped user is missing key properties and you want to omit them, just add another select filter to the end to test for those properties and replace with an empty object if nothing is found:
... | select(has("username") and has("password")) // {}
jqplay
For an input below:
[{
"commit": {
"author": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"committer": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
"url":"https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f"
},
{
...
}
}]
How can JQ generate a delimited string from different objects as shown below?
"Stephen Dolan", "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f", "2013-06-22T16:30:59Z"
Collect the fields you want in an array and use #csv to convert to a CSV row. Make sure you get the raw output.
jq -r '.[] | [ .commit.author.name, .commit.url, .commit.author.date ] | #csv' input.json