Selecting different keys based on success of failure of curl request - jq

I have the following output from a curl request which can differ based on success / failure of the request
Success:
{
"name": "appId (xxxxx-xxx-1234)",
"sent": 7939246,
"received": 6673490,
"details": [
{
"timestamp": "20180201000041",
"sentbytes": 1805,
"receivedbytes": 1485
}
]}
Failure:
{
"error": "Not found"
}
Can someone let me know how to check whether sent or error exists. I have tried
jq '. | select (.error | .sent)
but that does not seem to be the correct syntax.


Even shorter (no need for additional pipes):
jq '.sent or .error'
The output:
true


You were close.
jq '. | select(.error or .sent)'

Related

How to use jq when the response is not a list

My json response is like this.
{
"level": "info",
"timestamp": "2022-03-21T16:10:27.060Z",
"msg": "starting test",
}
{
"level": "warn",
"timestamp": "2022-03-21T16:10:27.060Z",
"msg": "message 2"
}
{
"level": "error",
"timestamp": "2022-03-21T16:10:27.060Z",
"msg": "message 3"
}
I want to extract.
Either 3rd message (or nth message) or the message associated with error
How can I do this with jq?

Use select to filter by condition
jq 'select(.level == "error")'
{
"level": "error",
"timestamp": "2022-03-21T16:10:27.060Z",
"msg": "message 3"
}
Demo
To extract the message, use the -r option
jq -r 'select(.level == "error").msg'
message 3
Demo
If you want to access the nth object, you may want to use a structure whose items you can count. Using the -s option would turn the input stream into an array. Using the --argjson option enables you using an external (JSON) value for reference:
jq -sr --argjson n 2 '.[$n].msg'
message 3
Demo
To directly address the nth element of a stream, use the nth function, in combination with inputs and the -n option to address the entire stream at once:
jq -nr --argjson n 2 'nth($n;inputs).msg'
message 3
Demo

How to use jq package to parse name and id from json?

I have an output that i am getting in this format :-
[
{
"_class": "hudson.model.FreeStyleProject",
"name": "my-name",
"id": "123"
},
{
"_class": "hudson.model.FreeStyleProject",
"name": "my-name2",
"id": "456"
},
{
"_class": "hudson.model.FreeStyleProject",
"name": "my-name3",
"id": "789"
}
]
How can i parse the name and id using jq?
I tried to use [].name
but i get curl: (23) Failed writing body (320 != 1338)
Any help will be appreciated. Thank you.

You failed to mention the relevant error:
jq: error (at <stdin>:17): Cannot index array with string "name"
The program should be
.[].name
Because you provided an incorrect program to jq, it exited earlier than it normally would. This caused the pipe between curl and jq to close, which cause curl to become unable to write to the pipe, which caused curl to emit the error message you did provide.

Demo
https://jqplay.org/s/nolGbk3sD1
Use filter
.[] | .name, .id

Use jq to transform a date value then filter by it

I have some data which takes the following format:
[
{
"level": "error",
"ts": "2021-09-08T17:34:39.163Z",
"caller": "controller/events.go:39",
"msg": "Metric query failed for p50 Latency: request failed: Get \"https://blah\": context deadline exceeded",
"canary": "blah",
"stacktrace": "github.com/fluxcd/flagger/pkg/controller.(*Controller).recordEventErrorf\n\t/workspace/pkg/controller/events.go:39\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).runMetricChecks\n\t/workspace/pkg/controller/scheduler_metrics.go:280\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).runAnalysis\n\t/workspace/pkg/controller/scheduler.go:682\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).advanceCanary\n\t/workspace/pkg/controller/scheduler.go:380\ngithub.com/fluxcd/flagger/pkg/controller.CanaryJob.Start.func1\n\t/workspace/pkg/controller/job.go:39"
},
{
"level": "error",
"ts": "2021-09-09T11:22:33.162Z",
"caller": "controller/events.go:39",
"msg": "Metric query failed for p50 Latency: request failed: Get \"https://blah\": context deadline exceeded",
"canary": "blah",
"stacktrace": "github.com/fluxcd/flagger/pkg/controller.(*Controller).recordEventErrorf\n\t/workspace/pkg/controller/events.go:39\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).runMetricChecks\n\t/workspace/pkg/controller/scheduler_metrics.go:280\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).runAnalysis\n\t/workspace/pkg/controller/scheduler.go:682\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).advanceCanary\n\t/workspace/pkg/controller/scheduler.go:380\ngithub.com/fluxcd/flagger/pkg/controller.CanaryJob.Start.func1\n\t/workspace/pkg/controller/job.go:39"
}
]
I am trying to filter this data by date, e.g. I want to find all of these events after a certain date. The timestamp 1631122500 should filter out the top entry here (with ts == 2021-09-08T17:34:39.163Z).
If I apply the command | jq -s '.[] | select(.ts | fromdateiso8601 > 1475625600)', it returns the error jq: error (at <stdin>:8): date "2021-08-30T21:52:34.039Z" does not match format "%Y-%m-%dT%H:%M:%SZ".
I have managed to amend the ts field by applying the following modifications: jq -s '.[].ts | split(".")[0] | strptime("%Y-%m-%dT%H:%M:%S") | mktime | strftime("%Y-%m-%dT%H:%M:%SZ")'. This returns values like "2021-09-08T18:34:39Z" which is the format specified in the aforementioned error.
What I can't get my head around is how I can transform each of these ts values to the correct format, then use select to filter by dates since a specified time. I tried piping jq <modify> to | jq <select> but this gave me an error about jq being unable to iterate over strings.

From OP's comment:
which is why I'm trying to apply a transformation then select by it. I can transform the date to the correct value required by JQ
Right after calling select, we can pipe the .ts value to remove the nanoseconds which JQ can't parse like so:
jq 'map(select(.ts | (split(".")[0] + "Z") | fromdateiso8601 > 1631184310))'
The above timestamp (1631184310) will just select the second object as you can try online here

Trying to get the correct output from JQ

I'm trying to get this output the device name "test"
My filter is .[] | [.deviceName] and it's returning error: (at :7): Cannot index array with string "deviceName"
{
"test": [
{
"deviceName": "test",
"monitoringServer": "server1"
}
]
}

Presumably you meant:
jq '.test[] | [.deviceName]'
or perhaps:
jq '.[][] | [.deviceName]'
but without knowing your requirements, it's hard to say. That's one of the reasons why the http://stackoverflow.com/help/mcve guidelines were formulated.

Transforming json to table using JQ gives "Cannot index string error"

Output of the following command
jq '.data[] | select(.vulnerable_configuration_cpe_2_2 | contains(["2.4.38"]))' search1.json
gives me
{
"Modified": "2019-06-11T17:29:00.647000",
"Published": "2019-06-11T17:29:00.630000",
"cvss": "5.0",
"cwe": "CWE-399",
"id": "CVE-2019-0220",
"last-modified": "2019-06-25T06:15:10.627000",
"reason": "Text search",
"references": [
"http://lists.opensuse.org/opensuse-security-announce/2019-04/msg00051.html"
],
"summary": "A vulnerability was found in Apache HTTP Server 2.4.0 to 2.4.38. When the path component of a request URL contains multiple consecutive slashes ('/'), directives such as LocationMatch and RewriteRule must account for duplicates in regular expressions while other aspects of the servers processing will implicitly collapse them.",
"vulnerable_configuration": [
"cpe:2.3:a:apache:http_server:2.4.0",
"cpe:2.3:a:apache:http_server:2.4.1",
"cpe:2.3:a:apache:http_server:2.4.2",
"cpe:2.3:a:apache:http_server:2.4.3",
"cpe:2.3:a:apache:http_server:2.4.4",
"cpe:2.3:a:apache:http_server:2.4.6",
"cpe:2.3:a:apache:http_server:2.4.7",
"cpe:2.3:a:apache:http_server:2.4.8",
"cpe:2.3:a:apache:http_server:2.4.9",
"cpe:2.3:a:apache:http_server:2.4.10",
"cpe:2.3:a:apache:http_server:2.4.12",
"cpe:2.3:a:apache:http_server:2.4.14",
"cpe:2.3:a:apache:http_server:2.4.16",
"cpe:2.3:a:apache:http_server:2.4.17",
"cpe:2.3:a:apache:http_server:2.4.18",
"cpe:2.3:a:apache:http_server:2.4.19",
"cpe:2.3:a:apache:http_server:2.4.20",
"cpe:2.3:a:apache:http_server:2.4.21",
"cpe:2.3:a:apache:http_server:2.4.22",
"cpe:2.3:a:apache:http_server:2.4.23",
"cpe:2.3:a:apache:http_server:2.4.24",
"cpe:2.3:a:apache:http_server:2.4.25",
"cpe:2.3:a:apache:http_server:2.4.26",
"cpe:2.3:a:apache:http_server:2.4.27",
"cpe:2.3:a:apache:http_server:2.4.28",
"cpe:2.3:a:apache:http_server:2.4.29",
"cpe:2.3:a:apache:http_server:2.4.30",
"cpe:2.3:a:apache:http_server:2.4.32",
"cpe:2.3:a:apache:http_server:2.4.33",
"cpe:2.3:a:apache:http_server:2.4.34",
"cpe:2.3:a:apache:http_server:2.4.35",
"cpe:2.3:a:apache:http_server:2.4.36",
"cpe:2.3:a:apache:http_server:2.4.37",
"cpe:2.3:a:apache:http_server:2.4.38",
"cpe:2.3:o:opensuse:leap:15.0",
"cpe:2.3:o:opensuse:leap:42.3",
"cpe:2.3:o:debian:debian_linux:8.0",
"cpe:2.3:o:debian:debian_linux:9.0",
"cpe:2.3:o:fedoraproject:fedora:28",
"cpe:2.3:o:fedoraproject:fedora:29",
"cpe:2.3:o:fedoraproject:fedora:30",
"cpe:2.3:o:canonical:ubuntu_linux:14.04:-:-:-:lts",
"cpe:2.3:o:canonical:ubuntu_linux:16.04:-:-:-:lts",
"cpe:2.3:o:canonical:ubuntu_linux:18.04:-:-:-:lts",
"cpe:2.3:o:canonical:ubuntu_linux:18.10"
],
"vulnerable_configuration_cpe_2_2": [
"cpe:/a:apache:http_server:2.4.0",
"cpe:/a:apache:http_server:2.4.1",
"cpe:/a:apache:http_server:2.4.2",
"cpe:/a:apache:http_server:2.4.3",
"cpe:/a:apache:http_server:2.4.4",
"cpe:/a:apache:http_server:2.4.6",
"cpe:/a:apache:http_server:2.4.7",
"cpe:/a:apache:http_server:2.4.8",
"cpe:/a:apache:http_server:2.4.9",
"cpe:/a:apache:http_server:2.4.10",
"cpe:/a:apache:http_server:2.4.12",
"cpe:/a:apache:http_server:2.4.14",
"cpe:/a:apache:http_server:2.4.16",
"cpe:/a:apache:http_server:2.4.17",
"cpe:/a:apache:http_server:2.4.18",
"cpe:/a:apache:http_server:2.4.19",
"cpe:/a:apache:http_server:2.4.20",
"cpe:/a:apache:http_server:2.4.21",
"cpe:/a:apache:http_server:2.4.22",
"cpe:/a:apache:http_server:2.4.23",
"cpe:/a:apache:http_server:2.4.24",
"cpe:/a:apache:http_server:2.4.25",
"cpe:/a:apache:http_server:2.4.26",
"cpe:/a:apache:http_server:2.4.27",
"cpe:/a:apache:http_server:2.4.28",
"cpe:/a:apache:http_server:2.4.29",
"cpe:/a:apache:http_server:2.4.30",
"cpe:/a:apache:http_server:2.4.32",
"cpe:/a:apache:http_server:2.4.33",
"cpe:/a:apache:http_server:2.4.34",
"cpe:/a:apache:http_server:2.4.35",
"cpe:/a:apache:http_server:2.4.36",
"cpe:/a:apache:http_server:2.4.37",
"cpe:/a:apache:http_server:2.4.38",
"cpe:/o:opensuse:leap:15.0",
"cpe:/o:opensuse:leap:42.3",
"cpe:/o:debian:debian_linux:8.0",
"cpe:/o:debian:debian_linux:9.0",
"cpe:/o:fedoraproject:fedora:28",
"cpe:/o:fedoraproject:fedora:29",
"cpe:/o:fedoraproject:fedora:30",
"cpe:/o:canonical:ubuntu_linux:14.04::~~lts~~~",
"cpe:/o:canonical:ubuntu_linux:16.04::~~lts~~~",
"cpe:/o:canonical:ubuntu_linux:18.04::~~lts~~~",
"cpe:/o:canonical:ubuntu_linux:18.10"
]
}
{
"Modified": "2019-04-08T17:29:00.860000",
"Published": "2019-04-08T17:29:00.843000",
"cvss": "6.0",
"cwe": "CWE-362",
"id": "CVE-2019-0217",
"last-modified": "2019-05-13T23:29:03.847000",
"reason": "Text search",
"references": [
"http://lists.opensuse.org/opensuse-security-announce/2019-04/msg00051.html"
],
"summary": "In Apache HTTP Server 2.4 release 2.4.38 and prior, a race condition in mod_auth_digest when running in a threaded server could allow a user with valid credentials to authenticate using another username, bypassing configured access control restrictions.",
"vulnerable_configuration": [
"cpe:2.3:a:apache:http_server:2.4.38",
"cpe:2.3:o:debian:debian_linux:8.0",
"cpe:2.3:o:debian:debian_linux:9.0",
"cpe:2.3:o:fedoraproject:fedora:29",
"cpe:2.3:o:fedoraproject:fedora:30",
"cpe:2.3:o:canonical:ubuntu_linux:12.04:-:-:-:esm",
"cpe:2.3:o:canonical:ubuntu_linux:14.04:-:-:-:lts",
"cpe:2.3:o:canonical:ubuntu_linux:16.04:-:-:-:lts",
"cpe:2.3:o:canonical:ubuntu_linux:18.04:-:-:-:lts",
"cpe:2.3:o:canonical:ubuntu_linux:18.10",
"cpe:2.3:o:redhat:enterprise_linux:7.0",
"cpe:2.3:o:opensuse:leap:15.0",
"cpe:2.3:o:opensuse:leap:42.3"
],
"vulnerable_configuration_cpe_2_2": [
"cpe:/a:apache:http_server:2.4.38",
"cpe:/o:debian:debian_linux:8.0",
"cpe:/o:debian:debian_linux:9.0",
"cpe:/o:fedoraproject:fedora:29",
"cpe:/o:fedoraproject:fedora:30",
"cpe:/o:canonical:ubuntu_linux:12.04::~~esm~~~",
"cpe:/o:canonical:ubuntu_linux:14.04::~~lts~~~",
"cpe:/o:canonical:ubuntu_linux:16.04::~~lts~~~",
"cpe:/o:canonical:ubuntu_linux:18.04::~~lts~~~",
"cpe:/o:canonical:ubuntu_linux:18.10",
"cpe:/o:redhat:enterprise_linux:7.0",
"cpe:/o:opensuse:leap:15.0",
"cpe:/o:opensuse:leap:42.3"
]
}
I just want to display all the possible fields in a simple table on the command line but I am not able to achieve it.
I tried the following command
echo $(jq '.data[] | select(.vulnerable_configuration_cpe_2_2 | contains(["2.4.38"]))' search1.json) | jq -r '.[] | "\(.summary)"'
but it gives me error
jq: error (at <stdin>:1): Cannot index string with string "summary"
I already researched about this and tried modifying this command but it sometimes gives me compilation error. I am sure I am making very stupid mistake somewhere
Expected Output
ID CWE CVSS Summary
xyz xyz xyz xyzxyzxyz

I figured it out myself
The correct command is
echo $(jq '.data[] | select(.vulnerable_configuration_cpe_2_2 | contains(["2.4.38"]))' search1.json) | jq -r '. | "\(.summary)"'
After jq -r there should be a . and not .[]

Resources