I've got JSON from cURL:
{
"log": {
"52911": {
"message": {
"level": "Info",
"label": "I",
"message": "DHCPREQUEST received (STATE_RENEWING) for 192.168.0.12 from b1:00:00:02:e1:2f. "
},
"timestamp": "Feb 10 13:06:14",
"ident": "ndhcps",
"id": 52911
},
"52912": {
"message": {
"level": "Info",
"label": "I",
"message": "sending ACK of 192.168.0.12 to b1:00:00:02:e1:2f. "
},
"timestamp": "Feb 10 13:06:14",
"ident": "ndhcps",
"id": 52912
}
}
}
How to filter all messages with pattern "DHCPREQUEST" in comand line?
I'd tryed | jq -r '.logs | .[].message.message | select(.!="DHCPREQUEST")', it didn't work.
The result I need to get is: DHCPREQUEST received (STATE_RENEWING) for 192.168.0.12 from b1:00:00:02:e1:2f.
You can use index for testing if a string contains another.
jq -r '.log[].message.message | select(index("DHCPREQUEST"))'
Related
Using ansible playbook, and lookup plugins, How to convert the result of "curl https://ifconfig.io/all" which has the dictionary format key: value to item list
country_code: US
encoding: gzip
forwarded: 29.493.593.012
host: XXX.XXX.X53.2XX
ifconfig_hostname: ifconfig.io
ip: XXX.XXX.X53.2XX
lang: ""
method: GET
mime: '/'
port: 27404
referer: ""
ua: Python-urllib/3.6
The required output: items list in this format:
[
{
"key": "country_code",
"value": "US"
},
{
"key": "encoding",
"value": "gzip"
},
{
"key": "forwarded",
"value": "29.493.593.012"
},
{
"key": "host",
"value": "XXX.XXX.X53.2XX"
},
{
"key": "ifconfig_hostname",
"value": "ifconfig.io"
},
{
"key": "ip",
"value": "XXX.XXX.X53.2XX"
},
{
"key": "lang",
"value": "''"
},
{
"key": "method",
"value": "GET"
},
{
"key": "mime",
"value": "''"
},
{
"key": "port",
"value": "28180"
},
{
"key": "referer",
"value": "''"
},
{
"key": "ua",
"value": "Python-urllib/3.6"
}
]
The script I have used is:
- name: Get url contents
vars:
curl_res: "{{ query('url','https://ifconfig.io/all') }}"
debug:
msg: "{{ item }}"
loop: "{{ curl_res | dict2items }}"
But I have got this error:
TASK [Get url contents] *****************************************************************************************************************
fatal: [servera]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ curl_res | dict2items }}): dict2items requires a dictionary, got <class 'list'> instead."}
So, I made it manually, saved the curl results in a local file then fine-tune it to this format:
cat files/curl_result
{
"country_code": "US",
"encoding": "gzip",
"forwarded": "29.493.593.012",
"host": "XXX.XXX.X53.2XX",
"ifconfig_hostname": "ifconfig.io",
"ip": "XXX.XXX.X53.2XX",
"lang": "''",
"method": "GET",
"mime": "''",
"port": "28580",
"referer": "''",
"ua": "Python-urllib/3.6"
}
then, I have run this task in my play :
- name: extract from file
vars:
items_curl_res: "{{ lookup('file', 'curl_result') | from_json | dict2items }}"
debug:
var: items_curl_res
The return you are getting is not a dictionary, but a list of string in the form key: value.
That form is actually a valid YAML format, so you could use map to apply a from_yaml filter to all those string, then, map again a dict2items and finally, flatten the list of list.
Ending up with this taks:
- debug:
var: >-
query('url','https://ifconfig.io/all')
| map('from_yaml')
| map('dict2items')
| flatten
This task would yield:
query('url','https://ifconfig.io/all') | map('from_yaml') | map('dict2items') | flatten:
- key: country_code
value: BE
- key: encoding
value: gzip
- key: forwarded
value: 93.184.216.34
- key: host
value: 93.184.216.34
- key: ifconfig_hostname
value: ifconfig.io
- key: ip
value: 93.184.216.34
- key: lang
value: ''
- key: method
value: GET
- key: mime
value: ''
- key: port
value: 24294
- key: referer
value: ''
- key: ua
value: ansible-httpget
Wow .. it is working, thanks β.εηοιτ.βε:
here the working script:
- name: Get url contents
debug:
msg: "{{ query('url','https://ifconfig.io/all') | map('from_yaml') | map('dict2items') | flatten }}"
and the output:
"msg": [
{
"key": "country_code",
"value": "US"
},
{
"key": "encoding",
"value": "gzip"
},
{
"key": "forwarded",
"value": "XXX.XX3.8XX.X9X"
},
{
"key": "host",
"value": "XXX.XX3.8XX.X9X"
},
{
"key": "ifconfig_hostname",
"value": "ifconfig.io"
},
{
"key": "ip",
"value": "3XX.X7X.XX4.XX0"
},
{
"key": "lang",
"value": ""
},
{
"key": "method",
"value": "GET"
},
{
"key": "mime",
"value": ""
},
{
"key": "port",
"value": 33470
},
{
"key": "referer",
"value": ""
},
{
"key": "ua",
"value": "Python-urllib/3.6"
}
]
I have two json files (these are from AWS). One is returned from amazon following a server state change (state.json), the other has details of the instance including a specific tag (The tag is called "Name" and the value has the hostname of the server) - file is called instance.json in my example. I'm trying to write some jq which uses the instanceID retrieved from the state to query the instance details document (with the same instanceID key). I think if I could get the hostname tag added to the state document in the right place, that would be ideal...
This would normally be something I'd possibly be capable of, but the keys are at different depths in the json and I can't figure out how to retrieve different depth/matching keys. (If you're familiar with aws you'll know the server state is also in the instance, however I'm changing the state and don't wish to make 3 amazon calls).
Some sample json below:
instance.json (this is a huge file, I've edited out all the useless bits and maintained the structure):
{
"Reservations": [
{
"Instances": [
{
"InstanceId": "i-1",
"Tags": [
{
"Value": "hostname1",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-1",
"Groups": []
},
{
"Instances": [
{
"InstanceId": "i-2",
"Tags": [
{
"Value": "hostname2",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-1",
"Groups": []
},
{
"Instances": [
{
"InstanceId": "i-3",
"Tags": [
{
"Value": "hostname3",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-1",
"Groups": []
}
]
}
state.json:
{
"StoppingInstances": [
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-1",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-2",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-3",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}
Desirable output (if possible):
{
"StoppingInstances": [
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-1",
"Hostname": "hostname1",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-2",
"Hostname": "hostname2",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
},
{
"CurrentState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-3",
"Hostname": "hostname3",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}
Here's a straightforward approach that just uses INDEX. It assumes an invocation of the following form, though different variations are of course possible:
jq -n --argfile state state.json --argfile instance instance.json -f program.jq
where program.jq contains:
INDEX($instance.Reservations[].Instances[]; .InstanceId)
| map_values(.Tags|from_entries.Name) as $dict
| $state
| .StoppingInstances |= map(. + {Hostname: $dict[.InstanceId]})
If you're not sure where .InstanceId is located in instance.json, you could modify the above as follows:
INDEX($instance | .. | objects | select(has("Instances")) | .Instances[];
.InstanceId)
| map_values(.Tags|from_entries.Name) as $dict
| $state
| .StoppingInstances |= map(. + {Hostname: $dict[.InstanceId]})
To query specific tags in instance.json based on ids shared with state.json, you can iterate over the targets by comparing their common ids using a variable and the select function.
jq -r --argfile state state.json --arg tag "Name" '
$state.StoppingInstances[].InstanceId as $id
| .Reservations[].Instances[]
| select(.InstanceId == $id).Tags[]
| select(.Key == $tag).Value
' instance.json
hostname1
hostname2
hostname3
To join records from instance.json with items in state.json, you could use the INDEX and JOIN builtins:
jq --argfile instance instance.json '
.StoppingInstances |= JOIN(
INDEX($instance.Reservations[].Instances[]; .InstanceId); .InstanceId
)
' state.json
{
"StoppingInstances": [
[
{"CurrentState":{"Code":80,"Name":"stopped"},"InstanceId":"i-1","PreviousState":{"Code":80,"Name":"stopped"}},
{"InstanceId":"i-1","Tags":[{"Value":"hostname1","Key":"Name"}],"AmiLaunchIndex":0}
],
[
{"CurrentState":{"Code":80,"Name":"stopped"},"InstanceId":"i-2","PreviousState":{"Code":80,"Name":"stopped"}},
{"InstanceId":"i-2","Tags":[{"Value":"hostname2","Key":"Name"}],"AmiLaunchIndex":0}
],
[
{"CurrentState":{"Code":80,"Name":"stopped"},"InstanceId":"i-3","PreviousState":{"Code":80,"Name":"stopped"}},
{"InstanceId":"i-3","Tags":[{"Value":"hostname3","Key":"Name"}],"AmiLaunchIndex":0}
]
]
}
URL:
https://firestore.googleapis.com/v1beta1/projects/projectname/databases/(default)/documents/:runQuery
POST buffer:
{ "structuredQuery": {"from": [{"collectionId": "example","allDescendants": true}],"where": {"fieldFilter": {"field": {"fieldPath": "Invoiceno"},"op": EQUAL,"value": {"stringValue": "1"}}}}}
Response:
[{
"error": {
"code": 400,
"message": "Invalid JSON payload received. Unknown name \"{ \"structuredQuery\": {\"from\": [{\"collectionId\": \"example\",\"allDescendants\": true}],\"where\": {\"fieldFilter\": {\"field\": {\"fieldPath\": \"Invoiceno\"},\"op\": EQUAL,\"value\": {\"stringValue\": \"1\"}}}}}\": Cannot bind query parameter. Field '{ \"structuredQuery\": {\"from\": [{\"collectionId\": \"example\",\"allDescendants\": true}],\"where\": {\"fieldFilter\": {\"field\": {\"fieldPath\": \"Invoiceno\"},\"op\": EQUAL,\"value\": {\"stringValue\": \"1\"}}}}}' could not be found in request message.",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"description": "Invalid JSON payload received. Unknown name \"{ \"structuredQuery\": {\"from\": [{\"collectionId\": \"example\",\"allDescendants\": true}],\"where\": {\"fieldFilter\": {\"field\": {\"fieldPath\": \"Invoiceno\"},\"op\": EQUAL,\"value\": {\"stringValue\": \"1\"}}}}}\": Cannot bind query parameter. Field '{ \"structuredQuery\": {\"from\": [{\"collectionId\": \"example\",\"allDescendants\": true}],\"where\": {\"fieldFilter\": {\"field\": {\"fieldPath\": \"Invoiceno\"},\"op\": EQUAL,\"value\": {\"stringValue\": \"1\"}}}}}' could not be found in request message."
}
]
}
]
}
}
]
I have given this for POST REST function, but i am unable to get the data.
I believe the issue is in your fieldFilter, where you are using the op as EQUAL and it should be "EQUAL" with quotes, try changing it to the following and it should work:
"fieldFilter":
{
"field":
{
"fieldPath": "Invoiceno"
},
"op": "EQUAL",
"value": {"stringValue": "1"}
}
I discover you need to create an index on root Document.
So when you run the query all is fine.
https://firestore.googleapis.com/v1/projects/{projectId}/databases/(default)/documents:runQuery
"fieldFilter":
{
"field":
{
"fieldPath": "Invoiceno"
},
"op": "EQUAL",
"value": {"stringValue": "1"}
}{ "structuredQuery": {"from": [{"collectionId": "example","allDescendants": true}],"where": {"fieldFilter": {"field": {"fieldPath": "Invoiceno"},"op": EQUAL,"value": {"stringValue": "1"}}}}}
Looking to extract values from api_http array. I am looking for output that looks like the following. Each element should have the name and the url value attached a key called api.
{ "name": "lookproduct1", "api": "http://testapi.api.com"}
{ "name": "lookproduct2", "api": "http://testapi2.api.com"}
{ "name": "lookproduct3", "api": "http://testapi3.api.com"}
{ "name": "lookproduct4", "api": "http://testapi4.api.com"}
the JSON data:
{
"meta": {
"details": {
"value": "Details"
},
"network": {
"label": "Network:",
"value": "test"
},
"title": {
"value": "Test Report"
},
"update": {
"label": "Validation last update:",
"value": "2020-07-15 17:40 UTC"
}
},
"report": {
"api_http": [
[
{
"html_name": "Product 1",
"name": "lookproduct1",
"rank": 3
},
"http://testapi.api.com",
"GB",
"TEST"
],
[
{
"html_name": "Product 2",
"name": "lookproduct2",
"rank": 3
},
"http://testapi2.api.com",
"GB",
"TEST"
],
[
{
"html_name": "Product 3",
"name": "lookproduct3",
"rank": 3
},
"http://testapi3.api.com",
"GB",
"TEST"
],
[
{
"html_name": "Product 4",
"name": "lookproduct4",
"rank": 3
},
"http://testapi.api.com",
"GB",
"TEST"
]
]
}
}
I got the following, but unsure to extract those final two values and create the new output.
.report[] | .[]
Try:
.report.api_http[]|{name:values[0]["name"],api:values[1]}
My output is:
{
"name": "lookproduct1",
"api": "http://testapi.api.com"
}
{
"name": "lookproduct2",
"api": "http://testapi2.api.com"
}
{
"name": "lookproduct3",
"api": "http://testapi3.api.com"
}
{
"name": "lookproduct4",
"api": "http://testapi.api.com"
}
You could use the -c command-line option in conjunction with the following jq filter:
.report.api_http[]
| {name: .[0].name, api: .[1]}
I have this code for searching with frog AQL api:
items.find({"name":{"$match":"abab-2.0.3.json"}})
its come back as response
{
"results": [
{
"repo": "npm-remote-cache",
"path": ".npm/abab",
"name": "abab-2.0.3.json",
"type": "file",
"size": 1124,
"created": "2020-05-07T21:58:12.633Z",
"created_by": "_system_",
"modified": "2020-05-07T21:58:12.604Z",
"modified_by": "_system_",
"updated": "2020-05-07T21:58:12.633Z"
}
],
"range": {
"start_pos": 0,
"end_pos": 1,
"total": 1
}
}
But I need only the "total" field.
There is a way to get only that field?
You can achieve this with jq:
cat response | jq .range.total