JQ: Transform UNIX Timestamp to Datetime - jq

I use actually JQ1.5 under a Windows 10 environment to transform several json files for an import to a MS SQL database. Part of the data are formatted in UNIX timestamp and I need to transform those data to ISO 8601 Format.
Following command i use actually for the transformation of the data:
jq '
[
{ nid, title, nights, zone: .zones[0].title} +
(.sails[] | { sails_nid: .nid, arrival, departure } ) +
(.sails[].cabins[] |
{ cabintype: .cabinType.kindName,
cabinid: .cabinType.nid,
catalogPrice,
discountPrice,
discountPercentage,
currency
}
)
]
' C:\Import\dreamlines_details.json > C:\Import\import_sails.json
Arrival and departure are the data that are in Unix time formated.
Data:
[
{
"nid": 434508,
"title": "Die schönsten Orte unserer Welt",
"nights": 121,
"zone": "Weltreise",
"sails_nid": 434516,
"arrival": 1525644000,
"departure": 1515193200,
"cabintype": "Innenkabine",
"cabinid": 379723,
"catalogPrice": 17879,
"discountPrice": 9519,
"discountPercentage": 0.4675876726886291,
"currency": "EUR"
},
{
"nid": 434508,
"title": "Die schönsten Orte unserer Welt",
"nights": 121,
"zone": "Weltreise",
"sails_nid": 434516,
"arrival": 1525644000,
"departure": 1515193200,
"cabintype": "Innenkabine",
"cabinid": 379730,
"catalogPrice": 18599,
"discountPrice": 10239,
"discountPercentage": 0.44948653153395346,
"currency": "EUR"
}
]
I experimented with built in operator "todate" and "strftime". But get only parsing Errors.

Use todateiso8601 function:
jq '.[].arrival |= todateiso8601 | .[].departure |= todateiso8601' C:\Import\import_sails.json
The output (for your input fragment):
[
{
"nid": 434508,
"title": "Die schönsten Orte unserer Welt",
"nights": 121,
"zone": "Weltreise",
"sails_nid": 434516,
"arrival": "2018-05-06T22:00:00Z",
"departure": "2018-01-05T23:00:00Z",
"cabintype": "Innenkabine",
"cabinid": 379723,
"catalogPrice": 17879,
"discountPrice": 9519,
"discountPercentage": 0.4675876726886291,
"currency": "EUR"
},
{
"nid": 434508,
"title": "Die schönsten Orte unserer Welt",
"nights": 121,
"zone": "Weltreise",
"sails_nid": 434516,
"arrival": "2018-05-06T22:00:00Z",
"departure": "2018-01-05T23:00:00Z",
"cabintype": "Innenkabine",
"cabinid": 379730,
"catalogPrice": 18599,
"discountPrice": 10239,
"discountPercentage": 0.44948653153395346,
"currency": "EUR"
}
]

I had a similar problem when parsing output of Perforce (with -Mj option),
but the epoch times were strings, not numbers.
$ p4 -z tag -Mj labels -e "test_build" | jq '.'
{
"Access": "1581356898",
"Description": "Created by p4build.\n",
"Options": "unlocked noautoreload",
"Owner": "p4build",
"Update": "1580936739",
"label": "test_build"
}
Adding tonumber into the filter fixes it:
$ p4 -z tag -Mj labels -e "test_build" > test.json
$ jq -s '.[].Access |= (tonumber | todateiso8601) | .[].Update |= (tonumber | todateiso8601)' test.json
[
{
"Access": "2020-02-10T17:48:18Z",
"Description": "Created by p4build.\n",
"Options": "unlocked noautoreload",
"Owner": "p4build",
"Update": "2020-02-05T21:05:39Z",
"label": "test_build"
}
]

Related

Flattening a nested json while printing outer json using jq

I have a file like below (excerpt)
cat input.json
{
"Metrics": [
{
"Namespace": "CWAgent",
"MetricName": "disk_used_percent",
"Dimensions": [
{
"Name": "path",
"Value": "/aem"
},
{
"Name": "host",
"Value": "uat2-author01.uat.cloud.abc.com.au"
},
{
"Name": "device",
"Value": "mapper/aem-aem"
},
{
"Name": "fstype",
"Value": "ext4"
}
]
},
{
"Namespace": "CWAgent",
"MetricName": "mem_used_percent",
"Dimensions": [
{
"Name": "host",
"Value": "uat2-dispatch02.uat.cloud.abc.com.au"
}
]
}
]
}
As you can see the "Dimensions" is a list of json objects of varying length. I am required to print each objects under "Metrics" in a single line like below -
CWAgent|disk_used_percent|path:/aem|host:uat2-author01.uat.cloud.abc.com.au|device:mapper/aem-aem|fstype:ext4
CWAgent|mem_used_percent|host:uat2-dispatch02.uat.cloud.abc.com.au
I used this -
cat input.json | jq -r ".Metrics[] | .Namespace + \"|\" + .MetricName + \"|\" + (.Dimensions[] | .Name + \":\" + .Value + \"|\")"
And the output is like below:
CWAgent|disk_used_percent|path:/aem|
CWAgent|disk_used_percent|host:uat2-author01.uat.cloud.abc.com.au|
CWAgent|disk_used_percent|device:mapper/aem-aem|
CWAgent|disk_used_percent|fstype:ext4|
CWAgent|mem_used_percent|host:uat2-dispatch02.uat.cloud.abc.com.au|
As you can see, we are getting 4 lines for first object each having different object under Dimensions. Can anyone please suggest what will be the jq command to get desired output. I am novice in jq and wasted too much time on this experimenting.
Thanks,
Bijitesh
You're looking for this:
jq -r '.Metrics[] | [.Namespace, .MetricName] + (.Dimensions | map("\(.Name):\(.Value)")) | join("|")' input.json

jq query to pull values from json file to create a listing

Given this input small sample:
{
"_format_version": "1.1",
"_workspace": "test",
"services": [
{
"connect_timeout": 60000,
"host": "host-name-test.com",
"name": "name-of-service",
"path": "/test/oauthpass",
"port": 777,
"protocol": "http",
"read_timeout": 1000,
"retries": 1,
"write_timeout": 1000,
"routes": [
{
"hosts": [
"Google.com"
],
"name": "com.testing.active.oauth",
"methods": [
"POST"
],
"paths": [
"/vendors/otest/pass/?$"
],
"path_handling": "v8",
"preserve_host": false,
"protocols": [
"https"
],
"regex_priority": 0,
"strip_path": true,
"https_redirect_status_code": 426,
"request_buffering": true,
"response_buffering": true
}
]
}
}
trying to get a listing from the data pulling certain values like the listing below:
host-name-test.com, Google.com, POST, HTTPS
the command that I have working so far is
cat /tmp/petecar.json | jq -r ' .services[] | .routes[] | ( .hosts[] + "/" + .paths[]) ' | more
but I can't access the values under services, please provide some sample on how to get the values
routes has an array value and as such cannot be concatenated with a string. You can use join to turn that array into a single string:
jq -r '.services[] | .host + " " + (.routes[].hosts | join(","))'
Output:
host-name-test.com Google.com
Alternatively, using string interpolation) which will automatically serialize any values into their string representation:
jq -r '.services[] | "\(.host) \(.routes[].hosts)"'
Output:
host-name-test.com ["Google.com"]
join and string interpolation can be combined, giving you the identical output of the first command:
jq -r '.services[] | "\(.host) \(.routes[].hosts|join(","))"'

Parsing on jq with_entries

I have the following data
Command output:
| jq '.rrsets[]'
{
"comments": [],
"name": "name1.",
"records": [
{
"content": "10.10.10.10",
"disabled": false
}
],
"ttl": 60,
"type": "A"
}
{
"comments": [],
"name": "name2.",
"records": [
{
"content": "20.20.20.20",
"disabled": false
}
],
"ttl": 60,
"type": "CNAME"
}
I want to get names where type is A.
Help, tell me how to do this?
| jq '.rrsets[] | with_entries(select(.key== "name", .value == "A"))'
{
"name": "name1."
}
{
"name": "name2.",
"type": "A"
}
Displays all the lines, but I only need where type = A
I'm not sure if this is what you are looking for, but wouldn't simply ... | select(.type == "A") do the trick?
... | jq '.rrsets[] | select(.type == "A")'
{
"comments": [],
"name": "name1.",
"records": [
{
"content": "10.10.10.10",
"disabled": false
}
],
"ttl": 60,
"type": "A"
}
Demo
And then just get the .name if you want only that (using -r to get rid of the JSON formatting):
... | jq -r '.rrsets[] | select(.type == "A").name'
name1.
Demo

Flatten JSON data with jq

I have output brakeman scan results to a json file and want to flatten the structure.
Brakeman json output sample looks like this
{
"scan_info": {
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"end_time": "2021-11-01 14:45:13 +1100",
"brakeman_version": "5.1.2"
},
"warnings": [
{
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"file": "app/controllers/my/controller.rb",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to....",
"render_path": null,
"location": {
"type": "method",
"class": "My::Controller",
"method": "request_token"
},
"user_input": "my.base_url",
"confidence": "High"
},
{.....}
],
"ignored_warnings": [],
"errors": [],
"obsolete": ["xxx","yyyy"]
}
I want to flatten the output into an array of objects similar to this. I want most data from the warnings array except .location and .render_path.
[
{
"brakeman_version": "5.1.2",
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f0d2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to...",
"confidence": "High"
},
{
"brakeman_version": "5.1.2",
"warning_type": "SQL Injection",
"check_name": "SQL",
"message": "Possible SQL injection",
"warning_code": 35,
....
"confidence": "High"
}
]
Something like this gives a nested object for each item in the warnings array but I'd like to move them up a level
jq '{brakeman_version: .scan_info.brakeman_version, start_time: .scan_info.start_time, warnings: .warnings[]}' ./brakeman.json
Result
{
"brakeman_version": "5.1.2",
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": {
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "xxxx",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"confidence": "High"
},
{...}
}
Something like this flattens the .scan_info part similarly but still leaves me with an array for warnings which is what was there in the first place.
jq '{
rails_version: .scan_info.rails_version,
security_warnings: .scan_info.security_warnings,
start_time: .scan_info.start_time,
brakeman_version: .scan_info.brakeman_version,
warnings: [.warnings[] | {warning_type: .warning_type,
warning_code: .warning_code,
fingerprint: .fingerprint,
check_name: .check_name,
confidence: .confidence} ],
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
Results
{
"rails_version": "6.1.4.1",
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": [
{
"warning_type": "SQL Injection",
"warning_code": 0,
"fingerprint": "xx",
"check_name": "SQL",
"message": "Possible SQL injection",
"line": 178,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
"confidence": "Medium"
},
{
"warning_type": "Format Validation",
"warning_code": 30,
"fingerprint": "xxx",
"check_name": "ValidationRegex",
"line": 92,
"link": "https://brakemanscanner.org/docs/warning_types/format_validation/",
"code": null,
"user_input": null,
"confidence": "High"
},
]
}
Using jq how do I correctly flatten the structure to output as desired?
I've tried some suggestions found on SO like
jq '{
rails_version: .scan_info.rails_version,
brakeman_version: .scan_info.brakeman_version,
.warnings[] as $in | $in | del(.location, .render_path) as $in2 | $in2,
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
but have been going around in circles at this stage.
With assumptions based on your description provided, I guess you are looking to do this
.scan_info + ( .warnings[] | del(.location, .render_path) )
If you want the entire results packed into an array, enclose the above filter within square brackets([ ])
jqplay demo

JQ Cross reference or how to replace one value with another part of the input

I want to parse terraform.tfstate (where openstack provider is used), to return instance name and it's internal + floating IP (if assigned).
First select what we are interested in:
jq -r '.modules?[]|.resources[]?|select(.type == "openstack_compute_floatingip_v2", .type == "openstack_compute_instance_v2")' < terraform.tfstate
For simplicity, pre-parsed example with the above part (one FIP and one instance):
{
"type": "openstack_compute_floatingip_v2",
"depends_on": [
"openstack_networking_router_interface_v2.management"
],
"primary": {
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"attributes": {
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"instance_id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"pool": "public",
"region": "RegionOne"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
{
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_floatingip_v2.management",
"openstack_compute_secgroup_v2.ssh_only",
"openstack_networking_network_v2.management"
],
"primary": {
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"attributes": {
"access_ip_v4": "10.10.10.5",
"access_ip_v6": "",
"all_metadata.%": "1",
"all_metadata.habitat": "sup",
"availability_zone": "nova",
"flavor_id": "eb36e84e-17c1-42ab-b359-4380f6f524ae",
"flavor_name": "m1.large",
"force_delete": "false",
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"image_id": "c574aeed-e47c-4fb7-9da0-75550b76ee56",
"image_name": "ubuntu-16.04",
"key_pair": "vault-etcd_test_tf",
"metadata.%": "1",
"metadata.habitat": "sup",
"name": "ctl01",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.10.10.5",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "",
"network.0.mac": "02:c6:61:f9:ee:7e",
"network.0.name": "management",
"network.0.port": "",
"network.0.uuid": "f2468669-e321-4eb4-9ede-003e362a8988",
"region": "RegionOne",
"security_groups.#": "1",
"security_groups.1845949017": "vault-etcd_test_ssh_only",
"stop_before_destroy": "false"
},
"meta": {
"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
"create": 1800000000000,
"delete": 1800000000000,
"update": 1800000000000
}
},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
Required is to take from "type": "openstack_compute_floatingip_v2" replace .primary.attributes.address and .fixed_ip and from corresponding .instance_id the .name.
So, sth like:
{"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"}
Well, I came with an idea while using walk, but miss how to actually assign the proper value from corresponding instance id:
jq -r "$(cat floating.jq)" terraform.tfstate
floating.jq:
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
.modules?[]|.resources[]?|select(.type ==
"openstack_compute_floatingip_v2", .type ==
"openstack_compute_instance_v2")|
.primary|walk( if type == "object" and .attributes.address then
.attributes.instance_id |= "REFERRED VALUE HERE") else . end)
Let's assume the two related objects are in a file named two.json. Then one way to merge the information from both objects is using the -s command-line option, e.g.
jq -s '
(.[0].primary.attributes | {address, fixed_ip})
+ {name: .[1].primary.attributes.name}' two.json
Output
With your example input, the output would be:
{
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"
}

Resources