[
{
"arguments": {
"leases": [
{
"cltt": 1658763299,
"fqdn-fwd": false,
"fqdn-rev": false,
"hostname": "",
"hw-address": "00:aa:bb:cc:dd:ee",
"ip-address": "192.168.0.2",
"state": 0,
"subnet-id": 1,
"valid-lft": 3600
},
{
"cltt": 1658763207,
"fqdn-fwd": false,
"fqdn-rev": false,
"hostname": "",
"hw-address": "00:11:22:33:44:55",
"ip-address": "192.168.1.3",
"state": 0,
"subnet-id": 1,
"valid-lft": 3600
}
]
},
"result": 0,
"text": "2 IPv4 lease(s) found."
}
]
This is a snippet, but in reality there's much more entries. Currently I filter out MAC and IP with jq expression:
jq --raw-output '.[0] | select(.result == 0) | .arguments.leases[] | "\(.["hw-address"]) \(.["ip-address"])"'
Now I'm wondering: does jq have ability to filter out by regexp? For instance I'd like to dump only entries where IP is 192.168.1.*, can it be done with jq? Ideally I'd like to pass regexp to my script as a parameter:
jq --raw-output --arg addr "$1" ...
Would appreciate suggestions on how to do this.
jq has test to test an input against a regular expression:
first
| select(.result == 0)
| .arguments.leases[]
| select(."ip-address"|test("^192\\.168\\.1"))
| "\(."hw-address") \(."ip-address")"
and to provide the regex as argument via command line:
jq -r --arg regex '^192\.168\.1\.' 'first
| select(.result == 0)
| .arguments.leases[]
| select(."ip-address"|test($regex))
| "\(."hw-address") \(."ip-address")"'
If you only want to check the start of the IP address, you could also use startswith: select(."ip-address"|startswith("192.168.1.")):
jq -r --arg prefix '192.168.1.' 'first
| select(.result == 0)
| .arguments.leases[]
| select(."ip-address"|startswith($prefix))
| "\(."hw-address") \(."ip-address")"'
You can use test with regular expressions, and select to filter:
jq -r --arg addr "192\\.168\\.1\\..*" '
.[0] | select(.result == 0) | .arguments.leases[]
| "\(.["hw-address"]) \(.["ip-address"] | select(test($addr)))"
'
00:11:22:33:44:55 192.168.1.3
Demo
Note: 192.168.1.* is not a regular expression (or at least not one you want to use, as it would also match 192.168.100.4 for instance, because . stands for any value; a literal dot has to be escaped)
I have been trying to understand jq, and the following piuzzle is giving me a headache: I can construct two expressions, A and B, which seem to produce the same output. And yet, when I surround them with [] array construction braces (as in [A] and [B]), they produce different output. In this case, the expressions are:
A := jq '. | add'
B := jq -s `.[] | add`
Concretely:
$ echo '[1,2] [3,4]' | jq '.'
[1,2]
[3,4]
$ echo '[1,2] [3,4]' | jq '. | add'
3
7
# Now surround with array construction and we get two values:
$ echo '[1,2] [3,4]' | jq '[. | add]'
[3]
[7]
$ echo '[1,2] [3,4]' | jq -s '.[]'
[1,2]
[3,4]
$ echo '[1,2] [3,4]' | jq -s '.[] | add'
3
7
# Now surround with array construction and we get only one value:
$ echo '[1,2] [3,4]' | jq -s '[.[] | add]'
[3,7]
What is going on here? Why is it that the B expression, which applies the --slurp setting but appears to produce identical intermediate output to the A expression, produces different output when surrounded with [] array construction brackets?
When jq is fed with a stream, just like [1,2] [3,4] with two inputs, it executes the filter independently for each. That's why jq '[. | add]' will produce two results; each addition will separately be wrapped into an array.
When jq is given the --slurp option, it combines the stream to an array, rendering it just one input. Therefore jq -s '[.[] | add]' will have one result only; the multiple additions will be caught by the array constructor, which is executed just once.
How can I (generically) transform the input file below to the output file below, using jq. The record format of the output file is: array_index | key | value
Input file:
[{"a": 1, "b": 10},
{"a": 2, "d": "fred", "e": 30}]
Output File:
0|a|1
0|b|10
1|a|2
1|d|fred
1|e|30
Here's a solution using tostream, which creates a stream of paths and their values. Filter out those having values using select, flatten to align both, and join for the output format:
jq -r 'tostream | select(has(1)) | flatten | join("|")'
0|a|1
0|b|10
1|a|2
1|d|fred
1|e|30
Demo
Or a very similar one using paths to get the paths, scalars for the filter, and getpath for the corresponding value:
jq -r 'paths(scalars) as $p | [$p[], getpath($p)] | join("|")'
0|a|1
0|b|10
1|a|2
1|d|fred
1|e|30
Demo
< file.json jq -r 'to_entries
| .[]
| .key as $k
| ((.value | to_entries )[]
| [$k, .key, .value])
| #csv'
Output:
0,"a",1
0,"b",10
1,"a",2
1,"d","fred"
1,"e",30
You just need to remove the double quotes.
to_entries can be used to loop over the elements of arrays and objects in a way that gives both the key (index) and the value of the element.
jq -r '
to_entries[] |
.key as $id |
.value |
to_entries[] |
[ $id, .key, .value ] |
join("|")
'
Demo on jqplay
Replace join("|") with #csv to get proper CSV.
$ jq --slurp '.[] | .a' <<< '{"a": 1}'$'\n''{"a": 2}'
1
2
I can process a one-column TSV file like above. When there are multiple columns and one column is JSON, how to print the processing result of the JSON column alone with other columns literally? In the following example, how to print the first column and the JSON processing result of the 2nd column?
$ jq --slurp '.[] | .a' <<< $'A\t{"a": 1}'$'\nB\t{"a": 2}'
parse error: Invalid numeric literal at line 1, column 2
Before piping your TSV file into jq you should extract the JSON column first. For instance, use cut from GNU coreutils to get the second field in a tab-separated line:
cut -f2 <<< $'A\t{"a": 1}'$'\nB\t{"a": 2}' | jq --slurp '.[] | .a'
In order to print the other columns as well, you may use paste to put the columns back together:
paste <(
cut -f1 <<< $'A\t{"a": 1}'$'\nB\t{"a": 2}'
) <(
cut -f2 <<< $'A\t{"a": 1}'$'\nB\t{"a": 2}' | jq --slurp '.[] | .a'
)
To solve this entirely in jq you have to read it as non-JSON first and interpret the second column as JSON using jq's fromjson
jq -Rr './"\t" | .[1] |= (fromjson | .a) | #tsv' <<< $'A\t{"a": 1}'$'\nB\t{"a": 2}'
jq --raw-input --raw-output --slurp 'split("\n") | map(split("\t")) | map(select(length>0)) | .[] | {"p":.[0], "j":.[1] | fromjson} | [.p, .j.a] | #tsv' <<< $'A\t{"a": 1}'$'\nB\t{"a": 2}'
A 1
B 2
or process line by line for huge data
cat ./data.txt | while read line;
do
echo "$line" | jq --raw-input --raw-output --slurp 'split("\t") | {"p":.[0], "j":.[1] | fromjson} | [.p, .j.a] | #tsv'
done
I have the following script:
curl -s -S 'https://bittrex.com/Api/v2.0/pub/market/GetTicks?marketName=BTC-NBT&tickInterval=thirtyMin&_=1521347400000' | jq -r '.result|.[] |[.T,.O,.H,.L,.C,.V,.BV] | #tsv | tostring | gsub("\t";",") | "(\(.))"'
This is the output:
(2018-03-17T18:30:00,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(2018-03-17T19:00:00,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(2018-03-17T19:30:00,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(2018-03-17T20:00:00,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
I want to replace the date with timestamp.
I can make this conversion with date in the shell
date -d '2018-03-17T18:30:00' +%s%3N
1521325800000
I want this result:
(1521325800000,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(1521327600000,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(1521329400000,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(1521331200000,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
This data is stored in MySQL.
Is it possible to execute the date conversion with jq or another command like awk, sed, perl in a single command line?
Here is an all-jq solution that assumes the "Z" (UTC+0) timezone.
In brief, simply replace .T by:
((.T + "Z") | fromdate | tostring + "000")
To verify this, consider:
timestamp.jq
[splits("[(),]")]
| .[1] |= ((. + "Z")|fromdate|tostring + "000") # milliseconds
| .[1:length-1]
| "(" + join(",") + ")"
Invocation
jq -rR -f timestamp.jq input.txt
Output
(1521311400000,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(1521313200000,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(1521315000000,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(1521316800000,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
Here is an unportable awk solution. It is not portable because it relies on the system date command; on the system I'm using, the relevant invocation looks like: date -j -f "%Y-%m-%eT%T" STRING "+%s"
awk -F, 'BEGIN{OFS=FS}
NF==0 { next }
{ sub(/\(/,"",$1);
cmd="date -j -f \"%Y-%m-%eT%T\" " $1 " +%s";
cmd | getline $1;
$1=$1 "000"; # milliseconds
printf "%s", "(";
print;
}' input.txt
Output
(1521325800000,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(1521327600000,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(1521329400000,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(1521331200000,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
Solution with sed :
sed -e 's/(\([^,]\+\)\(,.*\)/echo "(\$(date -d \1 +%s%3N),\2"/g' | ksh
test :
<commande_curl> | sed -e 's/(\([^,]\+\)\(,.*\)/echo "(\$(date -d \1 +%s%3N),\2"/g' | ksh
or :
<commande_curl> > results_curl.txt
cat results_curl.txt | sed -e 's/(\([^,]\+\)\(,.*\)/echo "(\$(date -d \1 +%s%3N),\2"/g' | ksh