I have the following script:
curl -s -S 'https://bittrex.com/Api/v2.0/pub/market/GetTicks?marketName=BTC-NBT&tickInterval=thirtyMin&_=1521347400000' | jq -r '.result|.[] |[.T,.O,.H,.L,.C,.V,.BV] | #tsv | tostring | gsub("\t";",") | "(\(.))"'
This is the output:
(2018-03-17T18:30:00,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(2018-03-17T19:00:00,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(2018-03-17T19:30:00,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(2018-03-17T20:00:00,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
I want to replace the date with timestamp.
I can make this conversion with date in the shell
date -d '2018-03-17T18:30:00' +%s%3N
1521325800000
I want this result:
(1521325800000,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(1521327600000,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(1521329400000,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(1521331200000,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
This data is stored in MySQL.
Is it possible to execute the date conversion with jq or another command like awk, sed, perl in a single command line?
Here is an all-jq solution that assumes the "Z" (UTC+0) timezone.
In brief, simply replace .T by:
((.T + "Z") | fromdate | tostring + "000")
To verify this, consider:
timestamp.jq
[splits("[(),]")]
| .[1] |= ((. + "Z")|fromdate|tostring + "000") # milliseconds
| .[1:length-1]
| "(" + join(",") + ")"
Invocation
jq -rR -f timestamp.jq input.txt
Output
(1521311400000,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(1521313200000,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(1521315000000,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(1521316800000,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
Here is an unportable awk solution. It is not portable because it relies on the system date command; on the system I'm using, the relevant invocation looks like: date -j -f "%Y-%m-%eT%T" STRING "+%s"
awk -F, 'BEGIN{OFS=FS}
NF==0 { next }
{ sub(/\(/,"",$1);
cmd="date -j -f \"%Y-%m-%eT%T\" " $1 " +%s";
cmd | getline $1;
$1=$1 "000"; # milliseconds
printf "%s", "(";
print;
}' input.txt
Output
(1521325800000,0.00012575,0.00012643,0.00012563,0.00012643,383839.45768188,48.465051)
(1521327600000,0.00012643,0.00012726,0.00012642,0.00012722,207757.18765437,26.30099514)
(1521329400000,0.00012726,0.00012779,0.00012698,0.00012779,97387.01596624,12.4229077)
(1521331200000,0.0001276,0.0001278,0.00012705,0.0001275,96850.15260027,12.33316229)
Solution with sed :
sed -e 's/(\([^,]\+\)\(,.*\)/echo "(\$(date -d \1 +%s%3N),\2"/g' | ksh
test :
<commande_curl> | sed -e 's/(\([^,]\+\)\(,.*\)/echo "(\$(date -d \1 +%s%3N),\2"/g' | ksh
or :
<commande_curl> > results_curl.txt
cat results_curl.txt | sed -e 's/(\([^,]\+\)\(,.*\)/echo "(\$(date -d \1 +%s%3N),\2"/g' | ksh
please help me with this problem, i have an array witch includes 1000 lines with number which are treated as strings and i want for all of them to reverse them one by one, my problem is how to reverse them because i have to use ksh or else with bash or something it would be so easy..... what i have now is this, but
rev="$rev${copy:$y:1}" doesnt work in ksh.
i=0
while [[ $i -lt 999 ]]
do
rev=""
var=${xnumbers[$i]}
copy=${var}
len=${#copy}
y=$(expr $len - 1)
while [[ $y -ge 0 ]]
do
rev="$rev${copy:$y:1}"
echo "y = " $y
y=$(expr $y - 1)
done
echo "i = " $i
echo "rev = " $rev
#xnumbers[$i]=$(expr $xnumbers[$i] "|" $rev)
echo "xum = " ${xnumbers[$i]}
echo "##############################################"
i=$(expr $i + 1)
done
I am not sure why we cannot use built in rev function.
$ echo 798|rev
897
You can also try:
$ echo 798 | awk '{ for(i=length;i!=0;i--)x=x substr($0,i,1);}END{print x}'
897
If, you can print the contents of the array to a file, you can then process the file with this awk oneliner.
awk '{s1=split($0,A,""); line=""; for (i=s1;i>0;i--) line=line A[i];print line}' file
Check this!!
other_var=`echo ${xnumbers[$i]} | awk '{s1=split($0,A,""); line=""; for (i=s1;i>0;i--) line=line A[i];print line}'`
I have tested this on Ubuntu with ksh, same results:
number="789"
other_var=`echo $number | awk '{s1=split($0,A,""); line=""; for (i=s1;i>0;i--) line=line A[i];print line}'`
echo $other_var
987
You could use cut, paste and rev together, just change printf to cat file.txt:
paste -d' ' <(printf "%s data\n" {1..100} | cut -d' ' -f1) <(printf "%s data\n" {1..100} | cut -d' ' -f2 |rev)
Or rev alone if, it's not a numbered file as clarified by the OP.
I am writing a script and which requires to calculate the difference between the two timestamps . I have done some search but didn't get a clue so far.
For Example say:
time1 = 20160314 10:16:27
time2 = 20160313 15:17:28
From the above I need to get result like below: difference is: "1 day 5 hours 1 minute 1 second"
Please help me in resolving this.
datediff() {
t1=$(date -d "$1" +%s)
t2=$(date -d "$2" +%s)
diff=$(( $t1 - $t2 ))
echo "Duration: $(( $diff / 86400 )) days $(($diff / 3600 )) hours $((( $diff % 3600) / 60)) minutes $(( $diff % 60 )) seconds"
}
date -d "$1" +%s converts given date to time stamp(counted from 1970).
More details on date conversion
http://www.gnu.org/software/coreutils/manual/html_node/Date-input-formats.html#Date-input-formats
example:
$ date -d 2013-07-05 +%m/%d/%Y
07/05/2013
$ date -d 07/05/2013 +%F
2013-07-05
The command you likely want to use is date, but its usage depends on your operating system. You've specified unix in the tags on your question, which isn't really specific enough for a single answer.
In general though, you'll calculate differences between dates by counting the seconds between them. So your method should be to figure out the unix epoch second for each of your dates, subtract to find the difference, and then print the results in whatever format suits you.
In Linux, GNU date (from "coreutils") has a -d option that can understand a number of date references.
$ time1="20160314 10:16:27"
$ date -d "$time1" '+%s'
1457964987
And you can calculate the difference in seconds between two dates like this:
echo "$(( $(date -d "$time1" '+%s') - $(date -d "$time2" '+%s') ))"
Converting that to your output format is a simple matter of division and remainders:
s=$(( $(date -d "$time1" '+%s') - $(date -d "$time2" '+%s') ))
printf "%d day(s) %d hours %d minute %d second\n" \
$(( s / 86400 )) \
$(( s % 86400 / 3600 )) \
$(( s % 3600 / 60 )) \
$(( s % 60 ))
In FreeBSD, on the other hand, the -d option does something completely different, and you'd use -f and -v to parse and adjust a date. For example:
$ time1="20160314 10:16:27"
$ date -j -f '%Y%m%d %T' "$time1" '+%s'
1457964987
Figuring out the difference looks similar to the Linux method then:
s=$(( $(date -j -f '%Y%m%d %T' "$time1" '+%s') - $(date -j -f '%Y%m%d %T' "$time2" '+%s') ))
You can then use the same printf command I've shown above to format your output.
I don't have details on how the date command works in Solaris or other operating systems, but you can man date from a shell to see if either of these strategies appears to be compatible with whatever operating system you're running.
At the moment I have my R function generate an awk script to load, selectively, a subset of a csv into fread.
The resulting awk string looks something like this:
tail -n +2 ../data/faults_main_only_dp_1_shopFlag.csv | parallel -k -q --block 500M --pipe awk -F , ' $5 > \"2013-01-01\" && $5 < \"2015-11-17\" && $2 ~ /^F59PHI$|^GP20ECO$|^GT42CU-ACE$/ && $20 ~ /^Disregard$|^EMD Work Item$|^Pending$|^Pre-Work Item$|^Road Failure$|^Unit Shopped$|^Watch$|^Work Item$|^NA$/ {print $2 \",\" $88 \",\" $17 \",\" $5 \",\" $9 \",\" $22 \",\" $3 \",\" $15 \",\" $14 } '
The thing is: as of recent, my csv is ordered by dates ($5), in descending order, so if the user enters a specific lower-bound date, and awk gets to that line, it make sense for it to stop. (I am not sure how that would work the parallelization I am doing above. Maybe there is a way to select only the part of the csv that is “above” the lower-bound of the date and then pass the resulting csv into the awk script.) Is there a way to do that?
Generate an awk program such as this which explitly exits on a condition (here if field 3 exceeds 4):
$2 > 3 & $2 < 10
$3 > 4 { exit}
Put the above in a file called myprog.awk, say, and assuming default separators run it with this (your awk may be called something else):
gawk -f myprog.awk mydata.dat
or put it on the command line but you will have to be careful regarding quoting depending on the shell you use:
$2 > 3 & $2 < 10; $3 > 4 { exit}
Using QNX Neutrino, I need to subtract hex-valued file names from their predecessors. The files are named by their creation time in hex. The following gets me a list of pure hex values, but I cannot subtrace them from eachother.
last=0
find /path/ -type f\(! iname ".*" \) -exec basename {} |
while read fname
do
current=$fname
echo "difference is $((current - last)) seconds
done
The find command gives me:
51b71f38
51b71f44
51b71f50
51b71f5c
51b71f74
I have tried using echo "ibase=16; $name" | bc but that only switches the value for the output. Is there a way to return an integer number which is the difference for these hex values?
May be something like this:
find /path/ -type f\(! iname ".*" \) -exec basename {} |
while read fname; do
last="$current"
current="$fname"
if [ $(( 0x$last )) -ne 0 ]; then
echo "difference is $(( 0x$current - 0x$last )) seconds"
fi
done
Test:
I used your find command as input from a file for the test.
$ cat ff
51b71f38
51b71f44
51b71f50
51b71f5c
51b71f74
$ while read fname; do last="$current" ; current="$fname" ; if [ $(( 0x$last )) -ne 0 ]; then echo "difference is $(( 0x$current - 0x$last )) seconds" ; fi ; done < ff
difference is 12 seconds
difference is 12 seconds
difference is 12 seconds
difference is 24 seconds
current=$(echo "ibase=16; $fname" |bc)
actually gives me the decimal value I needed inline