My log lines are like
2022-05-06T11:21:16.40-0400 [RTR/4] OUT saat-user-auth-web-edu.apps.stl.pcfdev00.mastercard.int - [2022-05-06T15:21:16.213506208Z] "POST /api/authenticate HTTP/1.1" 200 194 72 "-" "Apache-HttpClient/4.5.5 (Java/1.8.0_271)" "10.158.139.20:53588" "10.158.139.161:61084" x_forwarded_for:"10.158.139.20" x_forwarded_proto:"https" vcap_request_id:"48b5a532-6894-43d1-46eb-289a6829bbde" response_time:0.190081 gorouter_time:0.001345 app_id:"a12cb634-976a-40b1-99bd-00343d23f50b" app_index:"0" instance_id:"3a2a2756-d720-4235-4dd8-af7f" x_cf_routererror:"-" x_mc_correlation_id:"b3bc6ed5c8ff45a5bf7a65c49397ff33" x_correlation_id:"-" correlation_id:"-" x_b3_traceid:"5490ac6215197c9101243667ac074f49" x_b3_spanid:"01243667ac074f49" x_b3_parentspanid:"-" b3:"5490ac6215197c9101243667ac074f49-01243667ac074f49"
And when running:
goaccess /myLog.log \
log-format %v %h %^[%d:%t %^] "%r" %s %b "%R" "%u" \
date-format %d/%b/%Y \
time-format %H:%M:%S \
--date-spec hr \
--hour-spec min \
--invalid-requests /invalid-requests.log \
-o /report.html
and am getting:
==17507== Token 'UT' doesn't match specifier '%m'
==17507==
==17507== Format Errors - Verify your log/date/time format
Related
I am converting timestamps to EPOCH seconds in awk, getting incorrect output for repeated timestamps
Input:
20180614 00:00:00
20180614 00:00:23
20180614 22:45:00
20180614 22:45:21
20180614 00:00:00
20180614 00:00:23
Expected Output :
1528930800
1528930823
1528930800
1529012721
1528930800
1528930823
I did
awk '{ ts="\""$0"\""; ("date +%s -d "ts)| getline epochsec; print epochsec}'
output after running above command:
1528930800
1528930823
1529012700
1529012721
1529012721
1529012721
With GNU xargs:
xargs -I {} date +%s -d {} < file
Output:
1528927200
1528927223
1529009100
1529009121
1528927200
1528927223
A bit a shorter GNU awk version is using FIELDWIDTHS which is available from gawk-2.13 onwards:
awk 'BEGIN{FIELDWIDTHS="4 2 3 2 1 2 1 2"}{print mktime($1" "$2" "$3$4" "$6" "$8)}'
Since gawk-4.2 you can skip intervening fields:
awk 'BEGIN{FIELDWIDTHS="4 2 2 1:2 1:2 1:2"}{print mktime($1" "$2" "$3" "$4" "$5" "$6)}'
Or even shorter using FPAT
awk 'BEGIN{FPAT="[0-9][0-9]"}{print mktime($1$2" "$3" "$4" "$5" "$6" "$7)}
note: the usage of a single awk-mktime combination will be faster than anything which makes system calls to date as you do not constantly have to call a binary. With the awk mktime solution you call a single binary. Nonetheless, the xargs solution given by Cyrus is by far the most comfortable one.
You could use system function
$ awk '{system("date +%s -d \""$0"\"")}' ip.txt
1528914600
1528914623
1528996500
1528996521
1528914600
1528914623
Or use sed
$ sed 's/.*/date +%s -d "&"/e' ip.txt
1528914600
1528914623
1528996500
1528996521
1528914600
1528914623
As per AllAboutGetline article, you'll need
$ awk '{ ts="date +%s -d \""$0"\""; while ((ts|getline ep)>0) print ep; close(ts) }' ip.txt
1528914600
1528914623
1528996500
1528996521
1528914600
1528914623
However getline is not needed at all for this case and avoid using it unless you really need it and know how to use it
Using GNU awk mktime function:
awk '{gsub(":"," ",$2); print mktime(substr($1,1,4) " " substr($1,5,2) " " substr($1,7,2) " " $2)}' file
To add to Cyrus's answer, the following works on Mac OSX. Strangely, MAC has a different way of handling date-time format to epoch conversion.
xargs -I {} date -j -u -f "%a %b %d %T %Z %Y" {} +%s < file
I am converting timestamps to EPOCH seconds in awk, getting incorrect output for repeated timestamps
Input:
20180614 00:00:00
20180614 00:00:23
20180614 22:45:00
20180614 22:45:21
20180614 00:00:00
20180614 00:00:23
Expected Output :
1528930800
1528930823
1528930800
1529012721
1528930800
1528930823
I did
awk '{ ts="\""$0"\""; ("date +%s -d "ts)| getline epochsec; print epochsec}'
output after running above command:
1528930800
1528930823
1529012700
1529012721
1529012721
1529012721
With GNU xargs:
xargs -I {} date +%s -d {} < file
Output:
1528927200
1528927223
1529009100
1529009121
1528927200
1528927223
A bit a shorter GNU awk version is using FIELDWIDTHS which is available from gawk-2.13 onwards:
awk 'BEGIN{FIELDWIDTHS="4 2 3 2 1 2 1 2"}{print mktime($1" "$2" "$3$4" "$6" "$8)}'
Since gawk-4.2 you can skip intervening fields:
awk 'BEGIN{FIELDWIDTHS="4 2 2 1:2 1:2 1:2"}{print mktime($1" "$2" "$3" "$4" "$5" "$6)}'
Or even shorter using FPAT
awk 'BEGIN{FPAT="[0-9][0-9]"}{print mktime($1$2" "$3" "$4" "$5" "$6" "$7)}
note: the usage of a single awk-mktime combination will be faster than anything which makes system calls to date as you do not constantly have to call a binary. With the awk mktime solution you call a single binary. Nonetheless, the xargs solution given by Cyrus is by far the most comfortable one.
You could use system function
$ awk '{system("date +%s -d \""$0"\"")}' ip.txt
1528914600
1528914623
1528996500
1528996521
1528914600
1528914623
Or use sed
$ sed 's/.*/date +%s -d "&"/e' ip.txt
1528914600
1528914623
1528996500
1528996521
1528914600
1528914623
As per AllAboutGetline article, you'll need
$ awk '{ ts="date +%s -d \""$0"\""; while ((ts|getline ep)>0) print ep; close(ts) }' ip.txt
1528914600
1528914623
1528996500
1528996521
1528914600
1528914623
However getline is not needed at all for this case and avoid using it unless you really need it and know how to use it
Using GNU awk mktime function:
awk '{gsub(":"," ",$2); print mktime(substr($1,1,4) " " substr($1,5,2) " " substr($1,7,2) " " $2)}' file
To add to Cyrus's answer, the following works on Mac OSX. Strangely, MAC has a different way of handling date-time format to epoch conversion.
xargs -I {} date -j -u -f "%a %b %d %T %Z %Y" {} +%s < file
I am trying to find the day difference from the last time a password was rest to the current. So i have this so far. I am trying to just convert that date to days so i can subtract the current date in days - the last reset date in days and get a integer value.
$ LASTRESETDATE=$(echo $(passwd -s) | cut -d' ' -f3)
$ echo $LASTRESETDATE
12/15/16
Looking the the date verion i have there is no option for -d
$ date -h
date: illegal option -- h
Usage: date [-u] [+format]
date [-u] [mmddhhmm[[cc]yy]]
date [-a [-]sss.fff]
When I was unfortunate enough to have to slog through HP-UX idiosyncrasies, I'd often write my own little programs to do exactly what I wanted. Provided your hokey-pux machine is POSIXy, then you could do:
$ cat fromnow.c
#include <time.h>
#include <stdio.h>
int main(int argc, char *argv[]) {
struct tm parsed;
strptime(argv[1], argv[2], &parsed);
printf("%.0f\n", difftime(time(NULL), mktime(&parsed)));
return 0;
}
Compile it:
$ cc -o fromnow fromnow.c
Run it:
$ ./fromnow 5/13/13 %D
119451988
which will then calculate the number of seconds between the current time and the date "5/13/13" formatted in American style "%D". The first argument is the date, the second argument is the format specifier for parsing that date. See man strptime for options.
This could be made more general, or less, depending upon how much you need to do date calculations.
Not sure which version of ksh you're using.
[STEP 101] $ echo ${.sh.version}
Version AJM 93u+ 2012-08-01
[STEP 102] $ date=12/15/16
[STEP 103] $ [[ $date =~ (..)/(..)/(..) ]]
[STEP 104] $ date=20${.sh.match[3]}-${.sh.match[1]}-${.sh.match[2]}
[STEP 105] $ echo $date
2016-12-15
[STEP 106] $ old_seconds=$( printf '%(%s)T' $date )
[STEP 107] $ echo $old_seconds
1481797007
[STEP 108] $ now_seconds=$( printf '%(%s)T' )
[STEP 109] $ echo $now_seconds
1487845024
[STEP 110] $ (( diff = now_seconds - old_seconds ))
[STEP 111] $ echo $diff
6048017
[STEP 112] $ echo $(( diff / 86400 )) days
70 days
[STEP 113] $
When you have awk, you can calculate the difference with
LASTRESETDATE="12/15/16"
endd="$(date '+%m/%d/%y')"
awk -v startdate="${LASTRESETDATE}" -v enddate="${endd}" 'BEGIN {
split(startdate,A,"[/]");
T1=mktime(A[3] " " A[1] " " A[2] " 12 0 0");
split(enddate,B,"[/]");
T2=mktime(B[3] " " B[1] " " B[2] " 12 0 0");
diffdays=(T2-T1)/(3600*24)
printf("%s\n",diffdays);
}'
When you need this often, and you do not have (the right version of) awk, you can make a lookup-table on another system.
With awk:
startd="12/15/16"
endd="$(date '+%m/%d/%y')"
awk -v startdate="${startd}" -v enddate="${endd}" 'BEGIN {
split(startdate,A,"[/]");
T1=mktime(A[3] " " A[1] " " A[2] " 12 0 0");
split(enddate,B,"[/]");
T2=mktime(B[3] " " B[1] " " B[2] " 12 0 0");
linenr=1;
while (T1 < T2) {
printf("%d %s\n",linenr++, strftime("%m/%d/%y",T1));
T1+=3600*24;
}
}'
Of course you can make a list with Excel or another tool.
EDIT: removed var that I only used prototyping the solution.
I am writing a script and which requires to calculate the difference between the two timestamps . I have done some search but didn't get a clue so far.
For Example say:
time1 = 20160314 10:16:27
time2 = 20160313 15:17:28
From the above I need to get result like below: difference is: "1 day 5 hours 1 minute 1 second"
Please help me in resolving this.
datediff() {
t1=$(date -d "$1" +%s)
t2=$(date -d "$2" +%s)
diff=$(( $t1 - $t2 ))
echo "Duration: $(( $diff / 86400 )) days $(($diff / 3600 )) hours $((( $diff % 3600) / 60)) minutes $(( $diff % 60 )) seconds"
}
date -d "$1" +%s converts given date to time stamp(counted from 1970).
More details on date conversion
http://www.gnu.org/software/coreutils/manual/html_node/Date-input-formats.html#Date-input-formats
example:
$ date -d 2013-07-05 +%m/%d/%Y
07/05/2013
$ date -d 07/05/2013 +%F
2013-07-05
The command you likely want to use is date, but its usage depends on your operating system. You've specified unix in the tags on your question, which isn't really specific enough for a single answer.
In general though, you'll calculate differences between dates by counting the seconds between them. So your method should be to figure out the unix epoch second for each of your dates, subtract to find the difference, and then print the results in whatever format suits you.
In Linux, GNU date (from "coreutils") has a -d option that can understand a number of date references.
$ time1="20160314 10:16:27"
$ date -d "$time1" '+%s'
1457964987
And you can calculate the difference in seconds between two dates like this:
echo "$(( $(date -d "$time1" '+%s') - $(date -d "$time2" '+%s') ))"
Converting that to your output format is a simple matter of division and remainders:
s=$(( $(date -d "$time1" '+%s') - $(date -d "$time2" '+%s') ))
printf "%d day(s) %d hours %d minute %d second\n" \
$(( s / 86400 )) \
$(( s % 86400 / 3600 )) \
$(( s % 3600 / 60 )) \
$(( s % 60 ))
In FreeBSD, on the other hand, the -d option does something completely different, and you'd use -f and -v to parse and adjust a date. For example:
$ time1="20160314 10:16:27"
$ date -j -f '%Y%m%d %T' "$time1" '+%s'
1457964987
Figuring out the difference looks similar to the Linux method then:
s=$(( $(date -j -f '%Y%m%d %T' "$time1" '+%s') - $(date -j -f '%Y%m%d %T' "$time2" '+%s') ))
You can then use the same printf command I've shown above to format your output.
I don't have details on how the date command works in Solaris or other operating systems, but you can man date from a shell to see if either of these strategies appears to be compatible with whatever operating system you're running.
I need a Unix script which i can find the time difference between 2 times look like YYYYmmDDHHMMSS
Example
T1=20120624192555
T2=20120624204006
So the time difference will be =1:14:11
Any clue?
I'm not sure if you can get around specifying the date format, but here is a rather lengthy date solution:
# Get seconds since epoch
D1=$(date --date "${T1:0:4}-${T1:4:2}-${T1:6:2} ${T1:8:2}:${T1:10:2}: ${T1:12:2}" +%s)
D2=$(date --date "${T2:0:4}-${T2:4:2}-${T2:6:2} ${T2:8:2}:${T2:10:2}: ${T2:12:2}" +%s)
date --date "0 + $(( D2 - D1 )) seconds" +%T
See here for the ${VAR:OFF:LEN} reference.
Note: as mentioned by jrouquie below, this fails when exceeding 24 hours. This is a limitation of the date command, $(( D2 - D1 )) still gives the correct result in seconds though.
time1=`echo $1 | sed -r 's/([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})/\1-\2-\3 \4:\5:\6/'`
time2=`echo $2 | sed -r 's/([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})/\1-\2-\3 \4:\5:\6/'`
seconds1=`date --utc -d "$time1" +%s`
seconds2=`date --utc -d "$time2" +%s`
difference=`expr $seconds2 - $seconds1`
echo `expr $difference / 3600`:`expr "(" $difference / 60 ")" % 60`:`expr $difference % 60`
Note the use of --utc which prevents a bug with daylight saving time.
Caveats:
This script does not check input.
It assumes $2 > $1.
It could be rewritten to factor repeated code
EDIT: regarding the first comment, here is a version without the "-r" option to sed (-r means --regexp-extended, which here allowed to dispense with all the backslashes).
time1=`echo $1 | sed 's/\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3 \4:\5:\6/'`
time2=`echo $2 | sed 's/\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3 \4:\5:\6/'`
seconds1=`date --utc -d "$time1" +%s`
seconds2=`date --utc -d "$time2" +%s`
difference=`expr $seconds2 - $seconds1`
echo `expr $difference / 3600`:`expr "(" $difference / 60 ")" % 60`:`expr $difference % 60`
Here is a solution taking the best of the one by Thor/kjp and mine:
#!/bin/bash
function seconds_since_epoch() {
echo $(date --utc --date "${1:0:4}-${1:4:2}-${1:6:2}
${1:8:2}:${1:10:2}:${1:12:2}" +%s)
}
difference=$(( `seconds_since_epoch $2` - `seconds_since_epoch $1` ))
echo \
`expr $difference / 3600`:\
`expr "(" $difference / 60 ")" % 60`:\
`expr $difference % 60`