Related
As we all know, date parsing in Go has it's quirks*.
However, I have now come up against needing to parse a datetime string in CCYY-MM-DDThh:mm:ss[.sss...] to a valid date in Go.
This CCYY format is a format that seems to be ubiquitous in astronomy, essentially the CC is the current century, so although we're in 2022, the century is the 21st century, meaning the date in CCYY format would be 2122.
How do I parse a date string in this format, when we can't specify a coded layout?
Should I just parse in that format, and subtract one "century" e.g., 2106 becomes 2006 in the parsed datetime...?
Has anyone come up against this niche problem before?
*(I for one would never have been able to remember January 2nd, 3:04:05 PM of 2006, UTC-0700 if it wasn't the exact time of my birth! I got lucky)
The time package does not support parsing centuries. You have to handle it yourself.
Also note that a simple subtraction is not enough, as e.g. the 21st century takes place between January 1, 2001 and December 31, 2100 (the year may start with 20 or 21). If the year ends with 00, you do not have to subtract 100 years.
I would write a helper function to parse such dates:
func parse(s string) (t time.Time, err error) {
t, err = time.Parse("2006-01-02T15:04:05[.000]", s)
if err == nil && t.Year()%100 != 0 {
t = t.AddDate(-100, 0, 0)
}
return
}
Testing it:
fmt.Println(parse("2101-12-31T12:13:14[.123]"))
fmt.Println(parse("2122-10-29T12:13:14[.123]"))
fmt.Println(parse("2100-12-31T12:13:14[.123]"))
fmt.Println(parse("2201-12-31T12:13:14[.123]"))
Which outputs (try it on the Go Playground):
2001-12-31 12:13:14.123 +0000 UTC <nil>
2022-10-29 12:13:14.123 +0000 UTC <nil>
2100-12-31 12:13:14.123 +0000 UTC <nil>
2101-12-31 12:13:14.123 +0000 UTC <nil>
As for remembering the layout's time:
January 2, 15:04:05, 2006 (zone: -0700) is a common order in the US, and in this representation parts are in increasing numerical order: January is month 1, 15 hour is 3PM, year 2006 is 6. So the ordinals are 1, 2, 3, 4, 5, 6, 7.
I for one would never have been able to remember January 2nd, 3:04:05 PM of 2006, UTC-0700 if it wasn't the exact time of my birth! I got lucky.
The reason for the Go time package layout is that it is derived from the Unix (and Unix-like) date command format. For example, on Linux,
$ date
Fri Apr 15 08:20:43 AM EDT 2022
$
Now, count from left to right,
Month = 1
Day = 2
Hour = 3 (or 15 = 12 + 3)
Minute = 4
Second = 5
Year = 6
Note: Rob Pike is an author of The Unix Programming Environment
These lines give the date and time in UTC:
t:timedate(absolute_real_time() - (10*3600));
t0:substring(t,1,20);
t1:concat(substring(t,12,17), " ", substring(t,9,11), "/", substring(t,6,8), "/", substring(t,1,5));
t2:concat(substring(t,1,5), substring(t,6,8), substring(t,9,11), substring(t,12,14), substring(t,15,17), substring(t,18,20));
I know that '?\*autoconf\-version\*;' can give the Maxima version number, so maybe there is some undocumented way to get the local time.
Otherwise are there any ready-made functions that can convert
UTC time to local time given conditions for start/end of daylight saving time
e.g. UTC time to UK time (which is GMT/BST depending on the time of year)?
It's not clear to me exactly what you need, but perhaps the following helps. By the way, do you really need to extract the parts (year, month, day, etc)? If so, it might be more convenient to work directly in Lisp. See DECODE-UNIVERSAL-TIME at the Common Lisp Hyperspec (a web search will find it).
The timedate now (in the just-released Maxima 5.39) accepts an optional argument which is the time zone offset, in hours (plus or minus). The time zone offset may be noninteger (e.g. 2.5). Offset 0 indicates UTC. If the offset is omitted, the time is formatted in the local time zone.
(%i5) t:absolute_real_time();
(%o5) 3691202499
(%i6) timedate (t, 0);
(%o6) 2016-12-20 06:01:39+00:00
(%i7) timedate (t);
(%o7) 2016-12-19 22:01:39-08:00
Note that the daylight saving time flag is applied at the "time of the time". Here is a time from next summer, when daylight saving time is in effect.
(%i8) timedate (t + 6*30.25*24*3600);
(%o8) 2017-06-19 11:01:39-07:00
The parse_timedate function has also been (in Maxima 5.39) updated to recognize time zone offsets.
(%i9) parse_timedate ("2016-12-19 22:01:39-08:00");
(%o9) 3691202499
As with timedate if the offset is omitted, it is assumed to be in the local time zone.
(%i10) parse_timedate ("2016-12-19 22:01:39");
(%o10) 3691202499
Note also that Maxima does not recognize any symbolic time zone indicators such as "UTC", "GMT", "EDT", "America/New_York", etc., only numerical time zone offsets.
To clarify the problem, before revealing the solution:
these are the steps that I take in Maxima v5.30
to get the time in UTC, in a readable format:
Note: When I use Maxima v5.30 (in the UK),
for some unknown reason, the time is always UTC adjusted
by 10 hours, and does not adjust for DST.
/* 1st Jan 2017 12 noon: */
timedate(3692260800); /* "2017-01-01 22:00:00+10:00" */
timedate(3692260800-10*3600); /* "2017-01-01 12:00:00+10:00" */
substring(timedate(3692260800-10*3600),1,20); /* "2017-01-01 12:00:00" */
Note: timedate works better/differently in later versions of Maxima,
but some institutions recommend installing a specific version of Maxima.
Sometimes I want the date in the form: 'yyyyMMddHHmmss'.
A function for this is:
SecUTCToDate(vSec,vHour):=
block([d1,d2],
d1:timedate(vSec+vHour*3600),
d2:concat(substring(d1,1,5), substring(d1,6,8), substring(d1,9,11), substring(d1,12,14), substring(d1,15,17), substring(d1,18,20)),
parse_string(d2)
);
Note: [d1,d2] keeps those variables local to within the block, and not global.
To get the local time I have to add on hours based on my time zone (0 in the UK), and DST.
To calculate whether a time is within the DST period requires an individual function per time zone: in the UK, and many European countries, one such function is:
/* correct for the years 1900-2200 inclusive */
SecUTCIsDSTUK(vSec):=
block([vLeap,vDaysMar25,vDaysOct25,vWDayMar25,vWDayOct25,vRange1,vRange2],
vYear : parse_string(substring(timedate(vSec),1,5)),
vLeap : floor((vYear-1900)/4), if (vYear>=2100) then vLeap : vLeap-1,
vDaysMar25 : (vYear-1900)*365 + vLeap + 83,
vDaysOct25 : vDaysMar25 + 214,
vWDayMar25 : mod(vDaysMar25+1,7),
vWDayOct25 : mod(vDaysOct25+1,7),
vRange1 : (vDaysMar25+mod(-vWDayMar25,7))*86400 + 3600,
vRange2 : (vDaysOct25+mod(-vWDayOct25,7))*86400 + 3600,
if ((vSec >= vRange1) and (vSec < vRange2)) then 1 else 0);
You can create a mac file with such a function, and call up the the function when needed, e.g.:
load("C:\\MyFolder\\MyFile.mac");
SecUTCIsDSTUK(absolute_real_time());
SecUTCIsDSTUK(absolute_real_time()+86400*180);
thank you for your helpful response,
results (v. 5.39.0) (works fine, param 2 omitted gives local time, param 2 as 0 gives UTC):
t:3691202499;
timedate (t);
timedate (t + 6*30.25*24*3600);
timedate (t + 6*30*24*3600);
timedate (t, 0);
timedate (t + 6*30.25*24*3600, 0);
timedate (t + 6*30*24*3600, 0);
:lisp (decode-universal-time 3691202499)
:lisp (decode-universal-time 3691202499 0)
:lisp (decode-universal-time 3706754499)
:lisp (decode-universal-time 3706754499 0)
3691202499
"2016-12-20 06:01:39+00:00"
"2017-06-19 19:01:39+01:00"
"2017-06-18 07:01:39+01:00"
"2016-12-20 06:01:39+00:00"
"2017-06-19 18:01:39+00:00"
"2017-06-18 06:01:39+00:00"
39 1 6 20 12 2016 1 NIL 0
39 1 6 20 12 2016 1 NIL 0
39 1 7 18 6 2017 6 T 0
39 1 6 18 6 2017 6 NIL 0
results (v. 5.30.0) (it seems param 2 omitted gives UTC+10, with no daylight saving time):
(if this is true, I would have to find another way to get local time, possibly by Common LISP commands)
t:3691202499;
timedate (t);
timedate (t + 6*30.25*24*3600);
timedate (t + 6*30*24*3600);
:lisp (decode-universal-time 3691202499)
:lisp (decode-universal-time 3691202499 0)
:lisp (decode-universal-time 3706754499)
:lisp (decode-universal-time 3706754499 0)
3691202499
"2016-12-20 16:01:39+10:00"
"2017-06-20 04:01:39.0+10:00"
"2017-06-18 16:01:39+10:00"
39 1 16 20 12 2016 1 NIL -10
39 1 6 20 12 2016 1 NIL 0
39 1 16 18 6 2017 6 NIL -10
39 1 6 18 6 2017 6 NIL 0
(I can see that the timedate and decode-universal-time functions
have key differences between Maxima versions)
thank you for the website mention,
CLHS: Section The Environment Dictionary
http://clhs.lisp.se/Body/c_enviro.htm
is there a list of LISP commands that work in Maxima?
the main reason for the datestamp concerns:
to produce datestamps for filenames such as 'z title yyyymmddhhmmss.txt',
or for friendly dates inside those files such as 'hh:mm dd/mm/yyyy',
the string manipulation method was the simplest method
that I could successfully code (I don't explicitly need to extract individual d m y etc)
I have pattern namely QUARTERDATE and FILENAME inside the file.
Both will have some value as in below eg.
My requirement is, I should rename the file like FILENAME_QUARTERDATE.
My file(myfile.txt) will be as below:
QUARTERDATE: 03/31/14 - 06/29/14
FILENAME : LEAD
field1 field2
34567
20.0 5,678
20.0 5,678
20.0 5,678
20.0 5,678
20.0 5,678
I want the the file name to be as LEAD_201402.txt
Date range in the file is for Quarter 2, so i given as 201402.
Thanks in advance for the replies.
newname=$(awk '/QUARTERDATE/ { split($4, d, "/");
quarter=sprintf("%04d%02d", 2000+d[3], int((d[1]-1)/3)+1); }
/FILENAME/ { fn = $3; print fn "_" quarter; exit; }' "$file")
mv "$file" "$newname"
How is a quarter defined?
As noted in comments to the main question, the problem is as yet ill-defined.
What data would appear in the previous quarter's QUARTERDATE line? Could Q1 ever start with a date in December of the previous year? Could the end date of Q2 ever be in July (or Q1 in April, or Q3 in October, or Q4 in January)? Since the first date of Q2 is in March, these alternatives need to be understood. Could a quarter ever start early and end late simultaneously (a 14 week quarter)?
To which the response was:
QUARTERDATE of Q2 will start as 1st Monday of April and end as last Sunday of June.
Which triggered a counter-response:
2014-03-31 is a Monday, but hardly a Monday in April. What this mainly means is that your definition of a quarter is, as yet, not clear. For example, next year, 2015-03-30 is a Monday, but 'the first Monday in April' is 2015-04-06. The last Sunday in March 2015 is 2015-03-29. So which quarter does the week (Mon) 2015-03-30 to (Sun) 2015-04-05 belong to, and why? If you don't know (both how and why), we can't help you reliably.
Plausible working hypothesis
The lessons of Y2K have been forgotten already (why else are two digits used for the year, dammit!).
Quarters run for an integral number of weeks.
Quarters start on a Monday and end on a Sunday.
Quarters remain aligned with the calendar quarters, rather than drifting around the year. (There are 13 weeks in 91 days, and 4 such quarters in a year, but there's a single extra day in an ordinary year and two extra in a leap year, which mean that occasionally you will get a 14-week quarter, to ensure things stay aligned.)
The date for the first date in a quarter will be near 1st January, 1st April, 1st July or 1st October, but the month might be December, March (as in the question), June or September.
The date for the last date in a quarter will be near 31st March, 30th June, 30th September, 31st December, but the month might be April, July, October or January.
By adding 1 modulo 12 (values in the range 1..12, not 0..11) to the start month, you should end up with a month firmly in the calendar quarter.
By subtracting 1 modulo 12 (values in the range 1..12 again) to the end month, you should end up with a month firmly in calendar quarter.
If the data is valid, the 'start + 1' and 'end - 1' months should be in the same quarter.
The early year might be off-by-one if the start date is in December (but that indicates Q1 of the next year).
The end year might be off-by-one if the end date is in January (but that indicates Q4 of the prior year).
More resilient code
Despite the description above, it is possible to write code that detects the quarter despite any or all of the idiosyncrasies of the quarter start and end dates. This code borrows a little from Barmar's answer, but the algorithm is more resilient to the vagaries of the calendar and the quarter start and end dates.
#!/bin/sh
awk '/QUARTERDATE/ {
split($2, b, "/")
split($4, e, "/")
if (b[1] == 12) { q = 1; y = e[3] }
else if (e[1] == 1) { q = 4; y = b[3] }
else
{
if (b[3] != e[3]) {
print "Year mismatch (" $2 " vs " $4 ") in file " FILENAME
exit 1
}
m = int((b[1] + e[1]) / 2)
q = int((m - 1) / 3) + 1
y = e[3]
}
quarter = sprintf("%.4d%.2d", y + 2000, q)
}
/FILENAME/ {
print $3 "_" quarter
# exit
}' "$#"
The calculation for m adds the start month plus one to the end month minus one and then does integer division by two. With the extreme cases already taken care of, this always yields a month number that is in the correct quarter.
The comment in front of the exit associated with FILENAME allows testing more easily. When processing each file separately, as in Barmar's example, that exit is an important optimization. Note that the error message gives an empty file name if the input comes from standard input. (Offhand, I'm not sure how to print the error message to standard error rather than standard output, other than by a platform-specific technique such as print "message" > "/dev/stderr" or print "message" > "/dev/fd/2".)
Given this sample input data (semi-plausible start and end dates for 6 quarters from 2014Q1 through 2015Q2):
QUARTERDATE: 12/30/13 - 03/30/14
FILENAME : LEAD
QUARTERDATE: 03/31/14 - 06/29/14
FILENAME : LEAD
QUARTERDATE: 06/30/14 - 09/28/14
FILENAME : LEAD
QUARTERDATE: 09/29/14 - 12/28/14
FILENAME : LEAD
QUARTERDATE: 12/29/14 - 03/29/15
FILENAME : LEAD
QUARTERDATE: 03/30/15 - 06/29/15
FILENAME : LEAD
The output from this script is:
LEAD_201401
LEAD_201402
LEAD_201403
LEAD_201404
LEAD_201501
LEAD_201502
You can juggle the start and end dates of the quarters within reason and you should still get the required output. But always be wary of calendrical calculations; they are almost invariably harder than you expect.
The database query is returning the date and time in a unix timestamp. I am trying to use clojure to convert this into human readable time and then split the date and time into two separate columns under date and time.
Take a look at clj-time
For example here is how you would convert a (long) timestamp to DateTime:
user=> (clj-time.coerce/from-long 893362442345)
#<DateTime 1998-04-23T20:14:02.345Z>
Here is how to "split" it by date/time portions:
user=> (unparse (formatters :date) (clj-time.coerce/from-long 893362442345))
"1998-04-23"
user=> (unparse (formatters :time) (clj-time.coerce/from-long 893362442345))
"20:14:02.345Z"
Depending on how you need the result to be formatted, you can choose from many different built in formatters:
user=> (show-formatters)
:basic-date 20130828
:basic-date-time 20130828T175957.850Z
:basic-date-time-no-ms 20130828T175957Z
:basic-ordinal-date 2013240
:basic-ordinal-date-time 2013240T175957.850Z
:basic-ordinal-date-time-no-ms 2013240T175957Z
:basic-t-time T175957.850Z
:basic-t-time-no-ms T175957Z
:basic-time 175957.850Z
:basic-time-no-ms 175957Z
:basic-week-date 2013W353
:basic-week-date-time 2013W353T175957.850Z
:basic-week-date-time-no-ms 2013W353T175957Z
:date 2013-08-28
:date-hour 2013-08-28T17
:date-hour-minute 2013-08-28T17:59
:date-hour-minute-second 2013-08-28T17:59:57
:date-hour-minute-second-fraction 2013-08-28T17:59:57.850
:date-hour-minute-second-ms 2013-08-28T17:59:57.850
:date-time 2013-08-28T17:59:57.850Z
:date-time-no-ms 2013-08-28T17:59:57Z
:hour 17
:hour-minute 17:59
:hour-minute-second 17:59:57
:hour-minute-second-fraction 17:59:57.850
:hour-minute-second-ms 17:59:57.850
:mysql 2013-08-28 17:59:57
:ordinal-date 2013-240
:ordinal-date-time 2013-240T17:59:57.850Z
:ordinal-date-time-no-ms 2013-240T17:59:57Z
:rfc822 Wed, 28 Aug 2013 17:59:57 +0000
:t-time T17:59:57.850Z
:t-time-no-ms T17:59:57Z
:time 17:59:57.850Z
:time-no-ms 17:59:57Z
:week-date 2013-W35-3
:week-date-time 2013-W35-3T17:59:57.850Z
:week-date-time-no-ms 2013-W35-3T17:59:57Z
:weekyear 2013
:weekyear-week 2013-W35
:weekyear-week-day 2013-W35-3
:year 2013
:year-month 2013-08
:year-month-day 2013-08-28
or built a custom one:
(def custom-formatter (formatter "yyyyMMdd"))
The month format specifier doesn't seem to work.
from datetime import datetime
endDate = datetime.strptime('10 3 2011', '%j %m %Y')
print endDate
2011-01-10 00:00:00
endDate = datetime.strptime('21 5 1987', '%j %m %Y')
print endDate
1987-01-21 00:00:00
Now, according to the manual the manual:
%m = Month as a decimal number [01,12].
So, what am I missing, other than the hair I've pulled out trying to understand why my django __filter queries return nothing (the dates going in aren't valid!)? I've tried 03 and 05 to no avail.
Versions of things, platform, architecture et al:
$ python --version
Python 2.7
$ python3 --version
Python 3.1.2
$ uname -r
2.6.35.11-83.fc14.x86_64 (that's Linux/Fedora 14/64-bit).
You can't mix the %j with others format code like %m because if you look in the table that you linked %j is the Day of the year as a decimal number [001,366] so 10 correspondent to the 10 day of the year so it's 01 of January ...
So you have just to write :
>>> datetime.strptime('10 2011', '%j %Y')
datetime.datetime(2011, 1, 10, 0, 0)
Else if you you wanted to use 10 as the day of the mount you should do :
>>> datetime.strptime('10 3 2011', '%d %m %Y')
datetime.datetime(2011, 3, 10, 0, 0)
Isn't %j the "day of year" parser, which may be forcing strptime to choose January 21, overriding the %m rule?
%j specifies a day of the year. It's impossible for the 10th day of the year, January 10, to occur in March, so your month specification is being ignored. Garbage In, Garbage Out.