In my dataset a column contains Date of Births of many employees so many of them lies in the range 1960 to 1980. I am trying to format them using as.Date and in some of them the results are not per my expectation.
Example:
as.Date("7/1/61","%m/%d/%y")
i want it to return "1961-07-01" but it returns "2061-07-01".
Read:
?strptime # where all the formatting details are available
%y
Year without century (00–99). On input, values 00 to 68 are prefixed by 20 and 69 to 99 by 19 – that is the behavior specified by the 2004 and 2008 POSIX standards, but they do also say ‘it is expected that in a future version the default century inferred from a 2-digit year will change’.
So you need a regex to backdate and it's probably better to do as a string conversion before sending to as.Date:
dvec <- c("7/1/61", "7/1/79")
as.Date( sub("/(..$)", "/19\\1",dvec) , "%m/%d/%Y")
[1] "1961-07-01" "1979-07-01"
If this goes into production it will become an error waiting to happen when the age of your employees starts to creep above the last two digits of the current year.
Related
I am trying to use a dataset that uses Day of Year (DOY) and want to calculate dates:
To convert DOY format to a date I use:
as.Date(DOY, origin = "%Y-01-01")
But I seem to be getting a problem. For example: DOY = 121 for 2003
as.Date(121, origin = "2003-01-01")
[1] "2003-05-02"
This should be: "2003-05-01" (see here: https://asd.gsfc.nasa.gov/Craig.Markwardt/doy2003.html)
The code works for a leap year:
as.Date(121, origin = "2004-01-01")
[1] "2004-05-01"
(and shows up correctly here: https://asd.gsfc.nasa.gov/Craig.Markwardt/doy2004.html)
Problem persists for 2005.
Am I missing something obvious here or is there a problem in the code missing leap years?
as.Date will accept numeric data (the number of days since an epoch), but only if origin is supplied.
https://www.rdocumentation.org/packages/base/versions/3.6.1/topics/as.Date
So if you convert an integer to a date, the integer will count the number of days that have passed since the origin, which means the date will be the day after the last day has passed
An alternative would be to use the Julian date
as.Date('2005-121', '%Y-%j')
would yield
[1] "2005-05-01"
I'm importing time series data from a CSV file and one of the vectors/columns are dates in the format DD/MM/YYYY. Vector class is characters or factors if I chose the Strings as factors = True. I convert the imported file to a data frame and then run the following:
df$Date <- as.Date(df$Date , "%d/%m/%y")
I get no error message, but the dates are all messed up in the format YYYYMMDD and all the YYYY are the year 2020...
Before:
10/09/2009
11/09/2009
14/09/2009
After:
2020-09-10
2020-09-11
2020-09-14
You are using %y when it should be %Y. See the documentation here.
%y
Year without century (00–99). On input, values 00 to 68 are prefixed by 20 and 69 to 99 by 19 – that is the behaviour specified by the 2004 and 2008 POSIX standards, but they do also say ‘it is expected that in a future version the default century inferred from a 2-digit year will change’.
%Y
Year with century. Note that whereas there was no zero in the original Gregorian calendar, ISO 8601:2004 defines it to be valid (interpreted as 1BC): see http://en.wikipedia.org/wiki/0_(year). Note that the standards also say that years before 1582 in its calendar should only be used with agreement of the parties involved.
Try running the code again so that the data frame is not modified by any previous attempt but this time use
df$Date <- as.Date(df$Date , "%d/%m/%Y")
#Heroka is right.
If ever you need it you could also use posixct objects (they contain information of seconds)
Try this:
df$Date.time <- as.POSIXct(df$Date , format="%d/%m/%Y")
If you want the date and time in strings you can try the following:
df$Date.time <- format(as.POSIXct(df$Date , format="%d/%m/%Y"),format="%Y-%m-%d %H:%M")
or
df$Date <- format(as.POSIXct(df$Date , format="%d/%m/%Y"),format="%Y-%m-%d")
The %tw format in Stata has the form: 1960w1 which has no equivalent in R.
Therefore %tw dates must be post-processed.
Importing a .dta file into R, the date is an integer like 1304 (instead of 1985w5) or 1426 (instead of 1987w23). If it was a simple time series you could set a starting date as follows:
ts(df, start= c(1985,5), frequency=52)
Another possibility would be:
as.Date(Camp$date, format= "%Yw%W" , origin = "1985w5")
But if each row is not a single date, then you must convert it.
The package ISOweek is based on ISO-8601 with the form "1985-W05" and does not process the Stata %tw.
The Lubridate package does not work with this format. The week() returns the number of complete seven day periods that have occurred between the date and January 1st, plus one. week function
In Stata week 1 of any year starts on 1 January, whatever day of the week that is. Stata Documentation on Dates
In the format %W of Date in R the week starts as Monday as first day of the week.
From strptime %V is
the Week of the year as decimal number (00--53) as defined in ISO
8601. If the week (starting on Monday) containing 1 January has four or more days in the new year, then it is considered week 1. Otherwise,
it is the last week of the previous year, and the next week is week 1.
(Accepted but ignored on input.) Strptime
Larmarange noted on Github that Haven doesn't interpret dates properly:
months, week, quarter and halfyear are specific format from Stata,
respectively %tm, %tw, %tq and %th. I'm not sure that there are
corresponding formats available in R. So far they are imported as
integers.
Is there a way to convert Stata %tw to a date format R understands?
Here is an Stata file with dates
This won't be an answer in terms of R code, but it is commentary on Stata weeks that can't be fitted into a comment.
Strictly, dates in Stata are not defined by the display formats that make them intelligible to people. A date in Stata is always a numeric variable or scalar or macro defined with origin the first instance in 1960. Thus it is at best a shorthand to talk about %tw dates, etc. We can use display to see the effects of different date display formats:
. di %td 0
01jan1960
. di %tw 0
1960w1
. di %tq 0
1960q1
. di %td 42
12feb1960
. di %tw 42
1960w43
. di %tq 42
1970q3
A subtle point made explicit above is that changing the display format will not change what is stored, i.e. the numeric value.
Otherwise put, dates in Stata are not distinct data types; they are just integers made intelligible as dates by a pertinent display format.
The question presupposes that it was correct to describe some weekly dates in terms of Stata weeks. This seems unlikely, as I know no instance in which a body outside StataCorp uses the week rules of Stata, not only that week 1 always starts on 1 January, but also that week 52 always includes either 8 or 9 days and hence that there is never a week 53 in a calendar year.
So, you need to go upstream and find out what the data should have been. Failing some explanation, my best advice is to map the 52 weeks of each year to the days that start them, namely days 1(7)358 of each calendar year.
Stata weeks won't map one-to-one to any other scheme for defining weeks.
More in this article on Stata weeks
It's not completely clear what the question is but the year and week corresponding to 1304 are:
wk <- 1304
1960 + wk %/% 52
## [1] 1985
wk %% 52 + 1
## [1] 5
so assuming that the first week of the year is week 1 and starts on Jan 1st, the beginning of the above week is this date:
as.Date(paste(1960 + wk %/% 52, 1, 1, sep = "-")) + 7 * (wk %% 52)
## [1] "1985-01-29"
This question already has answers here:
Read csv with dates and numbers
(3 answers)
Closed 9 years ago.
I am working on "Localization Data for Person Activity Data Set" dataset from UCI and in this data set there is a column of date and time(both in one column) with following format:
27.05.2009 14:03:25:777
27.05.2009 14:03:25:183
27.05.2009 14:03:25:210
27.05.2009 14:03:25:237
...
I am wondering if there is anyway to convert this column to timestamp using R.
First of all, we need to substitute the colon separating the milliseconds from the seconds to a dot, otherwise the final step won't work (thanks to Dirk Eddelbuettel for this one). Since in the end R will use the separators it wants, to be quicker, I'll just go ahead and substitute all the colons for dots:
x <- "27.05.2009 14:03:25:777" # this is a simplified version of your data
y <- gsub(":", ".", x) # this is your vector with the aforementioned substitution
By the way, this is how your vector should look after gsub:
> y
[1] "27.05.2009 14.03.25.777"
Now, in order to have it show the milliseconds, you first need to adjust an R option and then use a function called strptime, which will convert your date vector to POSIXlt (an R-friendly) format. Just do the following:
> options(digits.secs = 3) # this tells R you want it to consider 3 digits for seconds.
> strptime(y, "%d.%m.%Y %H:%M:%OS") # this finally formats your vector
[1] "2009-05-27 14:03:25.777"
I've learned this nice trick here. This other answer also says you can skip the options setting and use, for example, strptime(y, "%d.%m.%Y %H:%M:%OS3"), but it doesn't work for me. Henrik noted that the function's help page, ?strptime states that the %OS3 bit is OS-dependent. I'm using an updated Ubuntu 13.04 and using %OS3 yields NA.
When using strptime (or other POSIX-related functions such as as.Date), keep in mind some of the most common conversions used (edited for brevity, as suggested by DWin. Complete list at strptime):
%a Abbreviated weekday name in the current locale.
%A Full weekday name in the current locale.
%b Abbreviated month name in the current locale.
%B Full month name in the current locale.
%d Day of the month as decimal number (01–31).
%H Hours as decimal number (00–23). Times such as 24:00:00 are accepted for input.
%I Hours as decimal number (01–12).
%j Day of year as decimal number (001–366).
%m Month as decimal number (01–12).
%M Minute as decimal number (00–59).
%p AM/PM indicator in the locale. Used in conjunction with %I and not with %H.
`%S Second as decimal number (00–61), allowing for up to two leap-seconds (but POSIX-compliant implementations will ignore leap seconds).
%U Week of the year as decimal number (00–53) using Sunday as the first day 1 of the week (and typically with the first Sunday of the year as day 1 of week 1). The US convention.
%w Weekday as decimal number (0–6, Sunday is 0).
%W Week of the year as decimal number (00–53) using Monday as the first day of week (and typically with the first Monday of the year as day 1 of week 1). The UK convention.
%y Year without century (00–99). On input, values 00 to 68 are prefixed by 20 and 69 to 99 by 19
%Y Year with century. Note that whereas there was no zero in the original Gregorian calendar, ISO 8601:2004 defines it to be valid (interpreted as 1BC)
I have a bunch of dates that I am parsing that are in the form "%m/%d/%y". as.Date(dates, format = "%m/%d/%y") converts a date like "1/01/64" to "2064-01-01" but I need that to be "1964-01-01." I suppose I can find instances where the year is in the future and then subtract a century, but that seems a little ridiculous.
Dates are stored internal as integer days, so there is only such formatting at the time of input or output. As for input without century information I think you are out of luck. Here's what ?strptime says about the %y format spec: "On input, values 00 to 68 are prefixed by 20 and 69 to 99 by 19 – that is the behaviour specified by the 2004 and 2008 POSIX standards, but they do also say ‘it is expected that in a future version the default century inferred from a 2-digit year will change’."
as.Date( "01/01/64", "%m/%d/%y", origin="1970-01-01") -100*365.25
#[1] "1964-01-01"
It might be possible to start a bar fight about programmers who allow removal of century information given that Y2K is so recent in the past.
Since the default is to assume year 00-68 is 2000-2068, it is certainly possible to create an as.Dateshift
Another way to fix the dates is to change all years that occur in the future (relative to today's date using Sys.Date()) as starting with 19 instead of 20.
dates=as.Date(c("01/01/64", "12/31/15"))
# [1] "2064-01-01" "2015-12-31" ## contains an incorrect date
## Now correct the dates that havn't yet occurred
as.Date(ifelse(dates > Sys.Date(), format(dates, "19%y-%m-%d"), format(dates)))
#[1] "1964-01-01" "2015-12-31"