How can I fix corrupted dates in R? - r

Question

I have a dataset as follows:
19/9/1997
22/9/1997
23/9/1997
24/9/1997
25/9/1997
26/9/1997
29/9/1997
30/9/1997
35440
35471
35499
35591
35621
35652
35683
35713
13/10/1997
14/10/1997
15/10/1997
16/10/1997
17/10/1997
20/10/1997
21/10/1997
22/10/1997
23/10/1997
24/10/1997
27/10/1997
28/10/1997
29/10/1997
30/10/1997
31/10/1997
35500
35531
35561
35592
35622
35714
35745
35775
13/11/1997
14/11/1997
17/11/1997
18/11/1997
19/11/1997
20/11/1997
21/11/1997
24/11/1997 ...
The Data that should be here are (for reproduction as requested)
19/9/1997
22/9/1997
23/9/1997
24/9/1997
25/9/1997
26/9/1997
29/9/1997
30/9/1997
10/01/1997
10/02/1997
10/03/1997
10/06/1997
10/07/1997
10/08/1997
10/09/1997
10/10/1997
13/10/1997
14/10/1997
15/10/1997
16/10/1997
17/10/1997
20/10/1997
21/10/1997
22/10/1997
23/10/1997
24/10/1997
27/10/1997
28/10/1997
29/10/1997
30/10/1997
31/10/1997
11/03/1997
11/04/1997
11/05/1997
11/06/1997
11/07/1997
11/10/1997
11/11/1997
11/12/1997
13/11/1997
14/11/1997
17/11/1997
18/11/1997
19/11/1997
20/11/1997
21/11/1997
24/11/1997
I have 5,149 rows of dates where there are numbers in places of dates. I tried fixing the missing dates with this:
ATTEMPT 1 BEFORE REVISION:
rm (list = ls(all=TRUE))
graphics.off()
library(readxl)
Dates <- read_excel("F:/OneDrive - University of Tasmania/Mardi Meetings/Dataset/Dates.xlsx")
x<-Dates[,1]
library(date)
library(datetime)
ans <- Reduce(function(prev, curr) {
f1 <- as.Date(curr, "%d/%m/%Y")
f2 <- as.Date(curr, "%m/%d/%Y")
if (is.na(f1)) return(f2)
if (is.na(f2)) return(f1)
if (prev < f1 && prev < f2) return(min(f1, f2))
if (prev < f1) return(f1)
if (prev < f2) return(f2)
}, x[-1], init=as.Date(x[1], "%d/%m/%Y"), accumulate=TRUE)
as.Date(ans, origin="1970-01-01")
But I am getting the following error:
+ }, x[-1], init=as.Date(x[1], "%d/%m/%Y"), accumulate=TRUE)
Error in Reduce(function(prev, curr) { : object 'x' not found
>
> as.Date(ans, origin="1970-01-01")
Error in as.Date(ans, origin = "1970-01-01") : object 'ans' not found
Any suggestions will be highly appreciated.
OK AS PER ADVICE I REVISED THE CODE ATTEMPT 2 AFTER REVISION
> rm (list = ls(all=TRUE))
> graphics.off()
> library(readxl)
> Dates <- read_excel("F:/OneDrive - University of Tasmania/Mardi Meetings/Dataset/Dates.xlsx")
> dput(head(Dates))
structure(list(Date = c("33274", "33302", "33394", "33424", "33455",
"33486")), row.names = c(NA, -6L), class = c("tbl_df", "tbl",
"data.frame"))
> x<-Dates[[1]]
> library(date)
> library(datetime)
Attaching package: ‘datetime’
The following object is masked from ‘package:date’:
as.date
> dates <- as.Date(x, format="%d/%m/%Y")
> dput(head(dates))
structure(c(NA_real_, NA_real_, NA_real_, NA_real_, NA_real_,
NA_real_), class = "Date")
> head(dates,10)
[1] NA NA NA NA NA NA NA
[8] "1991-05-13" "1991-05-14" "1991-05-15"
As you can see I have lost the corrupted dates completely
Today on 28th I tried again
> rm (list = ls(all=TRUE))
> graphics.off()
> library(readxl)
> Dates <- read_excel("F:/OneDrive - University of Tasmania/Mardi Meetings/Dataset/Dates.xlsx")
> x<-Dates[[1]]
>
> library(date)
> library(datetime)
Attaching package: ‘datetime’
The following object is masked from ‘package:date’:
as.date
> formats <- c("%m/%d/%Y", "%d/%m/%Y", "%Y/%m/%d")
> dates <- as.Date(rep(NA, length(x)))
> for (fmt in formats) {
+ nas <- is.na(dates)
+ dates[nas] <- as.Date(as.integer(x[nas], format=fmt))
+ }
Error in as.Date.numeric(as.integer(x[nas], format = fmt)) :
'origin' must be supplied
In addition: Warning message:
In as.Date(as.integer(x[nas], format = fmt)) : NAs introduced by coercion
> dates <- as.Date(x, format="%d/%m/%Y")
> head(dates)
[1] NA NA NA NA NA NA
> head(dates, 10)
[1] NA NA NA NA NA NA NA
[8] "1991-05-13" "1991-05-14" "1991-05-15"

Answer 1

You need none of the packages you've loaded, nor do you need to use Reduce, as functions we're using here are naturally "vectorized".
Here's a sample of your data. (A good question includes data in an easily copied format such as this.)
x <- c("19/9/1997", "22/9/1997", "23/9/1997", "24/9/1997", "25/9/1997",
"26/9/1997", "29/9/1997", "30/9/1997",
"35440", "35471", "35499", "35591", "35621",
"35652", "35683", "35713")
dates <- as.Date(x, format="%d/%m/%Y")
dates
# [1] "1997-09-19" "1997-09-22" "1997-09-23" "1997-09-24" "1997-09-25"
# [6] "1997-09-26" "1997-09-29" "1997-09-30" NA NA
# [11] NA NA NA NA NA
# [16] NA
Not surprisingly, the second-half of the dates are not recognized given format="%d/%m/%Y". You mentioned the use of "%m/%d/%Y" in your question, so we can (1) do a literal second-pass for this format (un-utilized with this example, but still relevant for your work?):
dates[is.na(dates)] <- as.Date(x[is.na(dates)], format="%m/%d/%Y")
where [is.na(dates)] only works on the un-converted elements.
(2) If we have more than one other format, you can always use a vector of them and loop over them. (For this, I'll start over, since this loop would replace/augment the first steps above.)
formats <- c("%m/%d/%Y", "%d/%m/%Y", "%Y/%m/%d")
dates <- as.Date(rep(NA, length(x)))
for (fmt in formats) {
nas <- is.na(dates)
dates[nas] <- as.Date(x[nas], format=fmt)
}
dates
# [1] "1997-09-19" "1997-09-22" "1997-09-23" "1997-09-24" "1997-09-25"
# [6] "1997-09-26" "1997-09-29" "1997-09-30" NA NA
# [11] NA NA NA NA NA
# [16] NA
This still leaves us with NAs for the integer-looking ones. For these you need to specify the origin= to be able to figure it out (as well as converting to an integer). R typically works with an origin of "1970-01-01", which you can confirm with
as.integer(Sys.Date())
# [1] 17787
Sys.Date() - 17787
# [1] "1970-01-01"
but it appears that your dates have an origin of "1900-01-01", I think that's Excel's default storage of dates (but it doesn't matter here):
x[9] # the first integer-looking element
# [1] "35440"
dates[1] - as.integer(x[9])
# [1] "1900-09-08"
(I'm assuming that your dates are from the same relative period of time.)
From here:
nas <- is.na(dates)
dates[nas] <- as.Date(as.integer(x[nas]), origin="1900-01-01")
dates
# [1] "1997-09-19" "1997-09-22" "1997-09-23" "1997-09-24" "1997-09-25"
# [6] "1997-09-26" "1997-09-29" "1997-09-30" "1997-01-12" "1997-02-12"
# [11] "1997-03-12" "1997-06-12" "1997-07-12" "1997-08-12" "1997-09-12"
# [16] "1997-10-12"
(Working on the indices of only NA elements is relatively efficient in that it only works on and replaces the not-yet-matched entries. If there is nothing left when it gets to another call to as.Date, it does still call it but with an argument of length 0, with which the function works rather efficiently. I don't think adding a conditional of if (any(nas)) ... would help, but if there are further methods you need that might be more "expensive", you can consider it.)