Someone gave me really bad data in Excel, where the date (such as July 1, 2015) is 20150701 and the time (such as 11:41:23) is 114123. There are over 50,000 rows of data and I need to convert these all into proper date and time objects. These aren't the number of seconds from any epoch, it is just the date or time without the dashes or the colons.
I imported them into a data frame and converted the dates using the ymd() function, but I can't find a function to do that for time, hms() gives me an error:
package(lubridate)
df <- readWorksheetFromFile(file="cktime2012.xls", sheet=1)
df$date <- ymd(df$date)
df$time <- hms(df$time)
# Warning message:
# In .parse_hms(..., order = "HM", quiet = quiet) :
# Some strings failed to parse
and I get a data frame that looks like this before running the last line. Once I run the last line, the TIMEIN column turns into all NA's:
DATEIN TIMEIN etc...
2012-02-01 200000 etc...
etc...
I need it to look like this for all 50,000 rows. I included POSIXct as a tag, because I don't know if there could be a way to use that to help convert:
DATEIN TIMEIN etc...
2012-02-01 20:00:00 etc...
etc...
If TIMEIN is always six characters (i.e., there's a leading zero for times before 10 AM), then you can do this:
df$TIMEIN = paste0(substr(df$TIMEIN,1,2),":",substr(df$TIMEIN,3,4),":", substr(df$TIMEIN,5,6))
df$TIMEIN = hms(df$TIMEIN)
You can try this too to get the specified time, but then you'd have to get rid of the date too.
> as.POSIXct("200000", format="%H%M%S")
[1] "2015-07-01 20:00:00 IST"
Edit-
Okay, as.POSIXct() works on date and time. So, to merge the whole into one you can do something like this.
> as.POSIXct("20120201 200000", format="%Y%m%d %H%M%S")
[1] "2012-02-01 20:00:00 IST"
Or simpler than the ones above, using the pipes in tidyverse you can get the following:
# make sure you have dates stores as POSIXct
# call in tidyverse library to make use of pipes and use the code bellow
df_hms <- df %>%
mutate(time = hms::as.hms(TIMEIN))
Related
As the title suggests, I am trying to use either lubridate or ANYTIME (or similar) to convert a time from 24 hour into 12 hour.. To make life easier I don't need the whole time converted.
What I mean is I have a column of dates in this format:
2021-02-15 16:30:33
I can use inbound$Hour <- hour(inbound$Timestamp) to grab just the hour from the Timestamp which is great.. except that it is still in 24hr time. (this creates an integer column for the hour number)
I have tried several mutates such as inbound <- inbound %>% mutate(Hour = ifelse(Hour > 12, sum(Hour - 12),Hour)
This technically works.. but I get some really wonky values (I get a -294 in several rows for example)..
is there an easier way to get the 12hr time converted?
Per recommendation below I tried to use a base FORMAT as follows:
inbound$Time <- format(inbound$Timestamp, "%H:%M:%S")
inbound$Time <- format(inbound$Time, "%I:%M:%S")
and on the second format I am getting an error
Error in format.default(inbound$Time, "%I:%M:%S") :
invalid 'trim' argument
I did notice the first format converts to a class CHARACTER column.. not sure if that is causing issues with the 2nd format or not..
I then also tried:
`inbound$time <- format(strptime(inbound$Timestamp, "%H:%M:%S"), "%I:%M %p")`
Which runs without error.. but it creates a full column of NA's
Final edit::::: I made the mistake of mis-reading/applying the solution and that caused errors.. when using the inbound$Time <- format(inbound$Time, "%I:%M:%S") or as.numeric(format(inbound$Timestamp, "%I")) from the comments... both worked and solved the issue I was having.
To be clear... From 2021-02-15 16:30:33 you want just 04:30:33 as a result?
No need for lubridate or anytime. Assuming that is a Posixct
a <- as.POSIXct("2021-02-15 16:30:33")
a
# [1] "2021-02-15 16:30:33 UTC"
b <- format(a, "%H:%M:%S")
b
#[1] "16:30:33"
c <- format(a, "%I:%M:%S")
c
#[1] "04:30:33"
Hi this question has been bugging me for some time.
So I am trying to convert the so-called dates in my R project into actual dates. Right now the dates are arranged in a numerical manner, ie after 2/28/2020 it's not 3/1/2020 but 2/3/2020.
I've tried the
as.Date(3/14/2020, origin = "14-03-2020")
and also
df <- data.frame(Date = c("10/9/2009 0:00:00", "10/15/2009 0:00:00"))
as.Date(df$Date, "%m/%d/%Y %H:%M:%S")
and
strDates <- c("01/28/2020", "05/03/2020")%>%
dates <- as.Date(strDates, "%m/%d/%Y")
i just plugged in two dates to test out if it works or not because there are about around 40 dates. However, my output is as follows:
Error in as.Date.default(., 3/14/2020, origin = "14-03-2020") : do not know how to convert '.' to class “Date”
for the first one and then
the second one is:
data frame not found
the third one is
Error in as.Date(strDates, "%m/%d/%Y") : object 'strDates' not found
Issues with your code:
as.Date(3/14/2020, origin = "14-03-2020")
First, R will replace 3/14/2020 with 0.000106082, since that's what 3 divided by 14 divided by 2020 equals. You need to identify it as a string using single or double quotes, as in: as.Date("3/14/2020", origin = "14-03-2020").
But that is still broken. When converting to Date, if you provide a character (string) input, then you may need to provide format=, since it needs to know which numbers in the string correspond to year, month, date, etc. If you provide a numeric (or integer) input, then you do need to provide origin=, so that it knows what "day 0" is. For unix, epoch is what you need, so origin="1970-01-01". If you're using dates from Excel, you need origin="1899-12-30" (see https://stackoverflow.com/a/43230524).
Your next error is because you are mixing magrittr ops with ... base R.
strDates <- c("01/28/2020", "05/03/2020")%>%
dates <- as.Date(strDates, "%m/%d/%Y")
The issue here has nothing to do with dates. The use of %>% on line 1 is taking the output of line 1 (in R, assignment to a variable invisibly returns the assigned numbers, which is why chaining assignment works, a <- b <- 2) and injecting it as the first argument in the next function call. With this your code was eventually interpreted as
strDates <- c("01/28/2020", "05/03/2020")%>%
{ dates <- as.Date(., strDates, "%m/%d/%Y") }
which is obviously not what you intended or need. I suspect that this is just an artifact of getting frustrated and was mid-stage converting from a %>% pipe to something else, and you forgot to clean up the %>%s. This could be
dates <- c("01/28/2020", "05/03/2020") %>%
as.Date("%m/%d/%Y")
dates
# [1] "2020-01-28" "2020-05-03"
Your data.frame code seems to work fine, though you do not assign the new Date-assigned values back to the frame. Try this slight adaptation:
df <- data.frame(Date = c("10/9/2009 0:00:00", "10/15/2009 0:00:00"))
df$Date <- as.Date(df$Date, "%m/%d/%Y %H:%M:%S")
df
# Date
# 1 2009-10-09
# 2 2009-10-15
str(df)
# 'data.frame': 2 obs. of 1 variable:
# $ Date: Date, format: "2009-10-09" "2009-10-15"
Here is just one date that I have (out of more than 6,000)
02/01/15
This is expressed as January 2nd, 2015. And I would like the date to instead look like the following,
2015/01/02
I read up on this thread: Changing date format to "%d/%m/%Y"
But unless my R does not work properly, none of the answers give the correct format, instead I get this output,
0002/01/15
You can do:
format(as.Date("02/01/15", format = "%d/%m/%y"), "%Y/%m/%d")
[1] "2015/01/02"
The lubridate package is very useful for date and time manipulation
library(lubridate)
x <- lubridate::dmy('02/01/15')
format(x, format = ('%Y/%m/%d'))
As a R novice I'm pulling my hair out trying to debug cryptic R errors. I have csv that containing 150k lines that I load into a data frame named 'date'. I then use lubridate to convert this character column to datetimes in hopes of finding min/max date.
dates <- csv[c('datetime')]
dates$datetime <- ymd_hms(dates$datetime)
Running this code I receive the following error message:
Warning message:
3 failed to parse.
I accept this as the CSV could have some janky dates in there and next run:
min(dates$datetime)
max(dates$datetime)
Both of these return NA, which I assume is from the few broken dates still stored in the data frame. I've searched around for a quick fix, and have even tried to build a foreach loop to identify the problem dates, but no luck. What would be a simple way to identify the 3 broken dates?
example date format: 2015-06-17 17:10:16 +0000
Credit to LawyeR and Stibu from above comments:
I first sorted the raw csv column and did a head() & tail() to find
which 3 dates were causing trouble
Alternatively which(is.na(dates$datetime)) was a simple one liner to also find the answer.
Lubridate will throw that error when attempting to parse dates that do not exist because of daylight savings time.
For example:
library(lubridate)
mydate <- strptime('2020-03-08 02:30:00', format = "%Y-%m-%d %H:%M:%S")
ymd_hms(mydate, tz = "America/Denver")
[1] NA
Warning message:
1 failed to parse.
My data comes from an unintelligent sensor which does not know about DST, so impossible (but correctly formatted) dates appear in my timeseries.
If the indices of where lubridate fails are useful to know, you can use a for loop with stopifnot() and print each successful parse.
Make some dates, throw an error in there at a random location.
library(lubridate)
set.seed(1)
my_dates<-as.character(sample(seq(as.Date('1900/01/01'),
as.Date('2000/01/01'), by="day"), 1000))
my_dates[sample(1:length(my_dates), 1)]<-"purpleElephant"
Now use a for loop and print each successful parse with stopifnot().
for(i in 1:length(my_dates)){
print(i)
stopifnot(!is.na(ymd(my_dates[i])))
}
To provide a more generic answer, first filter out the NAs, then try and parse, then filter only the NAs. This will show you the failures. Something like:
dates2 <- dates[!is.na(dates2$datetime)]
dates2$datetime <- ymd_hms(dates2$datetime)
Warning message:
3 failed to parse.
dates2[is.na(dates2$datetime)]
Here is a simple function that solves the generic problem:
parse_ymd = function(x){
d=lubridate::ymd(x, quiet=TRUE)
errors = x[!is.na(x) & is.na(d)]
if(length(errors)>0){
cli::cli_warn("Failed to parse some dates: {.val {errors}}")
}
d
}
x = c("2014/20/21", "2014/01/01", NA, "2014/01/02", "foobar")
my_date = lubridate::ymd(x)
#> Warning: 2 failed to parse.
my_date = parse_ymd(x)
#> Warning: Failed to parse some dates: "2014/20/21" and "foobar"
Created on 2022-09-29 with reprex v2.0.2
Of course, replace ymd() with whatever you want.
Use the truncate argument. The most common type of irregularity in date-time data is the truncation due to rounding or unavailability of the time stamp.
Therefore, try truncated = 1, then potentially go up to truncated = 3:
dates <- csv[c('datetime')]
dates$datetime <- ymd_hms(dates$datetime, truncated = 1)
I am new to R and programming in general and am looking for help with writing a function with dates and times. I have checked around but am still a bit stuck.
Basically, I have dates in the format "dd/mm/YYYY HH:MM" and I have to calculate how much time has passed between various events.
I have given the following command (where "date" is the column in my data frame that indicates the date and time in the above format):
date=as.Date.factor(date,format="%d/%m/%Y %H:%M")
However, this displays only the date, without the time.
I also have tried:
date=substr(argo1$date,1,907)
And it shows the date and time.
However, when I try to find the difference between two dates i.e.the time that has passed with the command: difftime(date[2],date[3],unit="secs"), it returns that 0 seconds have passed.
When I try to find the difference with the command:
date[3]-date[2]
it tells me
Error in date[3] - date[2] : non-numeric argument to binary operator
The class(date) is "character".
Any idea what I am doing wrong?
Try strptime:
date1 = "30/12/2009 11:59"
date2 = "30/12/2009 12:03"
d1 = strptime(date1, "%d/%m/%Y %H:%M")
d2 = strptime(date2, "%d/%m/%Y %H:%M")
difftime(d1, d2, unit="secs")
# Time difference of -240 secs