I have multiple lists of measurements. In each list have the timestramp formated as a string ("2009-12-24 21:00:07.0") and I know that each measurement in the list is separated by 5 seconds.
I want to combine all data into a huge data.frame in R. Afterwards I want to be able to easily access the time difference of two measurements so I probably should convert data into something different than characters.
Which format should I use to store the times? Is there some time format in some package that I should use?
You want the (standard) POSIXt type from base R that can be had in 'compact form' as a POSIXct (which is essentially a double representing fractional seconds since the epoch) or as long form in POSIXlt (which contains sub-elements). The cool thing is that arithmetic etc are defined on this -- see help(DateTimeClasses)
Quick example:
R> now <- Sys.time()
R> now
[1] "2009-12-25 18:39:11 CST"
R> as.numeric(now)
[1] 1.262e+09
R> now + 10 # adds 10 seconds
[1] "2009-12-25 18:39:21 CST"
R> as.POSIXlt(now)
[1] "2009-12-25 18:39:11 CST"
R> str(as.POSIXlt(now))
POSIXlt[1:9], format: "2009-12-25 18:39:11"
R> unclass(as.POSIXlt(now))
$sec
[1] 11.79
$min
[1] 39
$hour
[1] 18
$mday
[1] 25
$mon
[1] 11
$year
[1] 109
$wday
[1] 5
$yday
[1] 358
$isdst
[1] 0
attr(,"tzone")
[1] "America/Chicago" "CST" "CDT"
R>
As for reading them in, see help(strptime)
As for difference, easy too:
R> Jan1 <- strptime("2009-01-01 00:00:00", "%Y-%m-%d %H:%M:%S")
R> difftime(now, Jan1, unit="week")
Time difference of 51.25 weeks
R>
Lastly, the zoo package is an extremely versatile and well-documented container for matrix with associated date/time indices.
Related
I am designing a Flex dashboard. One of the column in my dashboard is a time stamp whose column contains entries like 2020-03-02T16:30:36Z. I want to convert it into dd/mm/yyy hh:mm:ss. Please help.
I tried this but nothing happened. In-fact, the entries got removed from the flex dashboard
df$time<- as.POSIXct(df$time,
format="%Y-%m-%dT%H:%M:%OSZ", tz="GMT")
The anytime package can help:
R> library(anytime)
R> anytime("2020-03-02T16:30:36Z")
[1] "2020-03-02 16:30:36 CST"
R> utctime("2020-03-02T16:30:36Z", tz="UTC")
[1] "2020-03-02 16:30:36 UTC"
R>
First, by not requiring an input format but rather by relying on a number of possibly / plausible formats it tries heuristically. Second, by also offering to parse at UTC (and, as we do here, impose UTC for the printed format / display, which is otherwise localtime). Third, we also have some output formats should you need them:
R> pt <- utctime("2020-03-02T16:30:36Z", tz="UTC")
R> iso8601(pt)
[1] "2020-03-02T16:30:36"
R> rfc2822(pt)
[1] "Mon, 02 Mar 2020 16:30:36.000000 +0000"
R> rfc3339(pt)
[1] "2020-03-02T16:30:36.000000+0000"
R> yyyymmdd(pt)
[1] "20200302"
R>
The underlying implementation is in C++ so it also tends to be faster than the equivalent alternatives (which require a format spec or hint).
libridate's function as_datetime also works:
library(lubridate)
as_datetime("2020-03-02T16:30:36Z")
[1] "2020-03-02 16:30:36 UTC"
I have a date that I convert to a numeric value and want to convert back to a date afterwards.
Converting date to numeric:
date1 = as.POSIXct('2017-12-30 15:00:00')
date1_num = as.numeric(date1)
# 1514646000
Reconverting numeric to date:
as.Date(date1_num, origin = '1/1/1970')
# "4146960-12-12"
What am I missing with the reconversion? I'd expect the last command to return my original date1.
As the numeric vector is created from an object with time component, reconversion can also be in the same way i.e. first to POSIXct and then wrap with as.Date
as.Date(as.POSIXct(date1_num, origin = '1970-01-01'))
#[1] "2017-12-30"
You could use anytime() and anydate() from the anytime package:
R> pt <- anytime("2017-12-30 15:00:00")
R> pt
[1] "2017-12-30 15:00:00 CST"
R>
R> anydate(pt)
[1] "2017-12-30"
R>
R> as.numeric(pt)
[1] 1514667600
R>
R> anydate(as.numeric(pt))
[1] "2017-12-30"
R>
POSIXct counts the number of seconds since the Unix Epoch, while Date counts the number of days. So you can recover the date by dividing by (60*60*24) (let's ignore leap seconds), or convert back to POSIXct instead.
as.Date(as.numeric(date1)/(60*60*24), origin="1970-01-01")
[1] "2017-12-30"
as.POSIXct(as.numeric(date1),origin="1970-01-01")
[1] "2017-12-30 15:00:00 GMT"
Using lubridate :
lubridate::as_datetime(1514646000)
[1] "2017-12-30 15:00:00 UTC"
Are there any built on functions that can be used on a data frame object to generate variables on a class Date time series to create day of the Week, Month, Year, Week of the Year, etc in R?
The weekdays, months, quarters, functions in the base package generate text output, looking for numerical output to denote that 3/5/2012, for example, is a Friday, 3rd day of the month, 1 week of the month, and the 63 day of the year, etc.
You get a few of those just from POSIXlt, with its weird convention. Year needs to 1900,
month is on the 0 to 11 range -- but you do get weekday and day-of-the-year.
R> dd <- as.Date("2012-05-03")
R> as.POSIXlt(dd)
[1] "2012-05-03 UTC"
Then
R> unclass(as.POSIXlt(dd))
$sec
[1] 0
$min
[1] 0
$hour
[1] 0
$mday
[1] 3
$mon
[1] 4
$year
[1] 112
$wday
[1] 4
$yday
[1] 123
$isdst
[1] 0
attr(,"tzone")
[1] "UTC"
R>
You can use the lubridate package to do a lot with dates.
From the help file: Lubridate provides tools that make it easier to parse and manipulate dates.
For example:
> library(lubridate)
> d <- today()
> d
[1] "2014-04-29"
> day(d)
[1] 29
> month(d)
[1] 4
> year(d)
[1] 2014
> week(d)
[1] 18
> weekdays(d)
[1] "Tuesday"
> days_in_month(d)
Apr
30
I prefer it to the built-in functions because it has a lot of date splicing, casting and arithmetic functions.
There are a couple of options that I can think of.
First, you could use the class as.POSIXlt so that you can subset with things like df$date$yday. The as.POSIXlt() includes the elements of dates as a list underneath that can be accessed that way.
Also, the package lubridate has functions like
yday(x)
wday(x)
mday(x)
I'm this has been asked before, but I just can't find the exact answer.
If I have a number which represents milliseconds since midnight, say 34200577, how do I turn this into an R time?
Construct a 'baseline time' at midnight, add the given millisecond once converted to seconds and interpret as a time:
R> as.POSIXct(as.numeric(ISOdatetime(2013,8,22,0,0,0)) + 34200577/1e3,
+ origin="1970-01-01")
[1] "2013-08-22 09:30:00.576 CDT"
R>
In fact, the shorter
R> ISOdatetime(2013,8,22,0,0,0) + 34200577/1e3
[1] "2013-08-22 09:30:00.576 CDT"
R>
works as well as ISOdatetime() returns a proper time object which operates in fractional seconds so we just apply the given offset.
This appears to be correct as
R> 34200577/1e3 # seconds
[1] 34200.6
R> 34200577/1e3/60 # minutes
[1] 570.01
R> 34200577/1e3/60/60 # hours
[1] 9.50016
R>
POSIXct uses 1970 as the origin of its time scale(measured in seconds.)
> time= as.POSIXct(34200577/1000 , origin=Sys.Date() )
> time
[1] "2013-08-22 02:30:00 PDT"
Note the discrepancy in results between Dirk's and my method. The POSIX times are input as assumed to occur in UCT, so there appeared the addition 8 hours for my location in UCT-8.
> difftime( as.POSIXct(34200577/1000 , origin=Sys.Date() ) , Sys.Date() )
Time difference of 9.50016 hours
You could get the time since midnight with:
format( as.POSIXct(34200577/1000 , origin=Sys.Date(), tz="UCT" ),
format="%H:%M:%S")
[1] "09:30:00"
A little "gottcha" which I think is worth pointing out...
In R 3.1.2 on windows 64 bit I get the following results for Dirk's example
> ISOdatetime(2013,8,22,0,0,0) + 34200577/1e3
[1] "2013-08-22 09:30:00 BST"
Note the lack of fractional seconds. This is due to the option setting for "digits.secs"
> getOption("digits.secs")
NULL
Setting this option as follows gives the expected result:
> options(digits.secs=3)
> ISOdatetime(2013,8,22,0,0,0) + 34200577/1e3
[1] "2013-08-22 09:30:00.576 BST"
As you can probably guess, this is to do with the formatting of output, not the actual values we get from our date arithmetic. See ?strptime and ?options for the documentation on this.
I am working in R and I need to change from a column in format
9/27/2011 3:33:00 PM
to a value format. In Excel I can use the function value() but I do not know how to do it in R.
My data looks like this:
9/27/2011 15:33 a 1 5 9
9/27/2011 15:33 v 2 6 2
9/27/2011 15:34 c 3 7 1
To convert a string into R date format, use as.POSIXct - then you can coerce it to a numeric value using as.numeric:
> x <- as.POSIXct("9/27/2011 3:33:00 PM", format="%m/%d/%Y %H:%M:%S %p")
> x
[1] "2011-09-27 03:33:00 BST"
> as.numeric(x)
[1] 1317090780
The value you get indicates the number of seconds since an arbitrary date, usually 1/1/1970. Note that this is different from Excel, where a date is stored as the number of days since an arbitrary date (1/1/1900 if my memory serves me well - I try not to use Excel any more.)
For more information, see ?DateTimeClasses
This was useful for me:
> test=as.POSIXlt("09/13/2006", format="%m/%d/%Y")
> test
[1] "2006-09-13"
> 1900+test$year
[1] 2006
> test$yday
[1] 255
> test$yday/365
[1] 0.6986301
> 1900+test$year+test$yday/366
[1] 2006.697
You can use similar approaches if you need day numbers like in Excel.