I try to find the difference between two timestamps.
The codeQ:
survey <- data.frame(date=c("07/2012","07/2012"),tx_start=c("01/2012","01/2012"))
survey$date_diff <- as.Date(as.character(survey$date), format="%m/%Y")-
as.Date(as.character(survey$tx_start), format="%m/%Y")
survey
I expect to have in the new column the different but I take NA
The results:
> survey
date tx_start date_diff
1 07/2012 01/2012 NA days
2 07/2012 01/2012 NA days
What should I change to replace as.Date for months or years?
Update based on comment of Gregor:
> survey <- data.frame(date=c("07/2012","07/2012"),tx_start=c("01/2012","01/2012"))
> survey$date <- as.Date(paste0("01/", as.character(survey$date)), "%d/%m/%Y")
> survey$tx_start <- as.Date(paste0("01/", as.character(survey$tx_start)), "%d/%m/%Y")
> survey$date_diff <- as.Date(survey$date, format="%d/%m/%Y")-
+ as.Date(survey$tx_start, format="%d/%m/%Y")
> survey
date tx_start date_diff
1 2012-07-01 2012-01-01 182 days
2 2012-07-01 2012-01-01 182 days
I usually convert my dates to POSIXct format. Then, when direct differences are taken with normal syntax, you get an answer in units of seconds. There is a difftime() function in base R that you can use as well:
survey <- data.frame(date=c("07/2012","07/2012"),tx_start=c("01/2012","01/2012"))
# Dates are finicky, add a day so that conversion will work
survey$date2 <- paste0("01/",survey$date)
survey$tx_start2 <- paste0("01/",survey$tx_start)
# conversion
survey$date2 <- as.POSIXct(x=survey$date2,format="%d/%m/%Y")
survey$tx_start2 <- as.POSIXct(x=survey$tx_start2,format="%d/%m/%Y")
# take the difference
survey$date_diff <- with(survey,difftime(time1=date2,time2=tx_start2,units="hours"))
Related
I have two data frames. One containing time periods marked with character unique IDs and another containing events with another set of unique IDs associated with them
Period DF (code):
periodID <- c("P_UID_00", "P_UID_01", "P_UDI_02", "P_UID_03")
periodStart <- as.POSIXct(c("2016/02/10 19:00", "2016/02/11 19:00",
"2016/02/12 19:00", "2016/02/13 19:00"))
periodEnd <- as.POSIXct(c("2016/02/10 21:00", "2016/02/11 21:00",
"2016/02/12 21:00", "2016/02/13 21:00"))
periodDF <- data.frame(periodID, periodStart, periodEnd)
Period DF:
periodID periodStart periodEnd
1 P_UID_00 2016-02-10 19:00:00 2016-02-10 21:00:00
2 P_UID_01 2016-02-11 19:00:00 2016-02-11 21:00:00
3 P_UDI_02 2016-02-12 19:00:00 2016-02-12 21:00:00
4 P_UID_03 2016-02-13 19:00:00 2016-02-13 21:00:00
Event DF (code):
eventID <- c("E_UID_00", "E_UID_01", "E_UDI_02", "E_UID_03")
eventTime <- as.POSIXct(c("2016/02/09 19:55:01", "2016/02/11 19:12:01",
"2016/02/11 20:22:01", "2016/02/15 19:00:01"))
eventDF <- data.frame(eventID, eventTime)
Event DF:
eventID eventTime
1 E_UID_00 2016-02-09 19:55:01
2 E_UID_01 2016-02-11 19:12:01
3 E_UDI_02 2016-02-11 20:22:01
4 E_UID_03 2016-02-15 19:00:01
I want to to map the event times in second DF to the time periods in the first DF in order to match the ID of the event to the ID of the period. Essentially the result table I want to see should look like:
eventID periodID
1 E_UID_00 NA
2 NA P_UID_00
3 E_UID_01 P_UID_01
4 E_UDI_02 P_UID_01
5 NA P_UID_02
6 NA P_UID_03
7 E_UID_03 NA
I suppose this can be achieved by using lubricate to transform the start and end cloumns in the first DF to intervals and the use some form of apply and instant %within% interval combination, but I am not really familiar with lubridate and did not manage to produce a working code
Additional considerations:
- periods are completely arbitrary and can last from seconds to years
- periods never overlap, so this is not an issue
- more than one event could be associated with a time period
- it is possible for DFs to contain unassociatable events and time periods
- the solution must not include loops
- does not have to be solved with lubridate, in fact a solution with the base R will be even more welcome.
I actually managed to come up with the code that produces exactly what I wanted using lubridate. So if anyone knows how to do this in base OR simply a better way than the one suggested below, sharing this will be greatly appreciated!
First off, the start and end times in the period DF should be converted to lubridate intervals:
intervalsP <- as.interval(periodStart, periodEnd)
Step 2: A function should be created for checking if an instant is located within a list of intervals. The only reason I have created a separate function is to be able using it with apply:
PeriodAssign <- function(x, y){
# x - instants
# y - intervals
variable1 <- mapply(`%within%`, x, y)
if (length(y[variable1]) != 0) {
as.character(y[variable1])
} else {
NA
}
}
NOTE: I had to use the interval to character coercion, because otherwise intervals were coerced to their length in seconds by the apply function and as such being not really useful for matching purposes - i.e. all four intervals in this example are the same length
Step 3: The function can the be used on the event DF and both DFs can then be merged to produce the DF I was looking for:
eventDF$intervals <- lapply(eventTime, PeriodAssign, intervalsP)
periodDF$intervals <- as.character(intervalsP)
mergedDF <- merge(periodDF, eventDF, by = "intervals")
presentableDF <- mergedDF[, c(2, 5)]
# adding in the unmatched Periods and Evenets
tDF1 <- data.frame(periodDF[!(periodDF$periodID %in% presentableDF$periodID), 1], NA)
colnames(tDF1) <- c("periodID", "eventID")
presentableDF <- rbind(presentableDF, tDF1)
tDF2 <- data.frame(NA, eventDF[!(eventDF$eventID %in% presentableDF$eventID), 1])
colnames(tDF2) <- c("periodID", "eventID")
presentableDF <- rbind(presentableDF, tDF2)
presentableDF <- presentableDF[order(presentableDF[,1]),]
The eventual DF looks like:
> presentableDF
periodID eventID
3 P_UID_00 <NA>
1 P_UID_01 E_UID_01
2 P_UID_01 E_UDI_02
4 P_UID_02 <NA>
5 P_UID_03 <NA>
6 <NA> E_UID_00
7 <NA> E_UID_03
I have dates of format 2015-03 (i.e year-month). Now I want to calculate the month difference in between 2 dates.
Example: difference between dates 2015-03 and 2014-12 should be 3 or 4 as December to March is 3 months or 4 months depending on whether we consider December or not.
You can do it via diff
require(lubridate)
a <- c("2015-03","2014-12")
a_parsed <- ymd(paste0(a,"-01")) # There might be a nicer solution to get the dates
diff(year(a_parsed)) * 12 + diff(month(a_parsed)) # Results in 3
Use + 1 to "consider December"
Explanation:
diff(year(a_parsed)) gives you the difference in the years, * 12 the month resulting from this. diff(month(a_parsed)) results in the monthly difference, ignoring the yearly difference. Combined it results in the Monthly difference you asked for.
a <- "2015-03"
b <- "2014-12"
a <- unlist(strsplit(a, "-"))
b <- unlist(strsplit(b, "-"))
a <- (as.numeric(a[1])*12) + as.numeric(a[2])
b <- (as.numeric(b[1])*12) + as.numeric(b[2])
difference <- diff(c(b,a))
difference
The result of this is 3
I have an irregular time series of all trades of a given ETF over a span of 4 years:
> head(BKF.xts)
BKF.xts
2008-01-02 09:30:01 59.870
2008-01-02 09:38:04 59.710
2008-01-02 09:39:51 59.612
2008-01-02 09:51:16 59.640
2008-01-02 10:06:08 59.500
> tail(BKF.xts)
BKF.xts
2011-12-30 15:59:23 36.26
2011-12-30 15:59:53 36.26
2011-12-30 15:59:56 36.27
2011-12-30 15:59:57 36.27
2011-12-30 15:59:58 36.27
2011-12-30 16:00:00 36.33
What I would like is to have the prices at every 5 minute interval for ALL trading days. Because I am dealing with ETFs it's possible that there are dates where the market is open that the ETF did not trade and so there will be no data for that date in my sample. However i need my final time series to account for all trading days. I have downloaded daily data for the same period so that I have another time series of every trading day. Not sure if that helps.
Also if there is no particular trade at one 5:00 minute time stamp I would like for the price of the most recent trade that took place. So for the data i posted above, what I would want is:
> head(BKF.xts)
BKF.xts
2008-01-02 09:35:00 59.870
2008-01-02 09:40:00 59.612
2008-01-02 09:45:00 59.612
2008-01-02 09:50:00 59.640
2008-01-02 09:55:00 59.640
Any help is greatly appreciated.
As mentionned in a previous question,
you can use to.period to have the last value in each 5-minute period,
align.time to replace the timestamps with the end of each period,
cbind to add the missing periods (with a missing value)
and na.locf to replace the missing values.
# Sample data
library(quantmod)
days <- seq(Sys.Date(), by=1, length=20)
days <- days[ ! format(days, "%A") %in% c("Saturday", "Sunday") ]
timestamps <- ISOdatetime(
year(days), month(days), day(days),
9, 0, 0 # You may want/need to add the timezone
)
timestamps <- timestamps[-2]
x <- lapply(timestamps, function(u) sort(u + sample(60*60*8,200)))
x <- do.call(c, x)
x <- xts(rnorm(length(x)), x)
# Value at the end of each 5-minute period
y <- to.minutes5(x)
y <- Cl(y)
y <- align.time(y, 5*60)
# All 5-minute periods, betweem 9am (excluded)
# and 5pm (included) for each day
z <- lapply(timestamps, function(u) u + 5*60*(1:(12*8)))
z <- do.call(c, z)
z <- cbind(y, xts(, z))
# Fill in missing values
z <- na.locf(z)
Thanks, I actually figured it out on my own after enough trial and error and discovering the xts subset function. Here's what I did:
#BKF here is my data set
BKF<-xts(BKF$PRICE,order.by=BKF$DATE)
colnames(BKF)=c("Price")
BKF<-to.minutes5(BKF)
BKF<-align.time(BKF,5*60)
#create a regular time series that has values for each 5 minute interval and use cbind to merge with my data
tmp<-xts(,seq.POSIXt(start(BKF),end(BKF),by="5 mins"))
BKF<-cbind(tmp,BKF)
# subset data from 9:30am to 4:00pm and replace NA's with last observation
BKF<-BKF["T09:30:00/T16:00:00"]
BKF<-na.locf(BKF)
# SP here is daily S&P data for the same sample period
SP<-xts(order.by=as.Date(td$Date,tz="",format="%y-%m-%d"))
# Subset observations for all trading days according to the daily S&P data
test<-bt[as.Date(index(bt),tz="")%in%as.Date(index(td),tz="")]
Done.
I’m attempting to transform two columns in my dataframe to the ‘good’ date & time class, and until now didn’t have much success with it. I’ve tried various classes (timeDate, Date, timeSeries, POSIXct, POSIXlt) but without success. Perhaps I’m just overlooking the obvious and because I’ve tried so many approaches I just don’t know what’s what anymore. I hope some of you can shed some light on where I go wrong.
Goal:
I want to calculate the difference between two dates using the earliest and latest date. I got this working with head() and tail(), but because those values aren’t necessary the earliest and latest date in my data, I need another way. (I can’t get the sorting of data to work, because it sorts the data only on the day of the date.)
Second goal: I want to convert the dates from daily format (i.e. 8-12-2010) to weekly, monthly, and yearly levels (i.e. '49-2010', 'december-10', and just '2010'). This can be done with the format settings (like %d-%m-%y). Can this be done with converting the data.frame to an time class, and than transforming the timeclass in the right format (8-12-2010 -> format("%B-%y") -> 'december-10'), and then transforming that time class into an factor with levels for each month?
For both goals I need to convert the dateframe in some way to an time class, and this is where I ran into some difficulties.
My dataframe looks like this:
> tradesList[c(1,10,11,20),14:15] -> tmpTimes4
> tmpTimes4
EntryTime ExitTime
1 01-03-07 10-04-07
10 29-10-07 02-11-07
11 13-04-07 14-05-07
20 18-12-07 20-02-08
Here’s an summary of what I’ve tried:
> class(tmpTimes4)
[1] "data.frame"
> as.Date(head(tmpTimes4$EntryTimes, n=1), format="%d-%m-%y")
Error in as.Date.default(head(tmpTimes4$EntryTimes, n = 1), format = "%d-%m-%y") :
do not know how to convert 'head(tmpTimes4$EntryTimes, n = 1)' to class "Date"
> as.timeDate(tmpTimes4, format="%d-%m-%y")
Error in as.timeDate(tmpTimes4, format = "%d-%m-%y") :
unused argument(s) (format = "%d-%m-%y")
> timeSeries(tmpTimes4, format="%d-%m-%y")
Error in midnightStandard2(charvec, format) :
'charvec' has non-NA entries of different number of characters
> tmpEntryTimes4 <- timeSeries(tmpTimes4$EntryTime, format="%d-%m-%y")
> tmpExitTimes4 <- timeSeries(tmpTimes4$ExitTime, format="%d-%m-%y")
> tmpTimes5 <- cbind(tmpEntryTimes4,tmpExitTimes4)
> colnames(tmpTimes5) <- c("Entry","Exit")
> tmpTimes5
Entry Exit
[1,] 01-03-07 10-04-07
[2,] 29-10-07 02-11-07
[3,] 13-04-07 14-05-07
[4,] 18-12-07 20-02-08
> class(tmpTimes5)
[1] "timeSeries"
attr(,"package")
[1] "timeSeries"
> as.timeDate(tmpTimes5, format="%d-%m-%y")
Error in as.timeDate(tmpTimes5, format = "%d-%m-%y") :
unused argument(s) (format = "%d-%m-%y")
> as.Date(tmpTimes5, format="%d-%m-%y")
Error in as.Date.default(tmpTimes5, format = "%d-%m-%y") :
do not know how to convert 'tmpTimes5' to class "Date"
> format.POSIXlt(tmpTimes5, format="%d-%m-%y", usetz=FALSE)
Error in format.POSIXlt(tmpTimes5, format = "%d-%m-%y", usetz = FALSE) :
wrong class
> as.POSIXlt(tmpTimes5, format="%d-%m-%y", usetz=FALSE)
Error in as.POSIXlt.default(tmpTimes5, format = "%d-%m-%y", usetz = FALSE) :
do not know how to convert 'tmpTimes5' to class "POSIXlt"
> as.POSIXct(tmpTimes5, format="%d-%m-%y", usetz=FALSE)
Error in as.POSIXlt.default(x, tz, ...) :
do not know how to convert 'x' to class "POSIXlt"
The TimeDate packages has an function for ‘range’, however, converting to the Date class works for an individual instance, but for some reason not for an data frame:
> as.Date(tmpTimes4[1,1], format="%d-%m-%y")
[1] "2007-03-01"
> as.Date(tmpTimes4, format="%d-%m-%y")
Error in as.Date.default(tmpTimes4, format = "%d-%m-%y") :
do not know how to convert 'tmpTimes4' to class "Date"
At this point I almost believe it’s impossible to do, so any thoughts would be highly appreciated!
Regards,
Start with some dummy data:
start <- as.Date("2010/01/01")
end <- as.Date("2010/12/31")
set.seed(1)
datewant <- seq(start, end, by = "days")[sample(15)]
tmpTimes <- data.frame(EntryTime = datewant,
ExitTime = datewant + sample(100, 15))
## reorder on EntryTime so in random order
tmpTimes <- tmpTimes[sample(NROW(tmpTimes)), ]
head(tmpTimes)
so we have something like this:
> head(tmpTimes)
EntryTime ExitTime
8 2010-01-14 2010-03-16
9 2010-01-05 2010-01-17
7 2010-01-10 2010-01-30
3 2010-01-08 2010-04-16
10 2010-01-01 2010-01-26
13 2010-01-12 2010-02-15
Using the above, look at Goal 1, compute difference between earliest and latest date. You can treat dates as if they were numbers (that is how they are stored internally anyway), so functions like min() and max() will work. You can use the difftime() function:
> with(tmpTimes, difftime(max(EntryTime), min(EntryTime)))
Time difference of 14 days
or use standard subtraction
> with(tmpTimes, max(EntryTime) - min(EntryTime))
Time difference of 14 days
to get the difference in days. head() and tail() will only work if you sort the dates as these take the first and the last value in a vector, not the highest and lowest actual value.
Goal 2: You seem to be trying to convert a data frame to a Date. You can't do this. What you can do is reformat the data in the components of the data frame. Here I add columns to tmpTimes by reformatting the EntryTime column into several different summaries of the date.
tmpTimes2 <- within(tmpTimes, weekOfYear <- format(EntryTime, format = "%W-%Y"))
tmpTimes2 <- within(tmpTimes2, monthYear <- format(EntryTime, format = "%B-%Y"))
tmpTimes2 <- within(tmpTimes2, Year <- format(EntryTime, format = "%Y"))
Giving:
> head(tmpTimes2)
EntryTime ExitTime weekOfYear monthYear Year
8 2010-01-14 2010-03-16 02-2010 January-2010 2010
9 2010-01-05 2010-01-17 01-2010 January-2010 2010
7 2010-01-10 2010-01-30 01-2010 January-2010 2010
3 2010-01-08 2010-04-16 01-2010 January-2010 2010
10 2010-01-01 2010-01-26 00-2010 January-2010 2010
13 2010-01-12 2010-02-15 02-2010 January-2010 2010
If you are American or want to use the US convention for the start of the week (%W starts the week on a Monday, in US convention is to start on a Sunday), change the %W to %U. ?strftime has more details of what %W and %U represent.
A final point on data format: In the above I have worked with dates in standard R format. You have your data stored in a data frame in a non-standard markup, presumably as characters or factors. So you have something like:
tmpTimes3 <- within(tmpTimes,
EntryTime <- format(EntryTime, format = "%d-%m-%y"))
tmpTimes3 <- within(tmpTimes3,
ExitTime <- format(ExitTime, format = "%d-%m-%y"))
> head(tmpTimes3)
EntryTime ExitTime
8 14-01-10 16-03-10
9 05-01-10 17-01-10
7 10-01-10 30-01-10
3 08-01-10 16-04-10
10 01-01-10 26-01-10
13 12-01-10 15-02-10
You need to convert those characters or factors to something R understands as a date. My preference would be the "Date" class. Before you try the above answers with your data, convert your data to the correct format:
tmpTimes3 <-
within(tmpTimes3, {
EntryTime <- as.Date(as.character(EntryTime), format = "%d-%m-%y")
ExitTime <- as.Date(as.character(ExitTime), format = "%d-%m-%y")
})
so that your data looks like this:
> head(tmpTimes3)
EntryTime ExitTime
8 2010-01-14 2010-03-16
9 2010-01-05 2010-01-17
7 2010-01-10 2010-01-30
3 2010-01-08 2010-04-16
10 2010-01-01 2010-01-26
13 2010-01-12 2010-02-15
> str(tmpTimes3)
'data.frame': 15 obs. of 2 variables:
$ EntryTime:Class 'Date' num [1:15] 14623 14614 14619 14617 14610 ...
$ ExitTime :Class 'Date' num [1:15] 14684 14626 14639 14715 14635 ...
Short answer:
Convert to date if not already done.
Then use min and max on the list
of dates.
date_list = structure(c(15401, 15405, 15405), class = "Date")
date_list
#[1] "2012-03-02" "2012-03-06" "2012-03-06"
min(date_list)
#[1] "2012-03-02"
max(date_list)
#[1] "2012-03-06"
More easy. Use summary() on date column directly giving Min and Max and more. Example: summary(df$date)
I need to convert date (m/d/y format) into 3 separate columns on which I hope to run an algorithm.(I'm trying to convert my dates into Julian Day Numbers). Saw this suggestion for another user for separating data out into multiple columns using Oracle. I'm using R and am throughly stuck about how to code this appropriately. Would A1,A2...represent my new column headings, and what would the format difference be with the "update set" section?
update <tablename> set A1 = substr(ORIG, 1, 4),
A2 = substr(ORIG, 5, 6),
A3 = substr(ORIG, 11, 6),
A4 = substr(ORIG, 17, 5);
I'm trying hard to improve my skills in R but cannot figure this one...any help is much appreciated. Thanks in advance... :)
I use the format() method for Date objects to pull apart dates in R. Using Dirk's datetext, here is how I would go about breaking up a date into its constituent parts:
datetxt <- c("2010-01-02", "2010-02-03", "2010-09-10")
datetxt <- as.Date(datetxt)
df <- data.frame(date = datetxt,
year = as.numeric(format(datetxt, format = "%Y")),
month = as.numeric(format(datetxt, format = "%m")),
day = as.numeric(format(datetxt, format = "%d")))
Which gives:
> df
date year month day
1 2010-01-02 2010 1 2
2 2010-02-03 2010 2 3
3 2010-09-10 2010 9 10
Note what several others have said; you can get the Julian dates without splitting out the various date components. I added this answer to show how you could do the breaking apart if you needed it for something else.
Given a text variable x, like this:
> x
[1] "10/3/2001"
then:
> as.Date(x,"%m/%d/%Y")
[1] "2001-10-03"
converts it to a date object. Then, if you need it:
> julian(as.Date(x,"%m/%d/%Y"))
[1] 11598
attr(,"origin")
[1] "1970-01-01"
gives you a Julian date (relative to 1970-01-01).
Don't try the substring thing...
See help(as.Date) for more.
Quick ones:
Julian date converters already exist in base R, see eg help(julian).
One approach may be to parse the date as a POSIXlt and to then read off the components. Other date / time classes and packages will work too but there is something to be said for base R.
Parsing dates as string is almost always a bad approach.
Here is an example:
datetxt <- c("2010-01-02", "2010-02-03", "2010-09-10")
dates <- as.Date(datetxt) ## you could examine these as well
plt <- as.POSIXlt(dates) ## now as POSIXlt types
plt[["year"]] + 1900 ## years are with offset 1900
#[1] 2010 2010 2010
plt[["mon"]] + 1 ## and months are on the 0 .. 11 intervasl
#[1] 1 2 9
plt[["mday"]]
#[1] 2 3 10
df <- data.frame(year=plt[["year"]] + 1900,
month=plt[["mon"]] + 1, day=plt[["mday"]])
df
# year month day
#1 2010 1 2
#2 2010 2 3
#3 2010 9 10
And of course
julian(dates)
#[1] 14611 14643 14862
#attr(,"origin")
#[1] "1970-01-01"
To convert date (m/d/y format) into 3 separate columns,consider the df,
df <- data.frame(date = c("01-02-18", "02-20-18", "03-23-18"))
df
date
1 01-02-18
2 02-20-18
3 03-23-18
Convert to date format
df$date <- as.Date(df$date, format="%m-%d-%y")
df
date
1 2018-01-02
2 2018-02-20
3 2018-03-23
To get three seperate columns with year, month and date,
library(lubridate)
df$year <- year(ymd(df$date))
df$month <- month(ymd(df$date))
df$day <- day(ymd(df$date))
df
date year month day
1 2018-01-02 2018 1 2
2 2018-02-20 2018 2 20
3 2018-03-23 2018 3 23
Hope this helps.
Hi Gavin: another way [using your idea] is:
The data-frame we will use is oilstocks which contains a variety of variables related to the changes over time of the oil and gas stocks.
The variables are:
colnames(stocks)
"bpV" "bpO" "bpC" "bpMN" "bpMX" "emdate" "emV" "emO" "emC"
"emMN" "emMN.1" "chdate" "chV" "cbO" "chC" "chMN" "chMX"
One of the first things to do is change the emdate field, which is an integer vector, into a date vector.
realdate<-as.Date(emdate,format="%m/%d/%Y")
Next we want to split emdate column into three separate columns representing month, day and year using the idea supplied by you.
> dfdate <- data.frame(date=realdate)
year=as.numeric (format(realdate,"%Y"))
month=as.numeric (format(realdate,"%m"))
day=as.numeric (format(realdate,"%d"))
ls() will include the individual vectors, day, month, year and dfdate.
Now merge the dfdate, day, month, year into the original data-frame [stocks].
ostocks<-cbind(dfdate,day,month,year,stocks)
colnames(ostocks)
"date" "day" "month" "year" "bpV" "bpO" "bpC" "bpMN" "bpMX" "emdate" "emV" "emO" "emC" "emMN" "emMX" "chdate" "chV"
"cbO" "chC" "chMN" "chMX"
Similar results and I also have date, day, month, year as separate vectors outside of the df.