I want to the frequency of observations "after 19:00" independently of date. What would be the quickest and most logical way?
As I told R that the Date column is a date as.Date, I would like to tell R that Time is a time column... and then just ask "Time > "19:00:00"" but this does not seem to be possible.
I tried as.POSIXct(Time, format= "%H:%M:%S") but this function adds a date of today to my column which creates annoying clutter and unprofessional look.
I could use substr(as.character(Time),1,2) > 19 but that doesn't feel very elegant either.
Date Time
1 2014-01-01 17:16:48
2 2014-01-01 18:57:36
3 2014-01-01 19:40:48
4 2014-01-01 19:40:48
5 2014-01-01 20:09:36
6 2014-01-01 20:24:00
library(data.table)
## Convert (by reference) your data to a data.table
setDT(dat)
dat[, .N
, by = list(above_1900 = hour(as.POSIXlt(Time, format="%H:%M:%S")) > 19)]
above_19 N
1: FALSE 4
2: TRUE 2
Related
I would like to extract ONLY the quarter from a date, e.g., to get an integer 1 from the date "2003-02-08". I have been trying something along this line
library(mondate)
as.yearqtr(dat$DATE)
"2003 Q1"
as.character(as.yearqtr(dat$DATE))[1]
"2003 Q1"
which hasn't been giving my desired result. Of course I can write conditions as follows
library(data.table)
data$DATE = as.Date(data$DATE, format='%d%b%Y')
data$month=month(data$DATE)
setDT(data)[month==1, quarter:=1]
...
This will work, but is not elegant at all. Is there a more beautiful way of doing this?
Thank you lmo and user2100721! I really wish I could accept all of the answers!
There is a base R function, quarters, that more or less accomplishes what you want, though it prepends "Q". So
quarters(as.Date("2001-05-01"))
[1] "Q2"
If it is important to get rid of the "Q", you could use substr
substr(quarters(as.Date("2001-05-01")), 2, 2)
[1] "2"
Other date-related base R functions, such as weekdays and months can be found in help page ?quarters.
I would do:
# example data
DT = data.table(id = 1:10, d = as.IDate("2003-02-08") + seq(100, by=50, length.out=10))
DT[, qtr := quarter(d)]
id d qtr
1: 1 2003-05-19 2
2: 2 2003-07-08 3
3: 3 2003-08-27 3
4: 4 2003-10-16 4
5: 5 2003-12-05 4
6: 6 2004-01-24 1
7: 7 2004-03-14 1
8: 8 2004-05-03 2
9: 9 2004-06-22 2
10: 10 2004-08-11 3
The quarter function is provided by data.table and works on both Date and IDate vectors. (IDate uses integer storage.)
lubridate package has the same function. We can use that also. I am using #Frank's DT
DT[, qtr := lubridate::quarter(d)]
dint package also is suitable for that:
library("dint")
d=as.Date("2015-01-01")
get_quarter(d)
you can find more about this package here.
I have data.frames in R containing values measured in an interval of 5 minutes. Its a huge amount of data (~2mio).
date close day
73 2015-01-02 00:05:00 0.00861385 0
74 2015-01-02 00:10:00 0.00861385 0
75 2015-01-02 00:15:00 0.00861385 0
I want to group the data by the daytime. For example adding the value "close" from 2015-01-02 00:05:00 to the "close" value from 2015-02-02 00:05:00 and so on...
I'm pretty sure its easy to manage, but I don't know the search terms I have to google. Any breadcrumbs would be highly appreciated.
PS: It would be cool, if there are any solutions avoiding loops, because they are very slow in R
You can do this with library(data.table).
with your dataset as df :
library(data.table)
setDT(df) # set your data.frame to data.table
df[, mins := .GRP, minute(date)] # group by minutes of your 'date' field
df[, .(sums = sum(close)), by=mins] # sum by group
Hi I am new to R and would like to know if there is a simple way to filter data over multiple dates.
I have a data which has dates from 07.03.2003 to 31.12.2016.
I need to split/ filter the data by multiple time series, as per below.
Dates require in new data frame:
07.03.2003 to 06/03/2005
and
01/01/2013 to 31/12/2016
i.e the new data frame should not include dates from 07/03/2005 to 31/12/2012
Let's take the following data.frame with dates:
df <- data.frame( date = c(ymd("2017-02-02"),ymd("2016-02-02"),ymd("2014-02-01"),ymd("2012-01-01")))
date
1 2017-02-02
2 2016-02-02
3 2014-02-01
4 2012-01-01
I can filter this for a range of dates using lubridate::ymd and dplyr::between and dplyr::between:
df1 <- filter(df, between(date, ymd("2017-01-01"), ymd("2017-03-01")))
date
1 2017-02-02
Or:
df2 <- filter(df, between(date, ymd("2013-01-01"), ymd("2014-04-01")))
date
1 2014-02-01
I would go with lubridate. In particular
library(data.table)
library(lubridate)
set.seed(555)#in order to be reproducible
N <- 1000#number of pseudonumbers to be generated
date1<-dmy("07-03-2003")
date2<-dmy("06-03-2005")
date3<-dmy("01-01-2013")
date4<-dmy("31-12-2016")
Creating data table with two columns (dates and numbers):
my_dt<-data.table(date_sample=c(sample(seq(date1, date4, by="day"), N),numeric_sample=sample(N,replace = F)))
> head(my_dt)
date_sample numeric_sample
1: 2007-04-11 2
2: 2006-04-20 71
3: 2007-12-20 46
4: 2016-05-23 78
5: 2011-10-07 5
6: 2003-09-10 47
Let's impose some cuts:
forbidden_dates<-interval(date2+1,date3-1)#create interval that dates should not fall in.
> forbidden_dates
[1] 2005-03-07 UTC--2012-12-31 UTC
test_date1<-dmy("08-03-2003")#should not fall in above range
test_date2<-dmy("08-03-2005")#should fall in above range
Therefore:
test_date1 %within% forbidden_dates
[1] FALSE
test_date2 %within% forbidden_dates
[1] TRUE
A good way of visualizing the cut:
before
>plot(my_dt)
my_dt<-my_dt[!(date_sample %within% forbidden_dates)]#applying the temporal cut
after
>plot(my_dt)
I would like to extract ONLY the quarter from a date, e.g., to get an integer 1 from the date "2003-02-08". I have been trying something along this line
library(mondate)
as.yearqtr(dat$DATE)
"2003 Q1"
as.character(as.yearqtr(dat$DATE))[1]
"2003 Q1"
which hasn't been giving my desired result. Of course I can write conditions as follows
library(data.table)
data$DATE = as.Date(data$DATE, format='%d%b%Y')
data$month=month(data$DATE)
setDT(data)[month==1, quarter:=1]
...
This will work, but is not elegant at all. Is there a more beautiful way of doing this?
Thank you lmo and user2100721! I really wish I could accept all of the answers!
There is a base R function, quarters, that more or less accomplishes what you want, though it prepends "Q". So
quarters(as.Date("2001-05-01"))
[1] "Q2"
If it is important to get rid of the "Q", you could use substr
substr(quarters(as.Date("2001-05-01")), 2, 2)
[1] "2"
Other date-related base R functions, such as weekdays and months can be found in help page ?quarters.
I would do:
# example data
DT = data.table(id = 1:10, d = as.IDate("2003-02-08") + seq(100, by=50, length.out=10))
DT[, qtr := quarter(d)]
id d qtr
1: 1 2003-05-19 2
2: 2 2003-07-08 3
3: 3 2003-08-27 3
4: 4 2003-10-16 4
5: 5 2003-12-05 4
6: 6 2004-01-24 1
7: 7 2004-03-14 1
8: 8 2004-05-03 2
9: 9 2004-06-22 2
10: 10 2004-08-11 3
The quarter function is provided by data.table and works on both Date and IDate vectors. (IDate uses integer storage.)
lubridate package has the same function. We can use that also. I am using #Frank's DT
DT[, qtr := lubridate::quarter(d)]
dint package also is suitable for that:
library("dint")
d=as.Date("2015-01-01")
get_quarter(d)
you can find more about this package here.
I have following data set:
>d
x date
1 1 1-3-2013
2 2 2-4-2010
3 3 2-5-2011
4 4 1-6-2012
I want:
> d
x date
1 1 31-12-2013
2 2 31-12-2010
3 3 31-12-2011
4 4 31-12-2012
i.e. Last day, last month and the year of the date object.
Please Help!
You can also just use the ceiling_date function in LUBRIDATE package.
You can do something like -
library(lubridate)
last_date <- ceiling_date(date,"year") - days(1)
ceiling_date(date,"year") gives you the first date of the next year and to get the last date of the current year, you subtract this by 1 or days(1).
Hope this helps.
Another option using lubridate package:
## using d from Roland answer
transform(d,last =dmy(paste0('3112',year(dmy(date)))))
x date last
1 1 1-3-2013 2013-12-31
2 2 2-4-2010 2010-12-31
3 3 2-5-2011 2011-12-31
4 4 1-6-2012 2012-12-31
d <- read.table(text="x date
1 1 1-3-2013
2 2 2-4-2010
3 3 2-5-2011
4 4 1-6-2012", header=TRUE)
d$date <- as.Date(d$date, "%d-%m-%Y")
d$date <- as.POSIXlt(d$date)
d$date$mon <- 11
d$date$mday <- 31
d$date <- as.Date(d$date)
# x date
#1 1 2013-12-31
#2 2 2010-12-31
#3 3 2011-12-31
#4 4 2012-12-31
1) cut.Date Define cut_year to give the first day of the year. Adding 366 gets us to the next year and then applying cut_year again gets us to the first day of the next year. Finally subtract 1 to get the last day of the year. The code uses base functionality only.
cut_year <- function(x) as.Date(cut(as.Date(x), "year"))
transform(d, date = cut_year(cut_year(date) + 366) - 1)
2) format
transform(d, date = as.Date(format(as.Date(date), "%Y-12-31")))
3) zoo A "yearmon" class variable stores the date as a year plus 0 for Jan, 1/12 for Feb, ..., 11/12 for Dec. Thus taking its floor and adding 11/12 gets one to Dec and as.Date.yearmon(..., frac = 1) uses the last of the month instead of the first.
library(zoo)
transform(d, date = as.Date(floor(as.yearmon(as.Date(date))) + 11 / 12, frac = 1))
Note: The inner as.Date in cut_year and in the other two solutions can be omitted if it is known that date is already of "Date" class.
ADDED additional solutions.