I have daily data for 7 years. I want to group this into weekly data (based on the actual date) and sum the frequency.
Date Frequency
1 2014-01-01 179
2 2014-01-02 82
3 2014-01-03 89
4 2014-01-04 109
5 2014-01-05 90
6 2014-01-06 66
7 2014-01-07 75
8 2014-01-08 106
9 2014-01-09 89
10 2014-01-10 82
What is the best way to achieve that? Thank you
These solutions all use base R and differ only in the definition and labelling of weeks.
1) cut the dates into weeks and then aggregate over those. Weeks start on Monday but you can add start.on.monday=FALSE to cut to start them on Sunday if you prefer.
Week <- as.Date(cut(DF$Date, "week"))
aggregate(Frequency ~ Week, DF, sum)
## Week Frequency
## 1 2013-12-30 549
## 2 2014-01-06 418
2) If you prefer to define a week as 7 days starting with DF$Date[1] and label them according to the first date in that week then use this. (Add 6 to Week if you prefer the last date in the week.)
weekno <- as.numeric(DF$Date - DF$Date[1]) %/% 7
Week <- DF$Date[1] + 7 * weekno
aggregate(Frequency ~ Week, DF, sum)
## Week Frequency
## 1 2014-01-01 690
## 2 2014-01-08 277
3) or if you prefer to label it with the first date existing in DF in that week then use this. This and the last Week definition give the same result if there are no missing dates as is the case here. (If you want the last existing date in the week rather than the first then replace match with findInterval.)
weekno <- as.numeric(DF$Date - DF$Date[1]) %/% 7
Week <- DF$Date[match(weekno, weekno)]
aggregate(Frequency ~ Week, DF, sum)
## Week Frequency
## 1 2014-01-01 690
## 2 2014-01-08 277
Note
The input in reproducible form is assumed to be:
Lines <- "Date Frequency
1 2014-01-01 179
2 2014-01-02 82
3 2014-01-03 89
4 2014-01-04 109
5 2014-01-05 90
6 2014-01-06 66
7 2014-01-07 75
8 2014-01-08 106
9 2014-01-09 89
10 2014-01-10 82"
DF <- read.table(text = Lines)
DF$Date <- as.Date(DF$Date)
I would use library(lubridate).
df <- read.table(header = TRUE,text = "date Frequency
2014-01-01 179
2014-01-02 82
2014-01-03 89
2014-01-04 109
2014-01-05 90
2014-01-06 66
2014-01-07 75
2014-01-08 106
2014-01-09 89
2014-01-10 82")
You can use base R or library(dplyr):
base R:
to be sure that the date is really a date:
df$date <- ymd(df$date)
df$week <- week(df$date)
or short:
df$week <- week(ymd(df$date))
or dplyr:
library(dplyr)
df %>%
mutate(week = week(ymd(date))) %>%
group_by(week)
Out:
Barring a good reason not to, you should be sure to use ISO weeks to be sure your aggregation intervals are equally sized.
data.table makes this work like so:
library(data.table)
setDT(myDF) # convert to data.table
myDF[ , .(weekly_freq = sum(Frequency)), by = isoweek(Date)]
Maybe you can try the base R code with aggregate + format, i.e.,
dfout <- aggregate(Frequency ~ yearweek,within(df,yearweek <- format(Date,"%Y,%W")),sum)
such that
> dfout
yearweek Frequency
1 2014,00 549
2 2014,01 418
DATA
df <- structure(list(Date = structure(c(16071, 16072, 16073, 16074,
16075, 16076, 16077, 16078, 16079, 16080), class = "Date"), Frequency = c(179L,
82L, 89L, 109L, 90L, 66L, 75L, 106L, 89L, 82L)), row.names = c("1",
"2", "3", "4", "5", "6", "7", "8", "9", "10"), class = "data.frame")
The new package slider from RStudio addresses this problem directly including the specification of the start of the weekly periods. Suppose the weekly periods were to start on a Monday so that the beginning of the first week would be Monday, 2013-12-30. Then the slider solution would be
library(slider)
slide_period_dfr(.x = DF, .i=as.Date(DF$Date),
.period = "week",
.f = ~data.frame(week_ending = tail(.x$Date,1),
week_freq = sum(.x$Frequency)),
.origin = as.Date("2013-12-30"))
with the result
week_ending week_freq
1 2014-01-05 549
2 2014-01-10 418
Related
I have daily data for 7 years. I want to group this into weekly data (based on the actual date) and sum the frequency.
Date Frequency
1 2014-01-01 179
2 2014-01-02 82
3 2014-01-03 89
4 2014-01-04 109
5 2014-01-05 90
6 2014-01-06 66
7 2014-01-07 75
8 2014-01-08 106
9 2014-01-09 89
10 2014-01-10 82
What is the best way to achieve that? Thank you
These solutions all use base R and differ only in the definition and labelling of weeks.
1) cut the dates into weeks and then aggregate over those. Weeks start on Monday but you can add start.on.monday=FALSE to cut to start them on Sunday if you prefer.
Week <- as.Date(cut(DF$Date, "week"))
aggregate(Frequency ~ Week, DF, sum)
## Week Frequency
## 1 2013-12-30 549
## 2 2014-01-06 418
2) If you prefer to define a week as 7 days starting with DF$Date[1] and label them according to the first date in that week then use this. (Add 6 to Week if you prefer the last date in the week.)
weekno <- as.numeric(DF$Date - DF$Date[1]) %/% 7
Week <- DF$Date[1] + 7 * weekno
aggregate(Frequency ~ Week, DF, sum)
## Week Frequency
## 1 2014-01-01 690
## 2 2014-01-08 277
3) or if you prefer to label it with the first date existing in DF in that week then use this. This and the last Week definition give the same result if there are no missing dates as is the case here. (If you want the last existing date in the week rather than the first then replace match with findInterval.)
weekno <- as.numeric(DF$Date - DF$Date[1]) %/% 7
Week <- DF$Date[match(weekno, weekno)]
aggregate(Frequency ~ Week, DF, sum)
## Week Frequency
## 1 2014-01-01 690
## 2 2014-01-08 277
Note
The input in reproducible form is assumed to be:
Lines <- "Date Frequency
1 2014-01-01 179
2 2014-01-02 82
3 2014-01-03 89
4 2014-01-04 109
5 2014-01-05 90
6 2014-01-06 66
7 2014-01-07 75
8 2014-01-08 106
9 2014-01-09 89
10 2014-01-10 82"
DF <- read.table(text = Lines)
DF$Date <- as.Date(DF$Date)
I would use library(lubridate).
df <- read.table(header = TRUE,text = "date Frequency
2014-01-01 179
2014-01-02 82
2014-01-03 89
2014-01-04 109
2014-01-05 90
2014-01-06 66
2014-01-07 75
2014-01-08 106
2014-01-09 89
2014-01-10 82")
You can use base R or library(dplyr):
base R:
to be sure that the date is really a date:
df$date <- ymd(df$date)
df$week <- week(df$date)
or short:
df$week <- week(ymd(df$date))
or dplyr:
library(dplyr)
df %>%
mutate(week = week(ymd(date))) %>%
group_by(week)
Out:
Barring a good reason not to, you should be sure to use ISO weeks to be sure your aggregation intervals are equally sized.
data.table makes this work like so:
library(data.table)
setDT(myDF) # convert to data.table
myDF[ , .(weekly_freq = sum(Frequency)), by = isoweek(Date)]
Maybe you can try the base R code with aggregate + format, i.e.,
dfout <- aggregate(Frequency ~ yearweek,within(df,yearweek <- format(Date,"%Y,%W")),sum)
such that
> dfout
yearweek Frequency
1 2014,00 549
2 2014,01 418
DATA
df <- structure(list(Date = structure(c(16071, 16072, 16073, 16074,
16075, 16076, 16077, 16078, 16079, 16080), class = "Date"), Frequency = c(179L,
82L, 89L, 109L, 90L, 66L, 75L, 106L, 89L, 82L)), row.names = c("1",
"2", "3", "4", "5", "6", "7", "8", "9", "10"), class = "data.frame")
The new package slider from RStudio addresses this problem directly including the specification of the start of the weekly periods. Suppose the weekly periods were to start on a Monday so that the beginning of the first week would be Monday, 2013-12-30. Then the slider solution would be
library(slider)
slide_period_dfr(.x = DF, .i=as.Date(DF$Date),
.period = "week",
.f = ~data.frame(week_ending = tail(.x$Date,1),
week_freq = sum(.x$Frequency)),
.origin = as.Date("2013-12-30"))
with the result
week_ending week_freq
1 2014-01-05 549
2 2014-01-10 418
I have a dataset who looks like this:
Date Electricity
janv-90 23
juin-90 24
juil-90 34
janv-91 42
juin-91 27
juil-91 13
But I want it looking like that:
Date Electricity
190 23
690 24
790 34
191 42
691 27
791 13
Note that my dataset goes from 90 to 10 (namely 1990 to 2010).
since your monts were in French, found a little long route, else we already have month names as constants in R like in month.abb or month.names
# first I create a look-up vector
month.abb.french <- c("janv", "fevr", "mars", "avril",
"mai", "juin", "juil", "aout", "sept",
"oct", "nov", "dec")
# extract the months
month <- unlist(strsplit(df$Date, "-"))[c(TRUE, FALSE)]
# similarily extract the years
year <- unlist(strsplit(df$Date, "-"))[c(FALSE, TRUE)]
# month
#[1] "janv" "juin" "juil" "janv" "juin" "juil"
# year
#[1] "90" "90" "90" "91" "91" "91"
df$newcol <- paste0(match(month, month.abb.french), year)
# Date Electricity newcol
#1: janv-90 23 190
#2: juin-90 24 690
#3: juil-90 34 790
#4: janv-91 42 191
#5: juin-91 27 691
#6: juil-91 13 791
We can just use match, substr and paste to get the expected output
df$Date <- as.numeric(paste0(match(substr(df$Date, 1, 4), month.abb), substring(df$Date, 6)))
df
# Date Electricity
# 1 190 23
# 2 690 24
# 3 790 34
# 4 191 42
# 5 691 27
# 6 791 13
Or using tidyverse by separating the 'Date' column into two columns ('Date' and 'val') by the - delimiter, then match the 'Date' with the mon_ab from the locale() and finally unite the 'Date' and 'val' columns together
library(dplyr)
library(tidyr)
library(readr)
separate(df, Date, into = c("Date", "val")) %>%
mutate(Date = match(Date, sub("\\.$", "", locale("fr")[[1]]$mon_ab))) %>%
unite(Date, Date, val, sep="")
# Date Electricity
#1 190 23
#2 690 24
#3 790 34
#4 191 42
#5 691 27
#6 791 13
data
df <- structure(list(Date = c("janv-90", "juin-90", "juil-90", "janv-91",
"juin-91", "juil-91"), Electricity = c(23L, 24L, 34L, 42L, 27L,
13L)), .Names = c("Date", "Electricity"), class = "data.frame", row.names = c(NA,
-6L))
I am new to R and currently working on some rainfall data. I have two data frames named df1 and df2.
df1
Date Duration_sum
5/28/2014 110
5/31/2014 20
5/31/2014 20
6/1/2014 10
6/1/2014 110
6/3/2014 140
6/4/2014 40
6/5/2014 60
6/12/2014 10
6/14/2014 100
df2
Date PercentRemoval
6/2/2014 25.8
6/5/2014 78.58
6/6/2014 15.6
6/13/2014 70.06
I want to look up the dates from df2 in df1. For example, if the 1st date from df2 is available in df1, I want to subset rows in df1 within the range of that specific date and 3 days prior to that. If that date is not available, then just look for the previous 3 days.
In case the data for previous 3 days are not available, then it will extract as many days as available but maximum limit is 3 days prior to the specific date of df2. If none of the dates are available in df1, then that date is ignored and look for the next date in df2. Also, for example, 3 days prior to 6/6/2014 is available in df1 but we have already considered those days for 6/5/2014. So, 6/6/2014 is ignored.
The resulted data frame should look something like this:
df3
col_1 Date Duration_sum
5/31/2014 20
5/31/2014 20
6/1/2014 10
6/2/2014 6/1/2014 110
6/3/2014 140
6/4/2014 40
6/5/2014 6/5/2014 60
6/13/2014 6/12/2014 10
I have used this code:
df3 <- df1[df1$Date %in% as.Date(c(df2)),]
this code gives me the results for specific dates but not for the previous 3 days. I would really appreciate If someone can help me out with this code or some other codes. Thanks in advance.
This may be one way to do the task. If I am correctly reading your question, you want to remove any date, which does not have more than 3 days as an interval with a previous date. In this way, you can avoid the overlapping issue you mentioned in your question; you can successfully remove the 5th of June, 2014. Once you filter dates in df2, you can subset df1 for each date in the revised df2 in the lapply() part. The output is a list, and you want to assign names to each data frame in the list. Finally, you bind all data frames.
library(dplyr)
mutate(df1, Date = as.Date(Date, format = "%m/%d/%Y")) -> df1
mutate(df2, Date = as.Date(Date, format = "%m/%d/%Y")) %>%
filter(!(Date - lag(Date, default = 0) < 3)) -> df2
lapply(df2$Date, function(x){
filter(df1, between(Date, x-3, x)) -> foo
foo
}) -> temp
names(temp) <- as.character(df2$Date)
bind_rows(temp, .id = "df2.date")
# df2.date Date Duration_sum
#1 2014-06-02 2014-05-31 20
#2 2014-06-02 2014-05-31 20
#3 2014-06-02 2014-06-01 10
#4 2014-06-02 2014-06-01 110
#5 2014-06-05 2014-06-03 140
#6 2014-06-05 2014-06-04 40
#7 2014-06-05 2014-06-05 60
#8 2014-06-13 2014-06-12 10
DATA
df1 <- structure(list(Date = c("5/28/2014", "5/31/2014", "5/31/2014",
"6/1/2014", "6/1/2014", "6/3/2014", "6/4/2014", "6/5/2014", "6/12/2014",
"6/14/2014"), Duration_sum = c(110L, 20L, 20L, 10L, 110L, 140L,
40L, 60L, 10L, 100L)), .Names = c("Date", "Duration_sum"), class = "data.frame", row.names = c(NA,
-10L))
df2 <- structure(list(Date = c("6/2/2014", "6/5/2014", "6/6/2014", "6/13/2014"
), PercentRemoval = c(25.8, 78.58, 15.6, 70.06)), .Names = c("Date",
"PercentRemoval"), class = "data.frame", row.names = c(NA, -4L
))
I need to aggregate my data Kg
Data Kg
1 2013-03-01 271
2 2013-03-06 374
3 2013-03-07 51
4 2013-03-12 210
5 2013-03-13 698
6 2013-03-15 328
by week or month. I have found this answer here in stackoverflow, but I really don't understand the answer. Who can show me how can I do this case. Thanx
The answer mentioned suggest that you should to use xts package.
library(xts)
## create you zoo objects using your data
## you replace text argument by read.zoo(yourfile, header = TRUE)
x.zoo <- read.zoo(text=' Data Kg
+ 1 2013-03-01 271
+ 2 2013-03-06 374
+ 3 2013-03-07 51
+ 4 2013-03-12 210
+ 5 2013-03-13 698
+ 6 2013-03-15 328',header=TRUE)
### then aggregate
apply.weekly(x.zoo, mean) ## per week
apply.monthly(x.zoo, mean) ## per month
see ??apply.xxxly:
Essentially a wrapper to the xts functions endpoints and period.apply, mainly as a convenience.
Or, you could use tapply to apply by group of weeks. Here I am using lubridate package to extract week part from a date.
# fake data
df <- structure(list(Datechar = c("2013-03-01", "2013-03-06", "2013-03-07",
"2013-03-12", "2013-03-13", "2013-03-15"), Kg = c(271L, 374L,
51L, 210L, 698L, 328L)), .Names = c("Datechar", "Kg"), class = "data.frame", row.names = c("1",
"2", "3", "4", "5", "6"))
# convert character to date
df$Date <- as.Date(df$Datechar)
# calculate mean kg for each week
library(lubridate)
tapply(df$Kg, week(df$Date), mean)
tapply(df$Kg, month(df$Date), mean)
Let's say I have several years worth of data which look like the following
# load date package and set random seed
library(lubridate)
set.seed(42)
# create data.frame of dates and income
date <- seq(dmy("26-12-2010"), dmy("15-01-2011"), by = "days")
df <- data.frame(date = date,
wday = wday(date),
wday.name = wday(date, label = TRUE, abbr = TRUE),
income = round(runif(21, 0, 100)),
week = format(date, format="%Y-%U"),
stringsAsFactors = FALSE)
# date wday wday.name income week
# 1 2010-12-26 1 Sun 91 2010-52
# 2 2010-12-27 2 Mon 94 2010-52
# 3 2010-12-28 3 Tues 29 2010-52
# 4 2010-12-29 4 Wed 83 2010-52
# 5 2010-12-30 5 Thurs 64 2010-52
# 6 2010-12-31 6 Fri 52 2010-52
# 7 2011-01-01 7 Sat 74 2011-00
# 8 2011-01-02 1 Sun 13 2011-01
# 9 2011-01-03 2 Mon 66 2011-01
# 10 2011-01-04 3 Tues 71 2011-01
# 11 2011-01-05 4 Wed 46 2011-01
# 12 2011-01-06 5 Thurs 72 2011-01
# 13 2011-01-07 6 Fri 93 2011-01
# 14 2011-01-08 7 Sat 26 2011-01
# 15 2011-01-09 1 Sun 46 2011-02
# 16 2011-01-10 2 Mon 94 2011-02
# 17 2011-01-11 3 Tues 98 2011-02
# 18 2011-01-12 4 Wed 12 2011-02
# 19 2011-01-13 5 Thurs 47 2011-02
# 20 2011-01-14 6 Fri 56 2011-02
# 21 2011-01-15 7 Sat 90 2011-02
I would like to sum 'income' for each week (Sunday thru Saturday). Currently I do the following:
Weekending 2011-01-01 = sum(df$income[1:7]) = 487
Weekending 2011-01-08 = sum(df$income[8:14]) = 387
Weekending 2011-01-15 = sum(df$income[15:21]) = 443
However I would like a more robust approach which will automatically sum by week. I can't work out how to automatically subset the data into weeks. Any help would be much appreciated.
First use format to convert your dates to week numbers, then plyr::ddply() to calculate the summaries:
library(plyr)
df$week <- format(df$date, format="%Y-%U")
ddply(df, .(week), summarize, income=sum(income))
week income
1 2011-52 413
2 2012-01 435
3 2012-02 379
For more information on format.date, see ?strptime, particular the bit that defines %U as the week number.
EDIT:
Given the modified data and requirement, one way is to divide the date by 7 to get a numeric number indicating the week. (Or more precisely, divide by the number of seconds in a week to get the number of weeks since the epoch, which is 1970-01-01 by default.
In code:
df$week <- as.Date("1970-01-01")+7*trunc(as.numeric(df$date)/(3600*24*7))
library(plyr)
ddply(df, .(week), summarize, income=sum(income))
week income
1 2010-12-23 298
2 2010-12-30 392
3 2011-01-06 294
4 2011-01-13 152
I have not checked that the week boundaries are on Sunday. You will have to check this, and insert an appropriate offset into the formula.
This is now simple using dplyr. Also I would suggest using cut(breaks = "week") rather than format() to cut the dates into weeks.
library(dplyr)
df %>% group_by(week = cut(date, "week")) %>% mutate(weekly_income = sum(income))
I Googled "group week days into weeks R" and came across this SO question. You mention you have multiple years, so I think we need to keep up with both the week number and also the year, so I modified the answers there as so format(date, format = "%U%y")
In use it looks like this:
library(plyr) #for aggregating
df <- transform(df, weeknum = format(date, format = "%y%U"))
ddply(df, "weeknum", summarize, suminc = sum(income))
#----
weeknum suminc
1 1152 413
2 1201 435
3 1202 379
See ?strptime for all the format abbreviations.
Try rollapply from the zoo package:
rollapply(df$income, width=7, FUN = sum, by = 7)
# [1] 487 387 443
Or, use period.sum from the xts package:
period.sum(xts(df$income, order.by=df$date), which(df$wday %in% 7))
# [,1]
# 2011-01-01 487
# 2011-01-08 387
# 2011-01-15 443
Or, to get the output in the format you want:
data.frame(income = period.sum(xts(df$income, order.by=df$date),
which(df$wday %in% 7)),
week = df$week[which(df$wday %in% 7)])
# income week
# 2011-01-01 487 2011-00
# 2011-01-08 387 2011-01
# 2011-01-15 443 2011-02
Note that the first week shows as 2011-00 because that's how it is entered in your data. You could also use week = df$week[which(df$wday %in% 1)] which would match your output.
This solution is influenced by #Andrie and #Chase.
# load plyr
library(plyr)
# format weeks as per requirement (replace "00" with "52" and adjust corresponding year)
tmp <- list()
tmp$y <- format(df$date, format="%Y")
tmp$w <- format(df$date, format="%U")
tmp$y[tmp$w=="00"] <- as.character(as.numeric(tmp$y[tmp$w=="00"]) - 1)
tmp$w[tmp$w=="00"] <- "52"
df$week <- paste(tmp$y, tmp$w, sep = "-")
# get summary
df2 <- ddply(df, .(week), summarize, income=sum(income))
# include week ending date
tmp$week.ending <- lapply(df2$week, function(x) rev(df[df$week==x, "date"])[[1]])
df2$week.ending <- sapply(tmp$week.ending, as.character)
# week income week.ending
# 1 2010-52 487 2011-01-01
# 2 2011-01 387 2011-01-08
# 3 2011-02 443 2011-01-15
df.index = df['week'] #the the dt variable as index
df.resample('W').sum() #sum using resample
With dplyr:
df %>%
arrange(date) %>%
mutate(week = as.numeric(date - date[1])%/%7) %>%
group_by(week) %>%
summarise(weekincome= sum(income))
Instead of date[1] you can have any date from when you want to start your weekly study.