I am attempting to limit my dataframe to the days of each month between the 20th and the 25th . I got a big dataset with many dates ranging over many years. It looks something like this:
Event Date
Football 20.12.2016
Work 15.10.2019
Holiday 30.11.2018
Running 24.01.2020
I would then like to restrict my results to:
Event Date
Football 20.12.2016
Running 24.01.2020
Any tips on how to do this?
This is a solution using dplyr/lubridate although I have converted your Date column using as.Date
df <-
data.frame(
Event = c("Football", "Work", "Holiday", "Running"),
Date = c("20.12.2016", "15.10.2019", "30.11.2018", "24.01.2020")
)
df$Date <- as.Date(df$Date, format = "%d.%m.%Y")
df %>% filter(day(Date) >= 20 & day(Date) <= 25)
Output
1 Football 2016-12-20
2 Running 2020-01-24
Doing a literal string-match, keeping your Date column as strings (not real dates):
# base R
subset(quux, between(as.integer(sub("\\..*", "", Date)), 20, 25))
# Event Date
# 1 Football 20.12.2016
# 4 Running 24.01.2020
# dplyr
quux %>%
filter(between(as.integer(sub("\\..*", "", Date)), 20, 25))
between can be from dplyr::, data.table::, or we can easily craft our own with:
between <- function(x, y, z) y <= x & x <= z
... though the two package versions are more robust to NAs and other issues.
Data
quux <- structure(list(Event = c("Football", "Work", "Holiday", "Running"), Date = c("20.12.2016", "15.10.2019", "30.11.2018", "24.01.2020")), class = "data.frame", row.names = c(NA, -4L))
Related
I have a continuous list of dates (yyyy-mm-dd) from 1985 to 2018 in one column (Colname = date). What I wish to do is generate another column which outputs a water season and year given the date.
To make it clearer I have two water season:
Summer = yyyy-04-01 to yyyy-09-31;
Winter = yyyy-10-01 to yyyy(+1)-03-31.
So for 2018 - Summer = 2018-04-01 to 2018-09-31; Winter 2018-10-01 to 2019-03-31.
What I would like to output is something like the following:
Many thanks.
A tidy verse approach
library(tidyverse)
df <-tibble(date = seq(from = as.Date('2000-01-01'), to = as.Date('2001-12-31'), by = '1 month'))
df
df %>%
mutate(water_season_year = case_when(
lubridate::month(date) %in% c(4:9) ~str_c('Su_', lubridate::year(date)),
lubridate::month(date) %in% c(10:12) ~str_c('Wi_', lubridate::year(date)),
lubridate::month(date) %in% c(1:3)~str_c('Wi_', lubridate::year(date) -1),
TRUE ~ 'Error'))
You can compare just the month part of the data to get the season, in base R consider doing
month <- as.integer(format(df$date, "%m"))
year <- format(df$date, "%Y")
inds <- month >= 4 & month <= 9
df$water_season_year <- NA
df$water_season_year[inds] <- paste("Su", year[inds], sep = "_")
df$water_season_year[!inds] <- paste("Wi", year[!inds], sep = "_")
#To add previous year for month <= 3 do
df$water_season_year[month <= 3] <- paste("Wi",
as.integer(year[month <= 3]) - 1, sep = "_")
df
# date water_season_year
#1 2019-01-03 Wi_2019
#2 2000-06-01 Su_2000
Make sure that date variable is of "Date" class.
data
df <-data.frame(date = as.Date(c("2019-01-03", "2000-06-01")))
I would like to retain my current date column in year-month format as date. It currently gets converted to chr format. I have tried as_datetime but it coerces all values to NA.
The format I am looking for is: "2017-01"
library(lubridate)
df<- data.frame(Date=c("2017-01-01","2017-01-02","2017-01-03","2017-01-04",
"2018-01-01","2018-01-02","2018-02-01","2018-03-02"),
N=c(24,10,13,12,10,10,33,45))
df$Date <- as_datetime(df$Date)
df$Date <- ymd(df$Date)
df$Date <- strftime(df$Date,format="%Y-%m")
Thanks in advance!
lubridate only handle dates, and dates have days. However, as alistaire mentions, you can floor them by month of you want work monthly:
library(tidyverse)
df_month <-
df %>%
mutate(Date = floor_date(as_date(Date), "month"))
If you e.g. want to aggregate by month, just group_by() and summarize().
df_month %>%
group_by(Date) %>%
summarize(N = sum(N)) %>%
ungroup()
#> # A tibble: 4 x 2
#> Date N
#> <date> <dbl>
#>1 2017-01-01 59
#>2 2018-01-01 20
#>3 2018-02-01 33
#>4 2018-03-01 45
You can solve this with zoo::as.yearmon() function. Follows the solution:
library(tidyquant)
library(magrittr)
library(dplyr)
df <- data.frame(Date=c("2017-01-01","2017-01-02","2017-01-03","2017-01-04",
"2018-01-01","2018-01-02","2018-02-01","2018-03-02"),
N=c(24,10,13,12,10,10,33,45))
df %<>% mutate(Date = zoo::as.yearmon(Date))
You can use cut function, and use breaks="month" to transform all your days in your dates to the first day of the month. So any date within the same month will have the same date in the new created column.
This is usefull to group all other variables in your data frame by month (essentially what you are trying to do). However cut will create a factor, but this can be converted back to a date. So you can still have the date class in your data frame.
You just can't get rid of the day in a date (because then, is not a date...). Afterwards you can create a nice format for axes or tables. For example:
true_date <-
as.POSIXlt(
c(
"2017-01-01",
"2017-01-02",
"2017-01-03",
"2017-01-04",
"2018-01-01",
"2018-01-02",
"2018-02-01",
"2018-03-02"
),
format = "%F"
)
df <-
data.frame(
Date = cut(true_date, breaks = "month"),
N = c(24, 10, 13, 12, 10, 10, 33, 45)
)
## here df$Date is a 'factor'. You could use substr to create a formated column
df$formated_date <- substr(df$Date, start = 1, stop = 7)
## and you can convert back to date class. format = "%F", is ISO 8601 standard date format
df$true_date <- strptime(x = as.character(df$Date), format = "%F")
str(df)
Each day I have a new csv file with ids and some variables. The ids can be differents over the days. I would like to take the IDs of one day and follow how a variable evolves over the time.
My goal is to create area plot like this :
For example I take all the ids the 31 march, each day I make a join with thoses ids, and I make a count group by the var "Code". If there is missing ids (Ids here the 31 march but not day D) their code become "NA" to show how many IDs I "lose" over time. I hope i'm clear enough.
Here is how I calculate this king of plot : (my real datas are like li and not datas)
library(plyr)
library(dplyr)
datas <- data.frame(id1 = c("x", "y", "x", "y", "z", "x", "z"),
id2 = c("x2", "y2", "x2", "y2", "z2", "x2", "z2"),
code = c("code1", "code2", "code1", "code2", "code2", "code1", "code2"),
var = runif(7),
date = do.call(c, mapply(rep, seq(Sys.Date() - 2, Sys.Date(), by = 1), c(2, 3, 2))))
li <- split(datas, datas$date)
dateStart <- Sys.Date() - 2
dateEnd <- Sys.Date()
# A "filter" if I want to start with another date than the date min or end with another date than the max date
li <- li[as.Date(names(li)) >= dateStart & as.Date(names(li)) <= dateEnd]
dfCounts <- ldply(li, function(x)
left_join(li[[1]], x, by = c("id1", "id2")) %>%
group_by(code.y) %>%
count(code = code.y) %>%
mutate(freq = n / sum(n),
code = ifelse(is.na(code), "NA", code))),
.id = "date")
> dfCounts
date code n freq
1 2015-07-04 1 1 0.5
2 2015-07-04 2 1 0.5
3 2015-07-05 1 1 0.5
4 2015-07-05 2 1 0.5
5 2015-07-06 1 1 0.5
6 2015-07-06 NA 1 0.5
dfCounts %>%
ggplot(aes(date, freq)) +
geom_area(aes(fill = code), position = "stack")
# I have no idea why in this example, nothing is shown in the plot, but it works on my real datas
So it works, but if I want to observe a longer period, I have to join over many days (files) and it can be slow. Do you have any ideas to do the same things without joins, using the binded datas (the object datas and not li) with dplyr or data.table ?
In your opinion, which approach is better ?
Thanks !
(Sorry for the title I couldn't find better...)
I am a new user of "R", and I couldn't find a good solution to solve it. I got a timeseries in the following format:
>dates temperature depth salinity
>12/03/2012 11:26 9.7533 0.48073 37.607
>12/03/2012 11:56 9.6673 0.33281 37.662
>12/03/2012 12:26 9.6673 0.33281 37.672
I have an irregular frequency for variable measurements, done every 15 or every 30 minutes depending on the period. I would like to calculate annual, monthly and daily averages for each of my variables, whatever the number of data in a day/month/year is. I read a lot of things about the packages zoo, timeseries, xts, etc. but I can't get a clear vision of what I nead (maybe cause I'm not skilled enough with R...).
I hope my post is clear, don't hesitate to tell me if it's not.
Convert your data to an xts object, then use apply.daily et al to calculate whatever values you want.
library(xts)
d <- structure(list(dates = c("12/03/2012 11:26", "12/03/2012 11:56",
"12/03/2012 12:26"), temperature = c(9.7533, 9.6673, 9.6673),
depth = c(0.48073, 0.33281, 0.33281), salinity = c(37.607,
37.662, 37.672)), .Names = c("dates", "temperature", "depth",
"salinity"), row.names = c(NA, -3L), class = "data.frame")
x <- xts(d[,-1], as.POSIXct(d[,1], format="%m/%d/%Y %H:%M"))
apply.daily(x, colMeans)
# temperature depth salinity
# 2012-12-03 12:26:00 9.695967 0.3821167 37.647
I'd add the day, month and year into the data frame and then use aggregate().
First convert your date column into a POSIXct objet:
d$timestamp <- as.POSIXct(d$dates,format = "%m/%d/%Y %H:%M",tz ="GMT")
Then get the date (e.g. 12/03/2012) into a column called Date, try this:
d$Date <- format(d$timestamp,"%y-%m-%d",tz = "GMT")
Next, aggregate by the date:
aggregate(cbind("temperature.mean" = temperature,
"salinity.mean" = salinity) ~ Date,
data = d,
FUN = mean)
Similarly, you can get the month into a column (let's call it M for month), and then...
d$M <- format(d$timestamp,"%B",tz = "GMT")
aggregate(cbind("temperature.mean" = temperature,
"salinity.mean" = salinity) ~ M,
data = d,
FUN = mean)
or if you want year-month
d$YM <- format(d$timestamp,"%y-%B",tz = "GMT")
aggregate(cbind("temperature.mean" = temperature,
"salinity.mean" = salinity) ~ YM,
data = d,
FUN = mean)
If you have any NA values in your data, you may need to account for those:
aggregate(cbind("temperature.mean" = temperature,
"salinity.mean" = salinity) ~ YM,
data = d,
function(x) mean(x,na.rm = TRUE))
Finally, if you want to average by week, you can do that as well. First generate the week number, and then use aggregate() again.
d$W <- format(d$timestamp,"%W",tz = "GMT")
aggregate(cbind("temperature.mean" = temperature,
"salinity.mean" = salinity) ~ W,
data = d,
function(x) mean(x,na.rm = TRUE))
This version of week number defines week 1 as being the week with the first Monday of the year. The weeks are from Monday to Sunday.
Yet, another method using plyr:
df <- structure(list(dates = c("12/03/2012 11:26", "12/03/2012 11:56",
"12/03/2012 12:26"), temperature = c(9.7533, 9.6673, 9.6673),
depth = c(0.48073, 0.33281, 0.33281), salinity = c(37.607,
37.662, 37.672)), .Names = c("dates", "temperature", "depth",
"salinity"), row.names = c(NA, -3L), class = "data.frame")
library(plyr)
# Change date to POSIXct
df$dates <- with(d,as.POSIXct(dates,format="%m/%d/%Y %H:%M"))
# Make new variables, year and month
df <- transform(d,month=as.numeric(format(dates,"%m")),year=as.numeric(format(dates,"%Y")))
## According to year
ddply(df,.(year),summarize,meantemp=mean(temperature),meandepth=mean(depth),meansalinity=mean(salinity))
year meantemp meandepth meansalinity
1 2012 9.695967 0.3821167 37.647
## According to month
ddply(df,.(month),summarize,meantemp=mean(temperature),meandepth=mean(depth),meansalinity=mean(salinity))
month meantemp meandepth meansalinity
1 12 9.695967 0.3821167 37.647
The package hydroTSM holds a multiple functions to creat annual and other summaries:
daily2annual(x, ...)
subdaily2annual(x, ...)
monthly2annual(x, ...)
annualfunction(x, FUN, na.rm = TRUE, ...)
I don't often have to work with dates in R, but I imagine this is fairly easy. I have a column that represents a date in a dataframe. I simply want to create a new dataframe that summarizes a 2nd column by Month/Year using the date. What is the best approach?
I want a second dataframe so I can feed it to a plot.
Any help you can provide will be greatly appreciated!
EDIT: For reference:
> str(temp)
'data.frame': 215746 obs. of 2 variables:
$ date : POSIXct, format: "2011-02-01" "2011-02-01" "2011-02-01" ...
$ amount: num 1.67 83.55 24.4 21.99 98.88 ...
> head(temp)
date amount
1 2011-02-01 1.670
2 2011-02-01 83.550
3 2011-02-01 24.400
4 2011-02-01 21.990
5 2011-02-03 98.882
6 2011-02-03 24.900
I'd do it with lubridate and plyr, rounding dates down to the nearest month to make them easier to plot:
library(lubridate)
df <- data.frame(
date = today() + days(1:300),
x = runif(300)
)
df$my <- floor_date(df$date, "month")
library(plyr)
ddply(df, "my", summarise, x = mean(x))
There is probably a more elegant solution, but splitting into months and years with strftime() and then aggregate()ing should do it. Then reassemble the date for plotting.
x <- as.POSIXct(c("2011-02-01", "2011-02-01", "2011-02-01"))
mo <- strftime(x, "%m")
yr <- strftime(x, "%Y")
amt <- runif(3)
dd <- data.frame(mo, yr, amt)
dd.agg <- aggregate(amt ~ mo + yr, dd, FUN = sum)
dd.agg$date <- as.POSIXct(paste(dd.agg$yr, dd.agg$mo, "01", sep = "-"))
A bit late to the game, but another option would be using data.table:
library(data.table)
setDT(temp)[, .(mn_amt = mean(amount)), by = .(yr = year(date), mon = months(date))]
# or if you want to apply the 'mean' function to several columns:
# setDT(temp)[, lapply(.SD, mean), by=.(year(date), month(date))]
this gives:
yr mon mn_amt
1: 2011 februari 42.610
2: 2011 maart 23.195
3: 2011 april 61.891
If you want names instead of numbers for the months, you can use:
setDT(temp)[, date := as.IDate(date)
][, .(mn_amt = mean(amount)), by = .(yr = year(date), mon = months(date))]
this gives:
yr mon mn_amt
1: 2011 februari 42.610
2: 2011 maart 23.195
3: 2011 april 61.891
As you see this will give the month names in your system language (which is Dutch in my case).
Or using a combination of lubridate and dplyr:
temp %>%
group_by(yr = year(date), mon = month(date)) %>%
summarise(mn_amt = mean(amount))
Used data:
# example data (modified the OP's data a bit)
temp <- structure(list(date = structure(1:6, .Label = c("2011-02-01", "2011-02-02", "2011-03-03", "2011-03-04", "2011-04-05", "2011-04-06"), class = "factor"),
amount = c(1.67, 83.55, 24.4, 21.99, 98.882, 24.9)),
.Names = c("date", "amount"), class = c("data.frame"), row.names = c(NA, -6L))
You can do it as:
short.date = strftime(temp$date, "%Y/%m")
aggr.stat = aggregate(temp$amount ~ short.date, FUN = sum)
Just use xts package for this.
library(xts)
ts <- xts(temp$amount, as.Date(temp$date, "%Y-%m-%d"))
# convert daily data
ts_m = apply.monthly(ts, FUN)
ts_y = apply.yearly(ts, FUN)
ts_q = apply.quarterly(ts, FUN)
where FUN is a function which you aggregate data with (for example sum)
Here's a dplyr option:
library(dplyr)
df %>%
mutate(date = as.Date(date)) %>%
mutate(ym = format(date, '%Y-%m')) %>%
group_by(ym) %>%
summarize(ym_mean = mean(x))
I have a function monyr that I use for this kind of stuff:
monyr <- function(x)
{
x <- as.POSIXlt(x)
x$mday <- 1
as.Date(x)
}
n <- as.Date(1:500, "1970-01-01")
nn <- monyr(n)
You can change the as.Date at the end to as.POSIXct to match the date format in your data. Summarising by month is then just a matter of using aggregate/by/etc.
One more solution:
rowsum(temp$amount, format(temp$date,"%Y-%m"))
For plot you could use barplot:
barplot(t(rowsum(temp$amount, format(temp$date,"%Y-%m"))), las=2)
Also, given that your time series seem to be in xts format, you can aggregate your daily time series to a monthly time series using the mean function like this:
d2m <- function(x) {
aggregate(x, format(as.Date(zoo::index(x)), "%Y-%m"), FUN=mean)
}