I have a questions that might be too basic, but here it is...
I want to extract monthly data from a dataset like this:
Date Obs
1 2001-01-01 120
2 2001-01-02 100
3 2001-01-03 150
4 2001-01-04 175
5 2001-01-05 121
6 2001-01-06 100
I just want to get the rows from the data where I have a certain month(e.g. January), this works perfectly:
output=which(strftime(dataset[,1],"%m")=="01",dataset[,1])
However when I try to create a loop to go through all the months using a variable that is declared has character it doesn't work and I only get "FALSE".
value=as.character(k)
output=which(strftime(dataset[,1],"%m")==value,dataset[,1])
Do not parse dates as strings. That is too error prone. Parse dates as dates, and do logical comparisons on them.
Here is one approach, creating January to March data and sub-setting February based on a comparison:
R> output <- data.frame(date=seq(as.Date("2011-01-01"), by=7, length=10),
+ value=cumsum(runif(10)*100))
R> output
date value
1 2011-01-01 8.29916
2 2011-01-08 44.82950
3 2011-01-15 72.08662
4 2011-01-22 134.19277
5 2011-01-29 221.67744
6 2011-02-05 245.77195
7 2011-02-12 314.82081
8 2011-02-19 396.34661
9 2011-02-26 437.14286
10 2011-03-05 442.41321
R> output[ output[,"date"] >= as.Date("2011-02-01") &
+ output[,"date"] <= as.Date("2011-02-28"), ]
date value
6 2011-02-05 245.772
7 2011-02-12 314.821
8 2011-02-19 396.347
9 2011-02-26 437.143
R>
Another approach uses the xts package:
R> oo <- xts(output[,"value"], order.by=output[,"date"])
R> oo
[,1]
2011-01-01 8.29916
2011-01-08 44.82950
2011-01-15 72.08662
2011-01-22 134.19277
2011-01-29 221.67744
2011-02-05 245.77195
2011-02-12 314.82081
2011-02-19 396.34661
2011-02-26 437.14286
2011-03-05 442.41321
R> oo["2011-02-01::2011-02-28"]
[,1]
2011-02-05 245.772
2011-02-12 314.821
2011-02-19 396.347
2011-02-26 437.143
R>
as xts has convenient date parsing for the index; see the package documentation for details.
I'm assuming k is an integer in 1:12. I suspect you may be better off using abbreviated month names:
value <- month.abb[k]
output <- which(strftime(dataset[,1],"%b")==value,dataset[,1])
The reason you way isn't working is because the month number is zero-padded and "1" != "01".
You can also use dates as dates with POSIXlt()$mon
as.POSIXlt(output$date)$mon # Note that Jan = 0 and Feb=1
[1] 0 0 0 0 0 1 1 1 1 2
There are several other packages such as chron, lubridate and gdata that provide date handling functions. I found the functions in lubridate particularly intuitive and less prone to errors in my clumsy hands.
Related
I'm having difficulty calculating the average time between the payment dates for my csv. I have tried multiple methods that I have seen online (changing to data.table, using ddply) with no success
WorkerID PaymentDate
1 2015-07-18
1 2015-08-18
3 2015-09-18
4 2015-10-18
4 2015-11-18
This is an example of my dataset- I wanted to calculate the average time between the PaymentDates (in number of days), in the simplest way possible. I would like to group by the workerID.
Thank you!
This is a perfect job for aggregate(). It groups PaymentDate by WorkerID and applies the function mean(diff(.)) to each group.
tt <- read.table(text="
WorkerID PaymentDate
1 2015-06-18
1 2015-07-18
1 2015-08-18
2 2015-09-18
3 2015-08-18
3 2015-09-18
4 2015-10-18
4 2015-11-18
4 2015-12-16", header=TRUE)
tt$PaymentDate <- as.Date(tt$PaymentDate)
aggregate(PaymentDate ~ WorkerID, data=tt, FUN=function(x) mean(diff(x)))
# WorkerID PaymentDate
# 1 1 30.5
# 2 2 NaN
# 3 3 31.0
# 4 4 29.5
An alternative to AkselA's answer, one can use the data.table package if one prefers this over base R.
This is similar to using aggregate, but may sometimes give a speed boost. In my example below I've handled single times by setting the difference to 0, to illustrate how this can be achieved.
library(lubridate)
library(data.table)
df <- fread("WorkerID PaymentDate
1 2015-07-18
1 2015-08-18
3 2015-09-18
4 2015-10-18
4 2015-11-18")
df[,PaymentDate := as.Date(PaymentDate)]
df[,{
if(length(PaymentDate) > 1){
mean(diff(as.numeric(PaymentDate)))
}else
0
}, by = WorkerID]
I'm working on time-series analyses and I'm hoping to develop multiple datasets with different units of analysis. Namely: the units in data set 1 will be districts in country X for 2-week periods within a span of 4 years (districtYearPeriodCode), the units in data set 2 will be districts in country X for 4-week periods within a span of 4 years, and so forth.
I have created a number of data frames containing start and end dates for each interval, as well as an interval ID. The one below is for the 2-week intervals.
begin <- seq(ymd('2004-01-01'),ymd('2004-06-30'), by = as.difftime(weeks(2)))
end <- seq(ymd('2004-01-14'),ymd('2004-06-30'), by = as.difftime(weeks(2)))
interval <- seq(1,13,1)
df2 <- data.frame(begin, end, interval)
begin end interval
1 2004-01-01 2004-01-14 1
2 2004-01-15 2004-01-28 2
3 2004-01-29 2004-02-11 3
4 2004-02-12 2004-02-25 4
5 2004-02-26 2004-03-10 5
6 2004-03-11 2004-03-24 6
7 2004-03-25 2004-04-07 7
8 2004-04-08 2004-04-21 8
9 2004-04-22 2004-05-05 9
10 2004-05-06 2004-05-19 10
11 2004-05-20 2004-06-02 11
12 2004-06-03 2004-06-16 12
13 2004-06-17 2004-06-30 13
In addition to this I have a data frame that contains observations for events, dates included. It looks something like this:
new.df3 <- data.frame(dates5, districts5)
new.df3
dates5 districts5
1 2004-01-01 d1
2 2004-01-02 d2
3 2004-01-03 d3
4 2004-01-04 d4
5 2004-01-05 d5
Is there a function I can write or a command I can use to end up with something like this?
dates5 districts5 interval5
1 2004-01-01 d1 1
2 2004-01-02 d2 1
3 2004-01-03 d3 1
4 2004-01-04 d4 1
5 2004-01-05 d5 1
I have been trying to find an answer in the lubridate package, or in other threads but all answers seem to be tailored at finding out whether a date falls within a specific time interval instead of identifying the interval a date falls into from a group of intervals.
Much appreiciated!
I used the purrr approached outlined by #alistair in here. I reproduce it below:
elements %>%
map(~intervals$phase[.x >= intervals$start & .x <= intervals$end]) %>%
# Clean up a bit. Shorter, but less readable: map_chr(~.x[1] %||% NA)
map_chr(~ifelse(length(.x) == 0, NA, .x))
## [1] "a" "a" "a" NA "b" "b" "c"
I have a dataframe in R, which has two variables that are dates and I need to calculate the difference in days between them. However, they are formatted as YYYYMMDD. How do I change it to a date format readable in R?
This should work
lubridate::ymd(given_date_format)
I like anydate() from the anytime package. Quick demo, with actual data:
R> set.seed(123) # be reproducible
R> data <- data.frame(inp=Sys.Date() + cumsum(runif(10)*10))
R> data$ymd <- format(data$inp, "%Y%m%d") ## as yyyymmdd
R> data$int <- as.integer(data$ymd) ## same as integer
R> library(anytime)
R> data$diff1 <- c(NA, diff(anydate(data$ymd))) # reads YMD
R> data$diff2 <- c(NA, diff(anydate(data$int))) # also reads int
R> data
inp ymd int diff1 diff2
1 2017-06-23 20170623 20170623 NA NA
2 2017-07-01 20170701 20170701 8 8
3 2017-07-05 20170705 20170705 4 4
4 2017-07-14 20170714 20170714 9 9
5 2017-07-24 20170724 20170724 10 10
6 2017-07-24 20170724 20170724 0 0
7 2017-07-29 20170729 20170729 5 5
8 2017-08-07 20170807 20170807 9 9
9 2017-08-13 20170813 20170813 6 6
10 2017-08-17 20170817 20170817 4 4
R>
Here the first column is actual dates we work from. Columns two and three are then generates to match OP's requirement: YMD, either in character or integer.
We then compute differences on them, account for the first 'lost' data point differences when we have no predecessor and show that either date format works.
I have a data frame with 3262 rows and 10 columns. One of the columns has date with format YY-MM-DD. I want to store all the rows with 10 different dates in a different data frame so I tried :
newdata= df[df$Date %in% as.Date(c('2011-08-05','2012-1-13','2012-03-2','2014-04-01')),]
but nothing. I thought it might need to specify again the format so I tried:
df$Date <- as.Date( as.character(df$Date), "%d-%m-%y")
newdata= df[df$Date %in% as.Date(c('2011-08-05','2012-1-13','2012-03-2','2014-04-01')),]
All I get is an empty data frame saying no data available in table. At which point I made the mistake (something stupid I guess)?
I created an example:
set.seed(1)
df=data.frame(col1=seq(1,10),
col2=seq(1,0),
Date=as.Date(floor(runif(min=15550,max=17000,n=10)),origin="1970-01-01"))
> df
col1 col2 Date
1 1 1 2013-08-17
2 2 0 2014-01-19
3 3 1 2014-11-06
4 4 0 2016-03-06
5 5 1 2013-05-17
6 6 0 2016-02-21
7 7 1 2016-04-28
8 8 0 2015-03-14
9 9 1 2015-01-27
10 10 0 2012-10-26
Using the same code you provided:
newdata= df[df$Date %in% as.Date(c('2013-08-17','2015-01-27')),]
Gives me
> newdata
col1 col2 Date
1 1 1 2013-08-17
9 9 1 2015-01-27
Are you sure that str(df$Date) shows you really have a Date format?
> str(df$Date)
Date[1:10], format: "2013-08-17" "2014-01-19" "2014-11-06" "2016-03-06" "2013-05-17" "2016-02-21" "2016-04-28" "2015-03-14" "2015-01-27" "2012-10-26"
I have a simple data set which has a date column and a value column. I noticed that the date sometimes comes in as mmddyy (%m/%d/%y) format and other times in mmddYYYY (%m/%d/%Y) format. What is the best way to standardize the dates so that i can do other calculations without this formatting causing issues?
I tried the answers provided here
Changing date format in R
and here
How to change multiple Date formats in same column
Neither of these were able to fix the problem.
Below is a sample of the data
Date, Market
12/17/09,1.703
12/18/09,1.700
12/21/09,1.700
12/22/09,1.590
12/23/2009,1.568
12/24/2009,1.520
12/28/2009,1.500
12/29/2009,1.450
12/30/2009,1.450
12/31/2009,1.450
1/4/2010,1.440
When i read it into a new vector using something like this
dt <- as.Date(inp$Date, format="%m/%d/%y")
I get the following output for the above segment
dt Market
2009-12-17 1.703
2009-12-18 1.700
2009-12-21 1.700
2009-12-22 1.590
2020-12-23 1.568
2020-12-24 1.520
2020-12-28 1.500
2020-12-29 1.450
2020-12-30 1.450
2020-12-31 1.450
2020-01-04 1.440
As you can see we skipped from 2009 to 2020 at 12/23 because of change in formatting. Any help is appreciated. Thanks.
> dat$Date <- gsub("[0-9]{2}([0-9]{2})$", "\\1", dat$Date)
> dat$Date <- as.Date(dat$Date, format = "%m/%d/%y")
> dat
Date Market
# 1 2009-12-17 1.703
# 2 2009-12-18 1.700
# 3 2009-12-21 1.700
# 4 2009-12-22 1.590
# 5 2009-12-23 1.568
# 6 2009-12-24 1.520
# 7 2009-12-28 1.500
# 8 2009-12-29 1.450
# 9 2009-12-30 1.450
# 10 2009-12-31 1.450
# 11 2010-01-04 1.440