Subset csv data based on Pentad dates using R - r
I would like to subset the following csv file based on Pentad dates (non overlapping average of dates). For example:
1.January 1 to January 5
2.January 6 to January 10
...
73.December 27 to December 31.
Here's the complete list of pentad dates:
List of Pentad dates
The Complete Data
Sample Data
SN,CY,Y,M,D,H,lat,lon,cat
198305,5,1983,8,5,0,9.1,140.7,"TD"
198305,5,1983,8,5,6,9.3,140.5,"TD"
198305,5,1983,8,5,12,9.6,139.9,"TD"
198305,5,1983,8,5,18,9.9,139.4,"TS"
198305,5,1983,8,6,0,10.2,138.8,"TS"
198305,5,1983,8,6,6,11,138.1,"TS"
198305,5,1983,8,6,12,11.8,137.3,"TS"
198305,5,1983,8,6,18,12.4,136.4,"Cat1"
198305,5,1983,8,7,0,12.8,135.8,"Cat1"
198305,5,1983,8,7,6,13.6,134.7,"Cat1"
198305,5,1983,8,7,12,14.4,133.9,"Cat2"
198305,5,1983,8,7,18,15,133.5,"Cat4"
198305,5,1983,8,8,0,15.8,132.8,"Cat4"
198305,5,1983,8,8,6,16.3,132.4,"Cat4"
198305,5,1983,8,8,12,17.1,132,"Cat5"
198305,5,1983,8,8,18,17.4,131.4,"Cat5"
198305,5,1983,8,9,0,17.8,130.8,"Cat5"
198305,5,1983,8,9,6,18.1,130.7,"Cat4"
198305,5,1983,8,9,12,18.7,130.3,"Cat4"
198305,5,1983,8,9,18,18.9,130.4,"Cat4
SN is a unique identifier, Y is years, M is months, D is days,H is hours. If the unique number falls in one pentad, it should not be included in the next subset anymore.
I have tried this for August (based from previous post):
P1 <- c(1,6,11,16,21,26)
P6 <- c(5,10,15,20,25,30)
res <- Map(function(x,y) subset(df1, M==8 & D >=x & D <= y), d1, d2)
But I'm having a problem with mapping with the starting pentads (P7) because it includes January 31 to February 4.
Can anyone suggest any methods to do this in R? Ill appreciate any help.
library(stringr)
df$Date = paste(df$Y, str_pad(df$M,2,'left','0'), str_pad(df$D,2,'left','0'), sep='-')
# Extract day of year (int 0 to 365) from POSIXlt date
df$yday = as.POSIXlt(df$Date)$yday + 1
Now it's trivial:
df$pentad = ceiling(df$yday/5)
Related
convert irregular 6hourly data to daily accumulated using R
I have the following data: Date,Rain 1979_8_9_0,8.775 1979_8_9_6,8.775 1979_8_9_12,8.775 1979_8_9_18,8.775 1979_8_10_0,0 1979_8_10_6,0 1979_8_10_12,0 1979_8_10_18,0 1979_8_11_0,8.025 1979_8_12_12,0 1979_8_12_18,0 1979_8_13_0,8.025 [1] The data is six hourly but some dates have incomplete 6 hourly data. For example, August 11 1979 has only one value at 00H. I would like to get the daily accumulated from this kind of data using R. Any suggestion on how to do this easily in R? I'll appreciate any help.
You can transform your data to dates very easily with: dat$Date <- as.Date(strptime(dat$Date, '%Y_%m_%d_%H')) After that you should aggregate with: aggregate(Rain ~ Date, dat, sum) The result: Date Rain 1 1979-08-09 35.100 2 1979-08-10 0.000 3 1979-08-11 8.025 4 1979-08-12 0.000 5 1979-08-13 8.025 Based on the comment of Henrik, you can also transform to dates with: dat$Date <- as.Date(dat$Date, '%Y_%m_%d')
# split the "date" variable into new, separate variable splitDate <- stringr::str_split_fixed(string = df$Date, pattern = "_", n = 4) df$Day <- splitDate[,3] # split data by Day, loop over each split and add rain variable unlist(lapply(split(df$Rain, df$Day), sum))
R Programming 30 day Months
I'm currently writing a script in the R Programming Language and I've hit a snag. I have time series data organized in a way where there are 30 days in each month for 12 months in 1 year. However, I need the data organized in a proper 365 days in a year calendar, as in 30 days in a month, 31 days in a month, etc. Is there a simple way for R to recognize there are 30 days in a month and to operate within that parameter? At the moment I have my script converting the number of days from the source in UNIX time and it counts up. For example: startingdate <- "20060101" endingdate <- "20121230" date <- seq(from = as.Date(startingdate, "%Y%m%d"), to = as.Date(endingdate, "%Y%m%d"), by = "days") This would generate an array of dates with each month having 29 days/30 days/31 days etc. However, my data is currently organized as 30 days per month, regardless of 29 days or 31 days present. Thanks.
The first 4 solutions are basically variations of the same theme using expand.grid. (3) uses magrittr and the others use no packages. The last two work by creating long sequence of numbers and then picking out the ones that have month and day in range.
1) apply This gives a series of yyyymmdd numbers such that there are 30 days in each month. Note that the line defining yrs in this case is the same as yrs <- 2006:2012 so if the years are handy we could shorten that line. Omit as.numeric in the line defining s if you want character string output instead. Also, s and d are the same because we have whole years so we could omit the line defining d and use s as the answer in this case and also in general if we are always dealing with whole years.
startingdate <- "20060101"
endingdate <- "20121230"
yrs <- seq(as.numeric(substr(startingdate, 1, 4)), as.numeric(substr(endingdate, 1, 4)))
g <- expand.grid(yrs, sprintf("%02d", 1:12), sprintf("%02d", 1:30))
s <- sort(as.numeric(apply(g, 1, paste, collapse = "")))
d <- s[ s >= startingdate & s <= endingdate ] # optional if whole years
Run some checks.
head(d)
## [1] 20060101 20060102 20060103 20060104 20060105 20060106
tail(d)
## 20121225 20121226 20121227 20121228 20121229 20121230
length(d) == length(2006:2012) * 12 * 30
## [1] TRUE
2) no apply An alternative variation would be this. In this and the following solutions we are using yrs as calculated in (1) so we omit it to avoid redundancy. Also, in this and the following solutions, the corresponding line to the one setting d is omitted, again, to avoid redundancy -- if you don't have whole years then add the line defining d in (1) replacing s in that line with s2.
g2 <- expand.grid(yr = yrs, mon = sprintf("%02d", 1:12), day = sprintf("%02d", 1:30))
s2 <- with(g2, sort(as.numeric(paste0(yr, mon, day))))
3) magrittr This could also be written using magrittr like this:
library(magrittr)
expand.grid(yr = yrs, mon = sprintf("%02d", 1:12), day = sprintf("%02d", 1:30)) %>%
with(paste0(yr, mon, day)) %>%
as.numeric %>%
sort -> s3
4) do.call Another variation.
g4 <- expand.grid(yrs, 1:12, 1:30)
s4 <- sort(as.numeric(do.call("sprintf", c("%d%02d%02d", g4))))
5) subset sequence Create a sequence of numbers from the starting date to the ending date and if each number is of the form yyyymmdd pick out those for which mm and dd are in range.
seq5 <- seq(as.numeric(startingdate), as.numeric(endingdate))
d5 <- seq5[ seq5 %/% 100 %% 100 %in% 1:12 & seq5 %% 100 %in% 1:30]
6) grep Using seq5 from (5)
d6 <- as.numeric(grep("(0[1-9]|1[0-2])(0[1-9]|[12][0-9]|30)$", seq5, value = TRUE))
Here's an alternative: date <- unclass(startingdate):unclass(endingdate) %% 30L month <- rep(1:12, each = 30, length.out = NN <- length(date)) year <- rep(1:(NN %/% 360 + 1), each = 360, length.out = NN) (of course, we can easily adjust by adding constants to taste if you want a specific day to be 0, or a specific month, etc.)
Convert Julian date to calendar dates within a data frame
I have a data frame > df Age year sex 12 80210 F 13 9123 M I want to convert the year 80210 as 26june1982. How can I do this that the new data frame contains year in day month year formate from Julian days.
You can convert Julian dates to dates using as.Date and specifying the appropriate origin:
as.Date(8210, origin=as.Date("1960-01-01"))
#[1] "1982-06-24"
However, 80210 needs an origin pretty long ago.
You should substract the origin from the year column. as.Date(c(80210,9123)-80210,origin='1982-06-26') [1] "1982-06-26" "1787-11-08"
There are some options for doing this job in the R package date.
See for example on page 4, the function date.mmddyy, which says:
Given a vector of Julian dates, this returns them in the form “10/11/89”, “28/7/54”, etc.
Try this code:
age = c(12,13)
year = c(8210,9123)
sex = c("F","M")
df = data.frame(cbind(age,year,sex))
library(date)
date = date.mmddyy(year, sep = "/")
df2 = transform(df,year=date) #hint provided by jilber
df2
age year sex
1 12 6/24/82 F
2 13 12/23/84 M
Extract Date in R
I struggle mightily with dates in R and could do this pretty easily in SPSS, but I would love to stay within R for my project. I have a date column in my data frame and want to remove the year completely in order to leave the month and day. Here is a peak at my original data. > head(ds$date) [1] "2003-10-09" "2003-10-11" "2003-10-13" "2003-10-15" "2003-10-18" "2003-10-20" > class((ds$date)) [1] "Date" I "want" it to be. > head(ds$date) [1] "10-09" "10-11" "10-13" "10-15" "10-18" "10-20" > class((ds$date)) [1] "Date" If possible, I would love to set the first date to be October 1st instead of January 1st. Any help you can provide will be greatly appreciated. EDIT: I felt like I should add some context. I want to plot an NHL player's performance over the course of a season which starts in October and ends in April. To add to this, I would like to facet the plots by each season which is a separate column in my data frame. Because I want to compare cumulative performance over the course of the season, I believe that I need to remove the year portion, but maybe I don't; as I indicated, I struggle with dates in R. What I am looking to accomplish is a plot that compares cumulative performance over relative dates by season and have the x-axis start in October and end in April.
> d = as.Date("2003-10-09", format="%Y-%m-%d")
> format(d, "%m-%d")
[1] "10-09"
Is this what you are looking for?
library(ggplot2)
## make up data for two seasons a and b
a = as.Date("2010/10/1")
b = as.Date("2011/10/1")
a.date <- seq(a, by='1 week', length=28)
b.date <- seq(b, by='1 week', length=28)
## make up some score data
a.score <- abs(trunc(rnorm(28, mean = 10, sd = 5)))
b.score <- abs(trunc(rnorm(28, mean = 10, sd = 5)))
## create a data frame
df <- data.frame(a.date, b.date, a.score, b.score)
df
## Since I am using ggplot I better create a "long formated" data frame
df.molt <- melt(df, measure.vars = c("a.score", "b.score"))
levels(df.molt$variable) <- c("First season", "Second season")
df.molt
Then, I am using ggplot2 for plotting the data:
## plot it
ggplot(aes(y = value, x = a.date), data = df.molt) + geom_point() +
geom_line() + facet_wrap(~variable, ncol = 1) +
scale_x_date("Date", format = "%m-%d")
If you want to modify the x-axis (e.g., display format), then you'll probably be interested in scale_date.
You have to remember Date is a numeric format, representing the number of days passed since the "origin" of the internal date counting :
> str(Date)
Class 'Date' num [1:10] 14245 14360 14475 14590 14705 ...
This is the same as in EXCEL, if you want a reference. Hence the solution with format as perfectly valid.
Now if you want to set the first date of a year as October 1st, you can construct some year index like this :
redefine.year <- function(x,start="10-1"){
year <- as.numeric(strftime(x,"%Y"))
yearstart <- as.Date(paste(year,start,sep="-"))
year + (x >= yearstart) - min(year) + 1
}
Testing code :
Start <- as.Date("2009-1-1")
Stop <- as.Date("2011-11-1")
Date <- seq(Start,Stop,length.out=10)
data.frame( Date=as.character(Date),
year=redefine.year(Date))
gives
Date year
1 2009-01-01 1
2 2009-04-25 1
3 2009-08-18 1
4 2009-12-11 2
5 2010-04-05 2
6 2010-07-29 2
7 2010-11-21 3
8 2011-03-16 3
9 2011-07-09 3
10 2011-11-01 4
Split date data (m/d/y) into 3 separate columns
I need to convert date (m/d/y format) into 3 separate columns on which I hope to run an algorithm.(I'm trying to convert my dates into Julian Day Numbers). Saw this suggestion for another user for separating data out into multiple columns using Oracle. I'm using R and am throughly stuck about how to code this appropriately. Would A1,A2...represent my new column headings, and what would the format difference be with the "update set" section? update <tablename> set A1 = substr(ORIG, 1, 4), A2 = substr(ORIG, 5, 6), A3 = substr(ORIG, 11, 6), A4 = substr(ORIG, 17, 5); I'm trying hard to improve my skills in R but cannot figure this one...any help is much appreciated. Thanks in advance... :)
I use the format() method for Date objects to pull apart dates in R. Using Dirk's datetext, here is how I would go about breaking up a date into its constituent parts:
datetxt <- c("2010-01-02", "2010-02-03", "2010-09-10")
datetxt <- as.Date(datetxt)
df <- data.frame(date = datetxt,
year = as.numeric(format(datetxt, format = "%Y")),
month = as.numeric(format(datetxt, format = "%m")),
day = as.numeric(format(datetxt, format = "%d")))
Which gives:
> df
date year month day
1 2010-01-02 2010 1 2
2 2010-02-03 2010 2 3
3 2010-09-10 2010 9 10
Note what several others have said; you can get the Julian dates without splitting out the various date components. I added this answer to show how you could do the breaking apart if you needed it for something else.
Given a text variable x, like this: > x [1] "10/3/2001" then: > as.Date(x,"%m/%d/%Y") [1] "2001-10-03" converts it to a date object. Then, if you need it: > julian(as.Date(x,"%m/%d/%Y")) [1] 11598 attr(,"origin") [1] "1970-01-01" gives you a Julian date (relative to 1970-01-01). Don't try the substring thing... See help(as.Date) for more.
Quick ones:
Julian date converters already exist in base R, see eg help(julian).
One approach may be to parse the date as a POSIXlt and to then read off the components. Other date / time classes and packages will work too but there is something to be said for base R.
Parsing dates as string is almost always a bad approach.
Here is an example:
datetxt <- c("2010-01-02", "2010-02-03", "2010-09-10")
dates <- as.Date(datetxt) ## you could examine these as well
plt <- as.POSIXlt(dates) ## now as POSIXlt types
plt[["year"]] + 1900 ## years are with offset 1900
#[1] 2010 2010 2010
plt[["mon"]] + 1 ## and months are on the 0 .. 11 intervasl
#[1] 1 2 9
plt[["mday"]]
#[1] 2 3 10
df <- data.frame(year=plt[["year"]] + 1900,
month=plt[["mon"]] + 1, day=plt[["mday"]])
df
# year month day
#1 2010 1 2
#2 2010 2 3
#3 2010 9 10
And of course
julian(dates)
#[1] 14611 14643 14862
#attr(,"origin")
#[1] "1970-01-01"
To convert date (m/d/y format) into 3 separate columns,consider the df,
df <- data.frame(date = c("01-02-18", "02-20-18", "03-23-18"))
df
date
1 01-02-18
2 02-20-18
3 03-23-18
Convert to date format
df$date <- as.Date(df$date, format="%m-%d-%y")
df
date
1 2018-01-02
2 2018-02-20
3 2018-03-23
To get three seperate columns with year, month and date,
library(lubridate)
df$year <- year(ymd(df$date))
df$month <- month(ymd(df$date))
df$day <- day(ymd(df$date))
df
date year month day
1 2018-01-02 2018 1 2
2 2018-02-20 2018 2 20
3 2018-03-23 2018 3 23
Hope this helps.
Hi Gavin: another way [using your idea] is: The data-frame we will use is oilstocks which contains a variety of variables related to the changes over time of the oil and gas stocks. The variables are: colnames(stocks) "bpV" "bpO" "bpC" "bpMN" "bpMX" "emdate" "emV" "emO" "emC" "emMN" "emMN.1" "chdate" "chV" "cbO" "chC" "chMN" "chMX" One of the first things to do is change the emdate field, which is an integer vector, into a date vector. realdate<-as.Date(emdate,format="%m/%d/%Y") Next we want to split emdate column into three separate columns representing month, day and year using the idea supplied by you. > dfdate <- data.frame(date=realdate) year=as.numeric (format(realdate,"%Y")) month=as.numeric (format(realdate,"%m")) day=as.numeric (format(realdate,"%d")) ls() will include the individual vectors, day, month, year and dfdate. Now merge the dfdate, day, month, year into the original data-frame [stocks]. ostocks<-cbind(dfdate,day,month,year,stocks) colnames(ostocks) "date" "day" "month" "year" "bpV" "bpO" "bpC" "bpMN" "bpMX" "emdate" "emV" "emO" "emC" "emMN" "emMX" "chdate" "chV" "cbO" "chC" "chMN" "chMX" Similar results and I also have date, day, month, year as separate vectors outside of the df.