Find daily percentile for large data set of irregular data - r

Question

I have a very large data set (> 1 million rows) with percentiles that need to be calculated for all of the same day (e.g., all Jan 1, all Jan 2, ..., all Dec 31). There are many rows of the same year, month and day with different data. Below is an example of the data:
Year Month Day A B C D
2007 Jan 1 1 2 3 4
2007 Jan 1 5 6 7 8
2007 Feb 1 1 2 3 4
2007 Feb 1 5 6 7 8
.
.
2010 Dec 30 1 2 3 4
2010 Dec 30 5 6 7 8
2010 Dec 31 1 2 3 4
2010 Dec 31 5 6 7 8
So to calculate the 95th percentile for Jan 1, it would need to include all Jan 1 for all years (e.g., 2007-2010) and for all columns (A, B, C and D). This is then done for all Jan 2, Jan 3, ..., Dec 30 and Dec 31. This can easily be done with small data sets in Excel by using nested if statements; e.g., ={PERCENTILE(IF(Month($B$2:$B$1000000)="Jan",IF(Day($C$2:$C$1000000)="1",$D$2:$G$1000000)),95%)}
The percentiles could then be added to a a new data table containing only the month and days:
Month Day P95 P05
Jan 1
Jan 2
Jan 3
.
.
Dec 30
Dec 31
Then using the percentiles, I need to evaluate whether each data value in column names A, B, C and D for their respective date (e.g., Jan 1) is larger than P95 or smaller than P05. Then new columns could be added to the first data table containing 1 or 0 (1 if larger or smaller, 0 if not larger or smaller than the percentiles):
Year Month Day A B C D A05 B05 C05 D05 A95 B95 C95 D95
2007 Jan 1 1 2 3 4 1 0 0 0 0 0 0 0
2007 Jan 1 5 6 7 8 0 0 0 0 0 0 1 1
.
.
2010 Dec 31 5 6 7 8 0 0 0 0 0 0 0 1

Answer 1

I've called your data dat:
library(plyr)
library(reshape2)
# melt values so all values are in 1 column
dat_melt <- melt(dat, id.vars=c("Year", "Month", "Day"), variable.name="letter", value.name="value")
# get quantiles, split by day
dat_quantiles <- ddply(dat_melt, .(Month, Day), summarise,
P05=quantile(value, 0.05), P95=quantile(value, 0.95))
# merge original data with quantiles
all_dat <- merge(dat_melt, dat_quantiles)
# See if in bounds
all_dat <- transform(all_dat, less05=ifelse(value < P05, 1, 0), greater95=ifelse(value > P95, 1, 0))
Month Day Year letter value P05 P95 less05 greater95
1 Dec 30 2010 A 1 1.35 7.65 1 0
2 Dec 30 2010 A 5 1.35 7.65 0 0
3 Dec 30 2010 B 2 1.35 7.65 0 0
4 Dec 30 2010 B 6 1.35 7.65 0 0
5 Dec 30 2010 C 3 1.35 7.65 0 0
6 Dec 30 2010 C 7 1.35 7.65 0 0
7 Dec 30 2010 D 4 1.35 7.65 0 0
8 Dec 30 2010 D 8 1.35 7.65 0 1
9 Dec 31 2010 A 1 1.35 7.65 1 0
10 Dec 31 2010 A 5 1.35 7.65 0 0
11 Dec 31 2010 B 2 1.35 7.65 0 0
12 Dec 31 2010 B 6 1.35 7.65 0 0
13 Dec 31 2010 C 3 1.35 7.65 0 0
14 Dec 31 2010 C 7 1.35 7.65 0 0
15 Dec 31 2010 D 4 1.35 7.65 0 0
16 Dec 31 2010 D 8 1.35 7.65 0 1
17 Feb 1 2007 A 1 1.35 7.65 1 0
18 Feb 1 2007 A 5 1.35 7.65 0 0
19 Feb 1 2007 B 2 1.35 7.65 0 0
20 Feb 1 2007 B 6 1.35 7.65 0 0
21 Feb 1 2007 C 3 1.35 7.65 0 0
22 Feb 1 2007 C 7 1.35 7.65 0 0
23 Feb 1 2007 D 4 1.35 7.65 0 0
24 Feb 1 2007 D 8 1.35 7.65 0 1
25 Jan 1 2007 A 1 1.35 7.65 1 0
26 Jan 1 2007 A 5 1.35 7.65 0 0
27 Jan 1 2007 B 2 1.35 7.65 0 0
28 Jan 1 2007 B 6 1.35 7.65 0 0
29 Jan 1 2007 C 3 1.35 7.65 0 0
30 Jan 1 2007 C 7 1.35 7.65 0 0
31 Jan 1 2007 D 4 1.35 7.65 0 0
32 Jan 1 2007 D 8 1.35 7.65 0 1

Answer 2

Something along these lines can be merged to the original dataframe:
aggregate(dfrm[ , c("A","B","C","D")] , list(dfrm$month, dfrm$day),
FUN=quantile, probs=c(0.05,0.95))
Notice I suggested merge(). Your description suggested (but was not explicit) that you wanted all years worth of Jan-1 values to be submitted together. I think this is a lot "easier" than the expression you are using in Excel. This does both 0.05 and 0.95 on all four columns.