Sums for certain colums and whole dataframe - r

Question

I want to analyze meteorological data and find out how often a station (Xn) exceeds a certain treshold per year and the same for the whole data frame (so i get one value per year).
I tried a few things with Aggregate and Dplyr, but it doesn´t really work.
So first I give you my Data Frame (It´s just a dummy DF, the original one has about 80 colums and 10000 rows):
set.seed(123)
df1<-data.frame(replicate(6,sample(0:200,1500,rep=TRUE)))
date_df1<-seq(as.Date("1995-01-01"), by = "day", length.out = 1500)
test_sto<-cbind(date_df1, df1)
test_sto$date_df1<-as.Date(test_sto$date_df1)
test_sto<-test_sto%>% dplyr::mutate( year = lubridate::year(date_df1),
month = lubridate::month(date_df1),
day = lubridate::day(date_df1))
date_df1 X1 X2 X3 X4 X5 X6 year month day
1 1995-01-01 158 149 51 136 70 140 1995 1 1
2 1995-01-02 178 135 134 109 38 144 1995 1 2
3 1995-01-03 13 64 70 148 177 130 1995 1 3
So first, I tried it with the aggregate function, this gives me the following result:
aggregate.data.frame(x=test_sto[2:7] > 120, by = list(test_sto$year), FUN = sum, na.rm=TRUE )
Group.1 X1 X2 X3 X4 X5 X6
1 1995 143 141 159 152 147 144
2 1996 153 141 148 153 160 165
3 1997 148 126 150 149 139 153
This works for each station, but how do i get exceeds per year for the whole data frame?
I tried the following
aggregate.data.frame(x=test_sto[2:7] > 120, by = list(test_sto$year), FUN = colSums(test_sto[2:7] > 120, na.rm = TRUE), na.rm=TRUE )
But this obviously doesn´t work, but i can´t figure it out how to get something like a colSums into this function.
My second approach was with dplyr, but here i´am struggeling with the syntax i think:
test_sto %>% group_by(year) %>%
summarise_all(funs((colSums(test_sto[2:7] > 120, na.rm=TRUE))))
year date_df1 X1 X2 X3 X4 X5 X6 month day
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1995 601 601 601 601 601 601 601 601 601
2 1995 576 576 576 576 576 576 576 576 576
3 1995 610 610 610 610 610 610 610 610 610
4 1995 622 622 622 622 622 622 622 622 622
5 1995 610 610 610 610 610 610 610 610 610
6 1995 616 616 616 616 616 616 616 616 616
7 1996 601 601 601 601 601 601 601 601 601
8 1996 576 576 576 576 576 576 576 576 576
9 1996 610 610 610 610 610 610 610 610 610
10 1996 622 622 622 622 622 622 622 622 622
Theres the same value for each column, which is definitely incorrect, but i can´t find my mistake. And also the years are incorrect somehow.
The goal here is the same as above, how often does a station exceeds the treshold and the same for the whole data frame.

Answer 1

With tidyverse you could do the following:
library(tidyverse)
test_sto %>%
group_by(year) %>%
summarise(across(starts_with("X"), ~sum(. > 120)))
Output
year X1 X2 X3 X4 X5 X6
<dbl> <int> <int> <int> <int> <int> <int>
1 1995 143 141 159 152 147 144
2 1996 153 141 148 153 160 165
3 1997 148 126 150 149 139 153
4 1998 144 150 138 152 145 138
5 1999 13 18 15 16 19 16
And for all stations each year, just group_by(year) after putting into long form with pivot_longer:
test_sto %>%
pivot_longer(cols = starts_with("X")) %>%
group_by(year) %>%
summarise(gtr_120 = sum(value > 120))
Output
year gtr_120
<dbl> <int>
1 1995 886
2 1996 920
3 1997 865
4 1998 867
5 1999 97

Answer 2

Not sure, but I believe this is what you are looking for?
library(data.table)
# make it a data.table
setDT(test_sto)
# part 1: >120 by year
test_sto[, lapply(.SD, function(x) sum(x > 120)),
by = .(year),
.SDcols = patterns("^X")]
# year X1 X2 X3 X4 X5 X6
# 1: 1995 143 141 159 152 147 144
# 2: 1996 153 141 148 153 160 165
# 3: 1997 148 126 150 149 139 153
# 4: 1998 144 150 138 152 145 138
# 5: 1999 13 18 15 16 19 16
# part 2: totals >120 by station over all years
test_sto[, lapply(.SD, function(x) sum(x > 120)),
.SDcols = patterns("^X")]
# X1 X2 X3 X4 X5 X6
# 1: 601 576 610 622 610 616