Dplyr: Summarise, mutate and rank within group - r

When I execute following query on mtcars data set, I get below results.
mtcars %>%
group_by(cyl,gear) %>%
summarise(total_cnt = n(), totalwt = sum(wt)) %>%
arrange(cyl, gear, desc(total_cnt), desc(totalwt)) %>%
mutate(rank = dense_rank(desc(total_cnt))) %>%
arrange(rank)
cyl gear total totalwt rank
<dbl> <dbl> <int> <dbl> <int>
1 4 4 8 19.025 1
2 6 4 4 12.375 1
3 8 3 12 49.249 1
4 4 5 2 3.653 2
5 6 3 2 6.675 2
6 8 5 2 6.740 2
7 4 3 1 2.465 3
8 6 5 1 2.770 3
Now within each group (of ranks), I want to sub rank the observations based on totalwt, so final output should look like (desc order of totalwt within each rank group)
cyl gear total_cnt totalwt rank subrank
<dbl> <dbl> <int> <dbl> <int> <int>
1 4 4 8 19.025 1 2
2 6 4 4 12.375 1 3
3 8 3 12 49.249 1 1
4 4 5 2 3.653 2 3
5 6 3 2 6.675 2 2
6 8 5 2 6.740 2 1
7 4 3 1 2.465 3 2
8 6 5 1 2.770 3 1
Then finally top 1 where each rank where sub rank = 1, so output would be:
cyl gear total_cnt totalwt rank subrank
<dbl> <dbl> <int> <dbl> <int> <int>
3 8 3 12 49.249 1 1
6 8 5 2 6.740 2 1
8 6 5 1 2.770 3 1


If 'mtcars1' is output from the OP's code, we can use rank to create the 'subrank' after grouping by 'rank'
mtcars2 <- mtcars1 %>%
group_by(rank) %>%
mutate(subrank = rank(-totalwt))
mtcars2
# cyl gear total_cnt totalwt rank subrank
# <dbl> <dbl> <int> <dbl> <int> <dbl>
#1 4 4 8 19.025 1 2
#2 6 4 4 12.375 1 3
#3 8 3 12 49.249 1 1
#4 4 5 2 3.653 2 3
#5 6 3 2 6.675 2 2
#6 8 5 2 6.740 2 1
#7 4 3 1 2.465 3 2
#8 6 5 1 2.770 3 1
Then, we filter the rows where 'subrank' is 1
mtcars2 %>%
filter(subrank ==1)
# cyl gear total_cnt totalwt rank subrank
# <dbl> <dbl> <int> <dbl> <int> <dbl>
#1 8 3 12 49.249 1 1
#2 8 5 2 6.740 2 1
#3 6 5 1 2.770 3 1

Related

reset a ranking when a variable exceeds a value using dplyr

Suppose I have the following data:
df <- tibble(ID=c(1,2,3,4,5,6,7,8,9,10),
ID2=c(1,1,1,1,2,2,2,3,4,4),
VAR=c(25,10,120,60,85,90,20,40,60,150))
I want to add a new column with a ranking that would be reset either when the ID2 changes or when VAR is greater than 100.
The desired result is:
# A tibble: 10 x 4
ID ID2 VAR RANK
<dbl> <dbl> <dbl> <dbl>
1 1 1 25 1
2 2 1 10 2
3 3 1 120 1
4 4 1 60 2
5 5 2 85 1
6 6 2 90 2
7 7 2 20 3
8 8 3 40 1
9 9 4 60 1
10 10 4 150 1
I know how to add a new column with a ranking that would be reset only when the ID2 changes:
df %>%
arrange(ID2) %>%
group_by(ID2) %>%
mutate(RANK = row_number())
... but treating both conditions at the same time is more difficult. How should I do using dplyr?

You can group_by ID2 and cumsum(VAR > 100), i.e.:
library(dplyr)
df %>%
group_by(ID2, cumVAR = cumsum(VAR > 100)) %>%
mutate(RANK = row_number())
output
# A tibble: 10 x 5
# Groups: ID2, cumVAR [6]
ID ID2 VAR cumVAR RANK
<dbl> <dbl> <dbl> <int> <int>
1 1 1 25 0 1
2 2 1 10 0 2
3 3 1 120 1 1
4 4 1 60 1 2
5 5 2 85 1 1
6 6 2 90 1 2
7 7 2 20 1 3
8 8 3 40 1 1
9 9 4 60 1 1
10 10 4 150 2 1

rowid from data.table would be useful as well
library(dplyr)
library(data.table)
df %>%
mutate(RANK = rowid(ID2, cumsum(VAR > 100)))
-output
# A tibble: 10 × 4
ID ID2 VAR RANK
<dbl> <dbl> <dbl> <int>
1 1 1 25 1
2 2 1 10 2
3 3 1 120 1
4 4 1 60 2
5 5 2 85 1
6 6 2 90 2
7 7 2 20 3
8 8 3 40 1
9 9 4 60 1
10 10 4 150 1

Using dplyr to apply a function to each group of a dataset

I have this dataframe.
Sub <- c(1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2)
trial <-c(1,1,1,1,2,2,2,2,2,2,1,1,1,1,2,2,2,2,2,2)
One <- c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)
Two <- c(1,0,0,0,1,0,0,0,0,0,1,0,0,0,1,1,1,0,0,1)
Three <- c(2,0,0,1,3,0,0,0,0,1,7,8,0,0,0,1,1,1,1,0)
Four <- c(3,4,5,4,3,4,5,6,7,8,6,5,4,5,6,7,6,5,6,5)
Five <- c(3,4,5,4,6,7,5,4,3,2,3,4,5,4,3,5,7,4,3,5)
Six <- c(3,4,5,4,6,7,5,4,3,2,3,4,5,4,3,5,7,4,3,5)
Seven <- c(3,4,5,4,9,7,5,4,3,2,3,4,5,4,3,5,7,4,3,5)
dat <- data.frame(Sub, trial, One, Two, Three, Four, Five, Six, Seven)
I created this function to calculate the correlation among my variables.
fun <- function(a,b,c,d,e,f,g) {
v = cor(a,b)
v1 = cor(a,c)
v2 = cor(a,d)
v3 = cor(a,e)
v4 = cor(a,f)
v5 = cor(a,g)
return(c(v,v1,v2,v3,v4,v5))
}
I need to apply this function to each group of my dataset (Sub,trial).
dat %>%
group_by(Sub,trial) %>%
summarize(as.data.frame(matrix(fun(One, Two, Three, Four, Five, Six, Seven), nr = 1)))
However I got this result:
Sub trial V1 V2 V3 V4 V5 V6
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1 1 NA NA NA NA NA NA
2 1 2 NA NA NA NA NA NA
3 2 1 NA NA NA NA NA NA
4 2 2 NA NA NA NA NA NA
Sub/trial are well grouped. But I got NA results for the other variables.
Do you have any advice?
Thank you.

The solution by user #user438383 is the correct one.
The reason you get NA has nothing to do with applying the function.
As you get the the warning that standard deviation is zero you may consider this:
R - Warning message: "In cor(...): the standard deviation is zero"
Here is an example:
# generate a list of dataframes with your groups:
my_list <- dat %>%
group_by(Sub, trial) %>%
group_split()
[[1]]
# A tibble: 5 x 9
Sub trial One Two Three Four Five Six Seven
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1 1 1 1 2 3 3 3 3
2 1 1 1 0 0 4 4 4 4
3 1 1 1 0 0 5 5 5 5
4 1 1 1 0 1 4 4 4 4
5 1 1 1 1 7 6 3 3 3
[[2]]
# A tibble: 6 x 9
Sub trial One Two Three Four Five Six Seven
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1 2 1 1 3 3 6 6 9
2 1 2 1 0 0 4 7 7 7
3 1 2 1 0 0 5 5 5 5
4 1 2 1 0 0 6 4 4 4
5 1 2 1 0 0 7 3 3 3
6 1 2 1 0 1 8 2 2 2
[[3]]
# A tibble: 3 x 9
Sub trial One Two Three Four Five Six Seven
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 2 1 1 0 8 5 4 4 4
2 2 1 1 0 0 4 5 5 5
3 2 1 1 0 0 5 4 4 4
[[4]]
# A tibble: 6 x 9
Sub trial One Two Three Four Five Six Seven
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 2 2 1 1 0 6 3 3 3
2 2 2 1 1 1 7 5 5 5
3 2 2 1 1 1 6 7 7 7
4 2 2 1 0 1 5 4 4 4
5 2 2 1 0 1 6 3 3 3
6 2 2 1 1 0 5 5 5 5
Now apply cor to the first group
my_list[[1]] %>%
summarise(across(Two:Seven, ~cor(One, .)))
# gives:
# A tibble: 1 x 6
Two Three Four Five Six Seven
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 NA NA NA NA NA NA
Warning messages:
1: In cor(One, Two) : Standardabweichung ist Null
2: In cor(One, Three) : Standardabweichung ist Null
3: In cor(One, Four) : Standardabweichung ist Null
4: In cor(One, Five) : Standardabweichung ist Null
5: In cor(One, Six) : Standardabweichung ist Null
6: In cor(One, Seven) : Standardabweichung ist Null
# or correlation of two columns only One and two of group one
cor(my_list[[1]]$One, my_list[[1]]$Two)
# gives:
[1] NA
Warning message:
In cor(my_list[[1]]$One, my_list[[1]]$Two) : Standardabweichung ist Null
An extrapolated example with the mtcars dataset:
mtcars %>%
relocate(cyl, vs, everything()) %>%
group_by(cyl, vs) %>%
summarise(across(hp:carb, ~cor(., mpg)))
cyl vs hp drat wt qsec am gear carb
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 4 0 NA NA NA NA NA NA NA
2 4 1 -0.522 0.466 -0.721 -0.296 0.557 0.442 -0.189
3 6 0 -1 1 -0.101 0.931 NA -1 -1
4 6 1 -0.248 -0.249 -0.936 -0.0424 NA -0.442 -0.442
5 8 0 -0.284 0.0479 -0.650 -0.104 0.0496 0.0496 -0.394
Warning messages:
1: In cor(am, mpg) : Standardabweichung ist Null
2: In cor(am, mpg) : Standardabweichung ist Null

DPLYR - merging rows together using a column value as a conditional

I have a series of rows in a single dataframe. I'm trying to aggregate the first two rows for each ID- i.e. - I want to combine events 1 and 2 for ID 1 into a single row, events 1 and 2 for ID 2 into a singlw row etc, but leave event 3 completely untouched.
id <- c(1,1,1,2,2,2,3,3,3,4,4,4,5,5,5)
event <- c(1,2,3,1,2,3,1,2,3,1,2,3,1,2,3)
score <- c(3,NA,1,3,NA,2,6,NA,1,8,NA,2,4,NA,1)
score2 <- c(NA,4,1,NA,5,2,NA,0,3,NA,5,6,NA,8,7)
df <- tibble(id, event, score, score2)
# A tibble: 15 x 4
id event score score2
<dbl> <dbl> <dbl> <dbl>
1 1 1 3 NA
2 1 2 NA 4
3 1 3 1 1
4 2 1 3 NA
5 2 2 NA 5
6 2 3 2 2
7 3 1 6 NA
8 3 2 NA 0
9 3 3 1 3
10 4 1 8 NA
11 4 2 NA 5
12 4 3 2 6
13 5 1 4 NA
14 5 2 NA 8
15 5 3 1 7
I've tried :
df_merged<- df %>% group_by (id) %>% summarise_all(funs(min(as.character(.),na.rm=TRUE))),
which aggregates these nicely, but then I struggle to merge these back into the orignal dataframe/tibble (there are really about 300 different "score" columns in the full dataset, so a right_join is a headache with score.x, score.y, score2.x, score2.y all over the place...)
Ideally, the situation would need to be dplyr as the rest of my code runs on this!
EDIT:
Ideally, my expected output would be:
# A tibble: 10 x 4
id event score score2
<dbl> <dbl> <dbl> <dbl>
1 1 1 3 4
3 1 3 1 1
4 2 1 3 5
6 2 3 2 2
7 3 1 6 0
9 3 3 1 3
10 4 1 8 5
12 4 3 2 6
13 5 1 4 8
15 5 3 1 7

We may change the order of NA elements with replace
library(dplyr)
df %>%
group_by(id) %>%
mutate(across(starts_with('score'),
~replace(., 1:2, .[1:2][order(is.na(.[1:2]))]))) %>%
ungroup %>%
filter(if_all(starts_with('score'), Negate(is.na)))
-output
# A tibble: 10 x 4
id event score score2
<dbl> <dbl> <dbl> <dbl>
1 1 1 3 4
2 1 3 1 1
3 2 1 3 5
4 2 3 2 2
5 3 1 6 0
6 3 3 1 3
7 4 1 8 5
8 4 3 2 6
9 5 1 4 8
10 5 3 1 7

Here is an alternative way to achieve your task with fill from tidyr package:
library(dplyr)
library(tidyr)
df %>%
group_by(id) %>%
fill(everything(), .direction = "down") %>%
fill(everything(), .direction = "up") %>%
slice(1,3)
id event score score2
<dbl> <dbl> <dbl> <dbl>
1 1 1 3 4
2 1 3 1 1
3 2 1 3 5
4 2 3 2 2
5 3 1 6 0
6 3 3 1 3
7 4 1 8 5
8 4 3 2 6
9 5 1 4 8
10 5 3 1 7

How about this?
library(dplyr)
df_e12 <- df %>%
filter(event %in% c(1, 2)) %>%
group_by(id) %>%
mutate(across(starts_with("score"), ~min(.x, na.rm = TRUE))) %>%
ungroup() %>%
distinct(id, .keep_all = TRUE)
df_e3 <- df %>%
filter(event == 3)
df <- bind_rows(df_e12, df_e3) %>%
arrange(id, event)
df
> df
# A tibble: 10 x 4
id event score score2
<dbl> <dbl> <dbl> <dbl>
1 1 1 3 4
2 1 3 1 1
3 2 1 3 5
4 2 3 2 2
5 3 1 6 0
6 3 3 1 3
7 4 1 8 5
8 4 3 2 6
9 5 1 4 8
10 5 3 1 7

Use dynamically generated column names in dplyr

I have a data frame with multiple columns, the user provides a vector with the column names, and I want to count maximum amount of times an element appears
set.seed(42)
df <- tibble(
var1 = sample(c(1:3),10,replace=T),
var2 = sample(c(1:3),10,replace=T),
var3 = sample(c(1:3),10,replace=T)
)
select_vars <- c("var1", "var3")
df %>%
rowwise() %>%
mutate(consensus=max(table(unlist(c(var1,var3)))))
# A tibble: 10 x 4
# Rowwise:
var1 var2 var3 consensus
<int> <int> <int> <int>
1 1 1 1 2
2 1 1 3 1
3 1 2 1 2
4 1 2 1 2
5 2 2 2 2
6 2 3 3 1
7 2 3 2 2
8 1 1 1 2
9 3 1 2 1
10 3 3 2 1
This does exactly what I want, but when I try to use a vector of variables i cant get it to work
df %>%
rowwise() %>%
mutate(consensus=max(unlist(table(select_vars)) )))

You can wrap it in c(!!! syms()) to get it working, and you don't need the unlist apparently. But honestly, I'm not sure what you are trying to do, and why table is needed here. Do you just want to check if var2 and var3 are the same value and if then 2 and if not then 1?
library(dplyr)
df <- tibble(
var1 = sample(c(1:3),10,replace=T),
var2 = sample(c(1:3),10,replace=T),
var3 = sample(c(1:3),10,replace=T)
)
select_vars <- c("var2", "var3")
df %>%
rowwise() %>%
mutate(consensus=max(table(c(!!!syms(select_vars)))))
#> # A tibble: 10 x 4
#> # Rowwise:
#> var1 var2 var3 consensus
#> <int> <int> <int> <int>
#> 1 2 3 2 1
#> 2 3 1 3 1
#> 3 3 1 1 2
#> 4 3 3 3 2
#> 5 1 1 2 1
#> 6 2 1 3 1
#> 7 3 2 3 1
#> 8 1 2 3 1
#> 9 2 1 2 1
#> 10 2 1 1 2
Created on 2021-07-22 by the reprex package (v0.3.0)

In the OP's code, we need select
library(dplyr)
df %>%
rowwise() %>%
mutate(consensus=max(table(unlist(select(cur_data(), select_vars))) ))
-output
# A tibble: 10 x 4
# Rowwise:
var1 var2 var3 consensus
<int> <int> <int> <int>
1 1 1 1 2
2 1 1 3 1
3 1 2 1 2
4 1 2 1 2
5 2 2 2 2
6 2 3 3 1
7 2 3 2 2
8 1 1 1 2
9 3 1 2 1
10 3 3 2 1
Or just subset from cur_data() which would only return the data keeping the group attributes
df %>%
rowwise %>%
mutate(consensus = max(table(unlist(cur_data()[select_vars]))))
# A tibble: 10 x 4
# Rowwise:
var1 var2 var3 consensus
<int> <int> <int> <int>
1 1 1 1 2
2 1 1 3 1
3 1 2 1 2
4 1 2 1 2
5 2 2 2 2
6 2 3 3 1
7 2 3 2 2
8 1 1 1 2
9 3 1 2 1
10 3 3 2 1
Or using pmap
library(purrr)
df %>%
mutate(consensus = pmap_dbl(cur_data()[select_vars], ~ max(table(c(...)))))
# A tibble: 10 x 4
var1 var2 var3 consensus
<int> <int> <int> <dbl>
1 1 1 1 2
2 1 1 3 1
3 1 2 1 2
4 1 2 1 2
5 2 2 2 2
6 2 3 3 1
7 2 3 2 2
8 1 1 1 2
9 3 1 2 1
10 3 3 2 1
As these are rowwise operations, can get some efficiency if we use collapse functions
library(collapse)
tfm(df, consensus = dapply(slt(df, select_vars), MARGIN = 1,
FUN = function(x) fmax(tabulate(x))))
# A tibble: 10 x 4
var1 var2 var3 consensus
* <int> <int> <int> <int>
1 1 1 1 2
2 1 1 3 1
3 1 2 1 2
4 1 2 1 2
5 2 2 2 2
6 2 3 3 1
7 2 3 2 2
8 1 1 1 2
9 3 1 2 1
10 3 3 2 1
Benchmarks
As noted above, collapse is faster (run on a slightly bigger dataset)
df1 <- df[rep(seq_len(nrow(df)), 1e5), ]
system.time({
tfm(df1, consensus = dapply(slt(df1, select_vars), MARGIN = 1,
FUN = function(x) fmax(tabulate(x))))
})
#user system elapsed
# 5.257 0.123 5.323
system.time({
df1 %>%
mutate(consensus = pmap_dbl(cur_data()[select_vars], ~ max(table(c(...)))))
})
#user system elapsed
# 54.813 0.517 55.246
The rowwise operation is taking too much time, so stopped the execution
df1 %>%
rowwise() %>%
mutate(consensus=max(table(unlist(select(cur_data(), select_vars))) ))
})
Timing stopped at: 575.5 3.342 581.3

What you need is to use the verb all_of
df %>%
rowwise() %>%
mutate(consensus=max(table(unlist(all_of(select_vars)))))
# A tibble: 10 x 4
# Rowwise:
var1 var2 var3 consensus
<int> <int> <int> <int>
1 2 3 3 1
2 2 2 2 1
3 1 2 2 1
4 2 3 3 1
5 1 2 1 1
6 2 1 2 1
7 2 2 2 1
8 3 1 2 1
9 2 1 3 1
10 3 2 1 1

is there a way in R to fill missing groups absent of observations?

Say I have something like:
df<-data.frame(group=c(1, 1,1, 2,2,2,3,3,3,4,4, 1, 1,1),
group2=c(1,2,3,1,2,3,1,2,3,1,3, 1,2,3))
group group2
1 1 1
2 1 2
3 1 3
4 2 1
5 2 2
6 2 3
7 3 1
8 3 2
9 3 3
10 4 1
11 4 3
12 1 1
13 1 2
14 1 3
My goal is to count the number of unique instances for group= something and group2= something. Like so:
df1<-df%>%group_by(group, group2)%>% mutate(want=n())%>%distinct(group, group2, .keep_all=TRUE)
group group2 want
<dbl> <dbl> <int>
1 1 1 2
2 1 2 2
3 1 3 2
4 2 1 1
5 2 2 1
6 2 3 1
7 3 1 1
8 3 2 1
9 3 3 1
10 4 1 1
11 4 3 1
however, notice that group=4, group2=2 was not in my dataset to begin with. Is there some sort of autofill function where I can fill these non-observations with a zero to get below easily?:
group group2 want
<dbl> <dbl> <int>
1 1 1 2
2 1 2 2
3 1 3 2
4 2 1 1
5 2 2 1
6 2 3 1
7 3 1 1
8 3 2 1
9 3 3 1
10 4 1 1
11 4 2 0
12 4 3 1

After getting the count, we can expand with complete to fill the missing combinations with 0
library(dplyr)
library(tidyr)
df %>%
count(group, group2) %>%
complete(group, group2, fill = list(n = 0))
# A tibble: 12 x 3
# group group2 n
# <dbl> <dbl> <dbl>
# 1 1 1 2
# 2 1 2 2
# 3 1 3 2
# 4 2 1 1
# 5 2 2 1
# 6 2 3 1
# 7 3 1 1
# 8 3 2 1
# 9 3 3 1
#10 4 1 1
#11 4 2 0
#12 4 3 1
Or if we do the group_by, instead of mutate and then do the distinct, directly use the summarise
df %>%
group_by(group, group2) %>%
summarise(n = n()) %>%
ungroup %>%
complete(group, group2, fill = list(n = 0))

Here is a data.table approach solution to this problem:
library(data.table)
setDT(df)[CJ(group, group2, unique = TRUE),
c(.SD, .(want = .N)), .EACHI,
on = c("group", "group2")]
# group group2 want
# 1 1 2
# 1 2 2
# 1 3 2
# 2 1 1
# 2 2 1
# 2 3 1
# 3 1 1
# 3 2 1
# 3 3 1
# 4 1 1
# 4 2 0
# 4 3 1

Resources