I am trying to find the mean of the variable disp in mtcars dataset after nesting it by cyl. I am able to get the result after nest_by but not with group_nest. Please explain what the rowwise is doing it differently here.
require(tidyverse)
# working
mtcars %>% nest_by(cyl) %>% mutate(avg = mean(data$disp))
#> # A tibble: 3 × 3
#> # Rowwise: cyl
#> cyl data avg
#> <dbl> <list<tibble[,10]>> <dbl>
#> 1 4 [11 × 10] 105.
#> 2 6 [7 × 10] 183.
#> 3 8 [14 × 10] 353.
# not working
mtcars %>% group_nest(cyl) %>%
mutate(avg = mean(data$disp))
#> Error in `mutate()`:
#> ! Problem while computing `avg = mean(data$disp)`.
#> Caused by error:
#> ! Corrupt x: no names
#> Backtrace:
#> ▆
#> 1. ├─mtcars %>% group_nest(cyl) %>% mutate(avg = mean(data$disp))
#> 2. ├─dplyr::mutate(., avg = mean(data$disp))
#> 3. ├─dplyr:::mutate.data.frame(., avg = mean(data$disp))
#> 4. │ └─dplyr:::mutate_cols(.data, dplyr_quosures(...), caller_env = caller_env())
#> 5. │ ├─base::withCallingHandlers(...)
#> 6. │ └─mask$eval_all_mutate(quo)
#> 7. ├─base::mean(data$disp)
#> 8. ├─data$disp
#> 9. ├─vctrs:::`$.vctrs_list_of`(data, disp)
#> 10. └─base::.handleSimpleError(`<fn>`, "Corrupt x: no names", base::quote(NULL))
#> 11. └─dplyr (local) h(simpleError(msg, call))
#> 12. └─rlang::abort(...)
Created on 2022-10-26 with reprex v2.0.2
rowwise changes the behavior of subsequent verbs, namely instead of operating on an entire column they will now operate only on values in a given row.
This works because the data in mutate refers to a single dataframe (due to rowwise provided by nest_by)
library(dplyr)
library(purrr)
mtcars %>% nest_by(cyl) %>% mutate(avg = mean(data$disp))
#> # A tibble: 3 × 3
#> # Rowwise: cyl
#> cyl data avg
#> <dbl> <list<tibble[,10]>> <dbl>
#> 1 4 [11 × 10] 105.
#> 2 6 [7 × 10] 183.
#> 3 8 [14 × 10] 353.
This will not work because data refers to a list of dataframes, and disp is not a name in that list
mtcars %>% group_nest(cyl) %>% mutate(avg = mean(data$disp))
#> Error in `mutate()`:
#> ! Problem while computing `avg = mean(data$disp)`.
#> Caused by error:
#> ! Corrupt x: no names
#> Backtrace:
#> ▆
#> 1. ├─mtcars %>% group_nest(cyl) %>% mutate(avg = mean(data$disp))
#> 2. ├─dplyr::mutate(., avg = mean(data$disp))
#> 3. ├─dplyr:::mutate.data.frame(., avg = mean(data$disp))
#> 4. │ └─dplyr:::mutate_cols(.data, dplyr_quosures(...), caller_env = caller_env())
#> 5. │ ├─base::withCallingHandlers(...)
#> 6. │ └─mask$eval_all_mutate(quo)
#> 7. ├─base::mean(data$disp)
#> 8. ├─data$disp
#> 9. ├─vctrs:::`$.vctrs_list_of`(data, disp)
#> 10. └─base::.handleSimpleError(`<fn>`, "Corrupt x: no names", base::quote(NULL))
#> 11. └─dplyr (local) h(simpleError(msg, call))
#> 12. └─rlang::abort(...)
You may obtain an equivalent calculation by e.g. mapping over the list of dataframes, to apply a function to each dataframe in the list
mtcars %>% group_nest(cyl) %>% mutate(avg = map_dbl(data, ~ mean(.x$disp)))
#> # A tibble: 3 × 3
#> cyl data avg
#> <dbl> <list<tibble[,10]>> <dbl>
#> 1 4 [11 × 10] 105.
#> 2 6 [7 × 10] 183.
#> 3 8 [14 × 10] 353.
Created on 2022-10-26 with reprex v2.0.2
We could use map to loop over the list as there is no rowwise grouping with group_nest
library(dplyr)
library(purrr)
mtcars %>%
group_nest(cyl) %>%
mutate(avg = map_dbl(data, ~ mean(.x$disp)))
-output
# A tibble: 3 × 3
cyl data avg
<dbl> <list<tibble[,10]>> <dbl>
1 4 [11 × 10] 105.
2 6 [7 × 10] 183.
3 8 [14 × 10] 353.
According to ?group_nest
The primary use case for group_nest() is with already grouped data frames, typically a result of group_by().
where as with ?nest_by
nest_by() is closely related to group_by(). However, instead of storing the group structure in the metadata, it is made explicit in the data, giving each group key a single row along with a list-column of data frames that contain all the other data.
> library(pacman)
> p_load(tidyverse)
> # working
> mtcars %>% nest_by(cyl) %>% class()
[1] "rowwise_df" "tbl_df" "tbl" "data.frame"
> mtcars %>% nest_by(cyl) %>% mutate(avg = mean(data$disp))
# A tibble: 3 × 3
# Rowwise: cyl
cyl data avg
<dbl> <list<tibble[,10]>> <dbl>
1 4 [11 × 10] 105.
2 6 [7 × 10] 183.
3 8 [14 × 10] 353.
> # not working
> mtcars %>% group_nest(cyl) %>% class()
[1] "tbl_df" "tbl" "data.frame"
> mtcars %>% group_nest(cyl) %>% mutate(avg = mean(data$disp))
Error in `mutate()`:
! Problem while computing `avg = mean(data$disp)`.
Caused by error:
! Corrupt x: no names
Run `rlang::last_error()` to see where the error occurred.
The nest_by call yields a rowwise_df which is amenable to the next step in the pipe, whereas group_nest yields a plain old tbl_df, hence the difference
Related
I'm trying to create a function that will map across a nested tibble. This function needs to take a vector of parameters that will vary for each row.
When I call purrr:map2() on the nested data, purrr tries to loop over all values of the parameter vector and all rows in the dataset. What can I do to pass the entire vector as a single argument?
library(tidyverse)
myf <- function(x, params) {
print(params)
x %>%
mutate(new_mpg = mpg + rnorm(n(), params[1], params[2])) %>%
summarise(old = mean(mpg), new = mean(new_mpg)) %>%
as.list()
}
# Calling function with params defined is great!
myf(mtcars, params = c(5, 10))
#> [1] 5 10
#> $old
#> [1] 20.09062
#>
#> $new
#> [1] 25.62049
# Cannot work in purr as vector, tries to loop over param
mtcars %>%
group_by(cyl) %>% # from base R
nest() %>%
mutate(
newold = map2(data, c(5, 10), myf),
)
#> [1] 5
#> Warning in rnorm(n(), params[1], params[2]): NAs produced
#> [1] 10
#> Warning in rnorm(n(), params[1], params[2]): NAs produced
#> Error: Problem with `mutate()` column `newold`.
#> ℹ `newold = map2(data, c(5, 10), myf)`.
#> ℹ `newold` must be size 1, not 2.
#> ℹ The error occurred in group 1: cyl = 4.
# New function wrapper with hard-coded params
myf2 <- function(x){
myf(x, c(5, 10))
}
# works great! but not what I need
mtcars %>%
group_by(cyl) %>% # from base R
nest() %>%
mutate(
mean = 5,
sd = 10,
newold = map(data, myf2),
)
#> [1] 5 10
#> [1] 5 10
#> [1] 5 10
#> # A tibble: 3 × 5
#> # Groups: cyl [3]
#> cyl data mean sd newold
#> <dbl> <list> <dbl> <dbl> <list>
#> 1 6 <tibble [7 × 10]> 5 10 <named list [2]>
#> 2 4 <tibble [11 × 10]> 5 10 <named list [2]>
#> 3 8 <tibble [14 × 10]> 5 10 <named list [2]>
Created on 2021-11-29 by the reprex package (v2.0.0)
Skip the group_by() step and just use nest() - otherwise your data will remain grouped after nesting and need to be ungrouped. To get your function to work, just pass the parameters as a list.
library(tidyverse)
mtcars %>%
nest(data = -cyl) %>%
mutate(
newold = map2_df(data, list(c(5, 10)), myf)
) %>%
unpack(newold)
# A tibble: 3 x 4
cyl data old new
<dbl> <list> <dbl> <dbl>
1 6 <tibble [7 x 10]> 19.7 30.7
2 4 <tibble [11 x 10]> 26.7 31.1
3 8 <tibble [14 x 10]> 15.1 17.0
You don't need map2. I think what you need is map.
mtcars %>%
group_by(cyl) %>% # from base R
nest() %>%
mutate(
newold = map(data, myf, params = c(5, 10)),
)
# [1] 5 10
# [1] 5 10
# [1] 5 10
# # A tibble: 3 x 3
# # Groups: cyl [3]
# cyl data newold
# <dbl> <list> <list>
# 1 6 <tibble [7 x 10]> <named list [2]>
# 2 4 <tibble [11 x 10]> <named list [2]>
# 3 8 <tibble [14 x 10]> <named list [2]>
If you have multiple sets of params. You can ungroup your data frame, add a list column with your params, and use map2.
mtcars %>%
group_by(cyl) %>%
nest() %>%
ungroup() %>%
# Add different sets of params
mutate(Params = list(a = c(5, 10), b = c(6, 11), c = c(7, 12))) %>%
mutate(
newold = map2(data, Params, myf)
)
# [1] 5 10
# [1] 6 11
# [1] 7 12
# # A tibble: 3 x 4
# cyl data Params newold
# <dbl> <list> <named list> <list>
# 1 6 <tibble [7 x 10]> <dbl [2]> <named list [2]>
# 2 4 <tibble [11 x 10]> <dbl [2]> <named list [2]>
# 3 8 <tibble [14 x 10]> <dbl [2]> <named list [2]>
Grouped regression is running well model1 with "do". But recently, it is told that do is superseded and suggested to use "across" but no example is given in the help file. Model2 is given in "do" help, and it is running well without "map" or "across". I don't understand how the regression is looping over those groups without map. When I tried using map in model3, I am getting errors. Model4 is given in Hadley's book, R for data science using split and working well. How to tell map function to consider the list "data". Any suggestions?
library(purrr)
#> Warning: package 'purrr' was built under R version 3.6.3
library(tidyverse)
#> Warning: package 'tidyverse' was built under R version 3.6.3
#> Warning: package 'ggplot2' was built under R version 3.6.3
#> Warning: package 'tidyr' was built under R version 3.6.3
#> Warning: package 'dplyr' was built under R version 3.6.3
#> Warning: package 'stringr' was built under R version 3.6.3
#> Warning: package 'forcats' was built under R version 3.6.3
model1 = mtcars %>%
group_by(cyl) %>%
do(mod = lm(mpg ~ disp, data = .))
model1
#> # A tibble: 3 x 2
#> # Rowwise:
#> cyl mod
#> <dbl> <list>
#> 1 4 <lm>
#> 2 6 <lm>
#> 3 8 <lm>
## from "do" help file
model2 = mtcars %>%
nest_by(cyl) %>%
mutate(mod = list(lm(mpg ~ disp, data = data)))
model2
#> # A tibble: 3 x 3
#> # Rowwise: cyl
#> cyl data mod
#> <dbl> <list<tbl_df[,10]>> <list>
#> 1 4 [11 x 10] <lm>
#> 2 6 [7 x 10] <lm>
#> 3 8 [14 x 10] <lm>
## using map
model3 = mtcars %>% nest_by(cyl) %>%
mutate(fit = map(data, ~lm(mpg ~ disp, data = .)))
#> Error: Problem with `mutate()` input `fit`.
#> x numeric 'envir' arg not of length one
#> i Input `fit` is `map(data, ~lm(mpg ~ disp, data = .))`.
#> i The error occured in row 1.
##model4
model4 = mtcars %>%
split(.$cyl) %>%
map(~lm(mpg ~ disp, data = .))
model4
#> $`4`
#>
#> Call:
#> lm(formula = mpg ~ disp, data = .)
#>
#> Coefficients:
#> (Intercept) disp
#> 40.8720 -0.1351
#>
#>
#> $`6`
#>
#> Call:
#> lm(formula = mpg ~ disp, data = .)
#>
#> Coefficients:
#> (Intercept) disp
#> 19.081987 0.003605
#>
#>
#> $`8`
#>
#> Call:
#> lm(formula = mpg ~ disp, data = .)
#>
#> Coefficients:
#> (Intercept) disp
#> 22.03280 -0.01963
Created on 2020-08-02 by the reprex package (v0.3.0)
It could be an issue with rowwise attribute, we could ungroup
library(dplyr)
library(purrr)
mtcars %>%
nest_by(cyl) %>% # // creates the rowwise attribute
ungroup %>% # // remove the rowwise
mutate(fit = map(data, ~lm(mpg ~ disp, data = .)))
# A tibble: 3 x 3
# cyl data fit
# <dbl> <list<tbl_df[,10]>> <list>
#1 4 [11 × 10] <lm>
#2 6 [7 × 10] <lm>
#3 8 [14 × 10] <lm>
library(tidyverse)
mtcars %>% group_by(cyl) %>% is_grouped_df()
#> [1] TRUE
I can group a data frame by a variable and confirm if it is grouped using the is_grouped_df() function (shown above).
I can run the same analysis on the dplyr rowwise() function and it appears that rowwise() does not group data sets by row. I have a question and a reading of the help page (?rowwise) does not clearly answer the question for me.
Group input by rows
Description: rowwise() allows you to compute on a data frame a row-at-a-time. This is most useful when a vectorised function doesn't exist.
A row-wise tibble maintains its row-wise status until explicitly removed by group_by(), ungroup(), or as_tibble().
My question: After calling the rowwise() function do I need to call the ungroup() function later in my pipe to ungroup my data set? Or is this done by default? The following pipe suggests that a pipe containing rowwise() is not grouped:
mtcars %>% rowwise() %>% is_grouped_df()
#> [1] FALSE
This sentence is confusing me, "A row-wise tibble maintains its row-wise status until explicitly removed by... ungroup()...". Why would I need to ungroup() a tibble that is already ungrouped?
Interesting observation. This might be a bug of is_grouped_df unless it's somehow a feature that I don't know about. But I DO think it's important to ungroup considering the testing done below (see comments):
library(tidyverse)
mtcars %>% select(1:3) %>% rowwise() %>% head(2)
#> Source: local data frame [2 x 3]
#> Groups: <by row>
##### ^ THIS DOES HAVE A GROUP ####
#>
#> # A tibble: 2 x 3
#> mpg cyl disp
#> <dbl> <dbl> <dbl>
#> 1 21 6 160
#> 2 21 6 160
mtcars %>% select(1:3) %>% rowwise() %>% mutate(n()) %>% head(2)
#> Source: local data frame [2 x 4]
#> Groups: <by row>
#>
#> # A tibble: 2 x 4
#> mpg cyl disp `n()`
#> <dbl> <dbl> <dbl> <int>
#> 1 21 6 160 1
#> 2 21 6 160 1
mtcars %>% select(1:3) %>% mutate(n()) %>% head(2)
#> mpg cyl disp n()
#> 1 21 6 160 32
#> 2 21 6 160 32
##### ^ THIS IS EXPECTED AND THE n BEHAVES DIFFERENTLY WHEN THE ROWWISE() IS APPLIED ####
##### IF WE WANT TO RESTORE "NORMAL" BEHAVIOR, IT'S PROBABLY WISE TO UNGROUP IN ORDER TO LOSE THE ROWWISE OPERATIONS #####
mtcars %>% select(1:3) %>% rowwise() %>% ungroup %>% mutate(n()) %>% head(2)
#> # A tibble: 2 x 4
#> mpg cyl disp `n()`
#> <dbl> <dbl> <dbl> <int>
#> 1 21 6 160 32
#> 2 21 6 160 32
## ^ NORMAL AFTER UNGROUP
I've got a dataset to which I want to apply the following operations:
Nest data by a set of columns
Modify the 'data' list column to a wide format
Use both the 'data' list column and values in the grouping columns to create a new column in the 'data' list column, using the columns created under 2. as inputs as well.
1 & 2 work fine using purrr and dplyr, but in step 3 when I try to reference one of the columns created in step 2 it gives me an 'object not found' error. It works fine if I reference only columns in the 'data' list column that already existed before step 2. If I inspect the contents of the list column after step 2 everything referenced in step 3 is there so why is the newly created column not recognized?
Reprex
library(tidyverse)
mtcarsT <- mtcars %>% as_tibble() %>%
group_by(cyl, gear, vs) %>%
mutate(cp_flag = rep(c('C', 'P'), length.out = n())) %>%
# step 1, works fine.
nest() %>%
mutate(data = map(data, ~ .x %>%
group_by(mpg, disp, hp, drat, am, carb) %>%
# step 2, also works fine, generates new columns 'wt_C', 'wt_P', 'qsec_C', 'qsec_P' in 'data'
pivot_wider(names_from = cp_flag, values_from = wt:qsec) %>%
ungroup()))
# > mtcarsT[1,]
# A tibble: 1 x 4
# Groups: cyl, vs, gear [1]
# cyl vs gear data
# <dbl> <dbl> <dbl> <list>
# 1 6 0 4 <tibble [1 x 10]>
#
# > mtcarsT$data[[1]]
# A tibble: 1 x 10
# mpg disp hp drat am carb wt_C wt_P qsec_C qsec_P
# <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
# 1 21 160 110 3.9 1 4 2.62 2.88 16.5 17.0
# step 3A: this one works fine when only referencing columns in 'data' that already existed before step 2.
mtcarsT %>%
mutate(data = pmap(.l = list(a = data, b = vs, c = gear),
.f = function(a, b, c) a %>%
dplyr::mutate(vs_gear = carb - b + c)))
# > .Last.value$data[[1]]
# A tibble: 1 x 11
# mpg disp hp drat am carb wt_C wt_P qsec_C qsec_P vs_gear
# <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
# 1 21 160 110 3.9 1 4 2.62 2.88 16.5 17.0 8
# Step 3B: this is what I want to do, use the column 'wt_P' in the 'data' list column
# that was created in step 2 along with other columns 'vs' and 'gear' in the nested tibble,
# but it throws the error 'object wt_P not found'
mtcarsT %>%
mutate(data = pmap(.l = list(a = data, b = vs, c = gear),
.f = function(a, b, c) a %>%
dplyr::mutate(vs_gear = wt_P - b + c)))
# Error: object 'wt_P' not found
# Called from: mutate_impl(.data, dots, caller_env())
'''
I'm using R 3.6.2 x64 with tidyverse 1.3.0 inside RStudio 1.2.5033 on Windows 10.
The example works as expected but the issue is present in step 1 itself.
To show that the example works you can change wt_P to wt_C and it works.
library(tidyverse)
mtcarsT %>%
mutate(data = pmap(.l = list(a = data, b = vs, c = gear),
.f = function(a, b, c)
a %>% dplyr::mutate(vs_gear = wt_C - b + c)))
# cyl vs gear data
# <dbl> <dbl> <dbl> <list>
# 1 6 0 4 <tibble [1 × 11]>
# 2 4 1 4 <tibble [8 × 11]>
# 3 6 1 3 <tibble [2 × 11]>
#....
The issue in step 1 is when you are doing
mtcars %>% as_tibble() %>%
group_by(cyl, gear, vs) %>%
mutate(cp_flag = rep(c('C', 'P'), length.out = n()))
there are certain groups with only 1 observation who do not get P value at all.
mtcars %>% count(cyl, gear, vs)
# A tibble: 10 x 4
# cyl gear vs n
# <dbl> <dbl> <dbl> <int>
# 1 4 3 1 1
# 2 4 4 1 8
# 3 4 5 0 1
# 4 4 5 1 1
# 5 6 3 1 2
# 6 6 4 0 2
# 7 6 4 1 2
# 8 6 5 0 1
# 9 8 3 0 12
#10 8 5 0 2
Hence, wt_P is not calculated for them and it returns an error whereas wt_C does not. If you change order in rep from c('C', 'P') to c('P', 'C') then you will get error for wt_C and wt_P would work as expected.
To add the missing column, we can do :
mtcars %>%
group_by(cyl, gear, vs) %>%
mutate(cp_flag = rep(c('C', 'P'), length.out = n())) %>%
nest() %>%
mutate(data = map(data, ~{
temp <- .x %>%
group_by(mpg, disp, hp, drat, am, carb) %>%
pivot_wider(names_from = cp_flag, values_from = wt:qsec,
values_fill = list(wt = NA, qsec = NA)) %>%
ungroup()
temp[setdiff(cols, names(temp))] <- NA;temp
}))
# cyl vs gear data
# <dbl> <dbl> <dbl> <list>
# 1 6 0 4 <tibble [1 × 10]>
# 2 4 1 4 <tibble [8 × 10]>
# 3 6 1 3 <tibble [2 × 10]>
# 4 8 0 3 <tibble [12 × 10]>
# 5 6 1 4 <tibble [2 × 10]>
# 6 4 1 3 <tibble [1 × 10]>
# 7 4 0 5 <tibble [1 × 10]>
# 8 4 1 5 <tibble [1 × 10]>
# 9 8 0 5 <tibble [2 × 10]>
#10 6 0 5 <tibble [1 × 10]>
So all of them have same number of columns.
I am teaching myself the R tidyverse purr() package and am having trouble implementing map() on a column of nested data frames. Could someone explain what I'm missing?
Using the base R ChickWeight dataset as an example I can easily get the number of observations for each timepoint under diet #1 if I first filter for diet #1 like so:
library(tidyverse)
ChickWeight %>%
filter(Diet == 1) %>%
group_by(Time) %>%
summarise(counts = n_distinct(Chick))
This is great but I would like to do it for each diet at once and I thought nesting the data and iterating over it with map() would be a good approach.
This is what I did:
example <- ChickWeight %>%
nest(-Diet)
Implementing this map function then achieves what I'm aiming for:
map(example$data, ~ .x %>% group_by(Time) %>% summarise(counts = n_distinct(Chick)))
However when I try and implement this same command using a pipe to put it in another column of the original data frame it fails.
example %>%
mutate(counts = map(data, ~ .x %>% group_by(Time) %>% summarise(counts = n_distinct(Chick))))
Error in eval(substitute(expr), envir, enclos) :
variable 'Chick' not found
Why does this occur?
I also tried it on the data frame split into a list and it didn't work.
ChickWeight %>%
split(.$Diet) %>%
map(data, ~ .x %>% group_by(Time) %>% summarise(counts = n_distinct(Chick)))
Because you're using dplyr non-standard evaluation inside of dplyr NSE, it's getting confused about what environment to search for Chick. It's probably a bug, really, but it can be avoided with the development version's new .data pronoun, which specifies where to look:
library(tidyverse)
ChickWeight %>%
nest(-Diet) %>%
mutate(counts = map(data,
~.x %>% group_by(Time) %>%
summarise(counts = n_distinct(.data$Chick))))
#> # A tibble: 4 × 3
#> Diet data counts
#> <fctr> <list> <list>
#> 1 1 <tibble [220 × 3]> <tibble [12 × 2]>
#> 2 2 <tibble [120 × 3]> <tibble [12 × 2]>
#> 3 3 <tibble [120 × 3]> <tibble [12 × 2]>
#> 4 4 <tibble [118 × 3]> <tibble [12 × 2]>
To pipe it through a list, leave the first parameter of map blank to pass in the list over which to iterate:
ChickWeight %>%
split(.$Diet) %>%
map(~ .x %>% group_by(Time) %>% summarise(counts = n_distinct(Chick))) %>% .[[1]]
#> # A tibble: 12 × 2
#> Time counts
#> <dbl> <int>
#> 1 0 20
#> 2 2 20
#> 3 4 19
#> 4 6 19
#> 5 8 19
#> 6 10 19
#> 7 12 19
#> 8 14 18
#> 9 16 17
#> 10 18 17
#> 11 20 17
#> 12 21 16
A simpler option would be to just group by both columns:
ChickWeight %>% group_by(Diet, Time) %>% summarise(counts = n_distinct(Chick))
#> Source: local data frame [48 x 3]
#> Groups: Diet [?]
#>
#> Diet Time counts
#> <fctr> <dbl> <int>
#> 1 1 0 20
#> 2 1 2 20
#> 3 1 4 19
#> 4 1 6 19
#> 5 1 8 19
#> 6 1 10 19
#> 7 1 12 19
#> 8 1 14 18
#> 9 1 16 17
#> 10 1 18 17
#> # ... with 38 more rows