I have a question on how to use eval(parse(text=...)) in dbplyr SQL translation.
The following code works exactly what I want with dplyr using eval(parse(text=eval_text))
selected_col <- c("wt", "drat")
text <- paste(selected_col, ">3")
implode <- function(..., sep='|') {
paste(..., collapse=sep)
}
eval_text <- implode(text)
mtcars %>% dplyr::filter(eval(parse(text=eval_text)))
But when I put it into the database it returns an error message. I am looking for any solution that allows me to dynamically set the column names and filter with the or operator.
db <- tbl(con, "mtcars") %>%
dplyr::filter(eval(parse(eval_text)))
db <- collect(db)
Thanks!
Right approach, but dbplyr tends to work better with something that can receive the !! operator ('bang-bang' operator). At one point dplyr had *_ versions of functions (e.g. filter_) that accepted text inputs. This is now done using NSE (non-standard evaluation).
A couple of references: shiptech and r-bloggers (sorry couldn't find the official dplyr reference).
For your purposes you should find the following works:
library(rlang)
df %>% dplyr::filter(!!parse_expr(eval_text))
Full working:
library(dplyr)
library(dbplyr)
library(rlang)
data(mtcars)
df = tbl_lazy(mtcars, con = simulate_mssql()) # simulated database connection
implode <- function(..., sep='|') { paste(..., collapse=sep) }
selected_col <- c("wt", "drat")
text <- paste(selected_col, ">3")
eval_text <- implode(text)
df %>% dplyr::filter(eval(parse(eval_text))) # returns clearly wrong SQL
df %>% dplyr::filter(!!parse_expr(eval_text)) # returns valid & correct SQL
df %>% dplyr::filter(!!!parse_exprs(text)) # passes filters as a list --> AND (instead of OR)
Related
I want to write a helper function that summarizes the percentage change for column A, B and C in one shot. I want to pass a string to the "mutate" argument of dplyr with the help of rlang. Unfortunately, I get an error saying that I have an unexpected ",". Could you please take a look? Thanks in advance!
library(rlang) #read text inputs and return vars
library(dplyr)
set.seed(10)
dat <- data.frame(A=rnorm(10,0,1),
B=rnorm(10,0,1),
C=rnorm(10,0,1),
D=2001:2010)
calc_perct_chg <- function(input_data,
target_Var_list,
year_Var_name){
#create new variable names
mutate_varNames <- paste0(target_Var_list,rep("_pct_chg = ",length(target_Var_list)))
#generate text for formula
mutate_formula <- lapply(target_Var_list,function(x){output <- paste0("(",x,"-lag(",x,"))/lag(",x,")");return(output)})
mutate_formula <- unlist(mutate_formula) #convert list to a vector
#generate arguments for mutate
mutate_args <<- paste0(mutate_varNames,collapse=",",mutate_formula)
#data manipulation
output <- input_data %>%
arrange(!!parse_quo(year_Var_name,env=caller_env())) %>%
mutate(!!parse_quo(mutate_args,env=caller_env()))
#output data frame
return(output)
}
# error: unexpected ','
calc_perct_chg(input_data =dat,
target_Var_list=list("A","B","C"),
year_Var_name="D")
I don't think it's a good idea to evaluate string as code, also I think you are over-complicating it. Using across this should be easier.
library(dplyr)
calc_perct_chg <- function(input_data,
target_Var_list,
year_Var_name){
input_data %>%
arrange(across(all_of(year_Var_name))) %>%
mutate(across(all_of(target_Var_list), ~(.x - lag(.x))/lag(.x)))
}
calc_perct_chg(input_data = dat,
target_Var_list = c("A","B","C"),
year_Var_name = "D")
I would like to create a function that updates a data frame from a different environment. Specifically, I would like to update the labels of a data frame using the Hmisc::label() function.
assign_label <- function(df, col) {
col <- rlang::as_name(rlang::ensym(col))
Hmisc::label(df[,col]) <- fetch_label(col)
}
fetch_label <- function(col) {
val <- c("mpg" = "MPG",
"hp" = "HP")
unname(val[col])
}
The following code executes without issue: assign_label(mtcars, hp)
However, it does not actually alter the data frame in the calling environment. I just can't figure out how to make it do what I imagine.
Ideally, I would like to be able to pipe a dataframe to this function as such:
mtcars %>% assign_label(mpg)
1) Return modified object Modifying objects in place is discouraged in R. The usual way to do this is to return the data frame and then assign it to a new name or back to the original name clobbering or shadowing it.
assign_label <- function(df, col) {
col <- deparse(substitute(col))
Hmisc::label(df[[col]]) <- fetch_label(col)
df
}
mtcars_labelled <- mtcars %>% assign_label(mpg)
2) magrittr Despite what we have said above there are some facilties for modifying in place in R and in some R packages. The magrittr package provides a syntax for overwriting or shadowing the input. Using the definition in (1) we can write:
library(mtcars)
mtcars %<>% assign_label(mpg)
If mtcars were in the global environment it would ovewrite it with the new value but in this case mtcars is in datasets so a new mtcars is written to the caller and the original in datasets is unchanged.
3) replacement function Although not widely used, R does provide replacement functions which are defined and used like this. This does overwite or shadow the input.
`assign_label<-` <- function(df, value) {
Hmisc::label(df[[value]]) <- fetch_label(value)
df
}
assign_label(mtcars) <- "mpg"
Note
As an aside, if the aim is for an interface that is consistent with tidyverse then use tidyselect to retrieve the column name(s) so that examples like the following work:
assign_labels <- function(df, col) {
nms <- names(select(df, {{col}}))
for(nm in nms) Hmisc::label(df[[nm]]) <- fetch_label(nm)
df
}
mtcars_labelled <- mtcars %>% assign_labels(starts_with("mp"))
str(mtcars_labelled)
mtcars_labelled <- mtcars %>% assign_labels(mpg|hp)
str(mtcars_labelled)
In regards to the comments about not modifying outside of the scope of a function, I created two functions that assign new dataframes with labels.
fetch_label <- function(col) {
val <- c("mpg" = "MPG",
"hp" = "HP")
unname(val[col])
}
assign_label <- function(df, col) {
col <- rlang::as_name(rlang::ensym(col))
Hmisc::label(df[[col]]) <- fetch_label(col)
return(df)
}
assign_labels <- function(df) {
purrr::iwalk(df, function(.x, .y) {
lab <- fetch_label(.y)
Hmisc::label(df[[col]]) <<- lab
})
return(df)
}
mtcars <- mtcars %>% assign_label(hp)
mtcars <- mtcars %>% assign_labels()
I have four functions, clean, clean2, cleanFun, and trim. Currently I apply the functions to one column, like so.
library(tidyverse)
library(data.table)
py17$CE.Finding.Description <- clean(py17$CE.Finding.Description)
py17$CE.Finding.Description <- clean2(py17$CE.Finding.Description)
py17$CE.Finding.Description <- cleanFun(py17$CE.Finding.Description)
py17$CE.Finding.Description <- trim(py17$CE.Finding.Description)
This process does the trick but I have to copy and paste this multiple times, and I'd eventually like to expand this to multiple columns.
For now, I'd like to save time and add an apply function but I'm not sure how to create that apply function. I've tried creating this.
maxclean <- function(cleaner) {
c(clean(cleaner), clean2(cleaner), cleanFun(cleaner), trim(cleaner))
}
py17$CE.Finding.Description <- sapply(py17$CE.Finding.Description, maxclean)
After trying this I just get
Error in `$<-.data.frame`(`*tmp*`, CE.Finding.Description, value = c(NA, :
replacement has 4 rows, data has 4318
I do not get any errors doing this the long way. Where am I going wrong on this?
Your maxclean function should take the same arguments as the separate functions. In your case - a vector. And then call each function in a row. Like this:
maxclean <- function(x) {
x <- clean(x)
x <- clean2(x)
x <- cleanFun(x)
x <- trim(x)
return(x)
}
Apparently, the OP has created a cleaning pipeline where the output of one step is fed into the next step and the final result of the pipeline overwrites the original input.
The magrittr package has the freduce() function which applies one function after the other in the described way. Thus,
py17$CE.Finding.Description <- clean(py17$CE.Finding.Description)
py17$CE.Finding.Description <- clean2(py17$CE.Finding.Description)
py17$CE.Finding.Description <- cleanFun(py17$CE.Finding.Description)
py17$CE.Finding.Description <- trim(py17$CE.Finding.Description)
can be written as:
library(magrittr)
fcts <- list(clean, clean2, cleanFun, trim)
py17$CE.Finding.Description %<>% freduce(fcts)
which is a shortcut for
py17$CE.Finding.Description <- py17$CE.Finding.Description %>%
clean() %>%
clean2() %>%
cleanFun() %>%
trim()
Here, %>% is the magrittr forward-pipe operator and %<>% is the magrittr compound assignment pipe-operator which updates the left-hand side object with the resulting value.
Reproducible example
Using the mtcars dataset:
data(mtcars)
mycars <- mtcars
mycars$mpg %<>%
{. - mean(.)} %>%
abs() %>%
sqrt()
mycars
or
mycars <- mtcars
mycars$mpg %<>% freduce(list(function(.) {. - mean(.)}, abs, sqrt))
mycars
Applying on multiple columns
The OP has mentioned that he eventually like to expand this to multiple columns
This can be achieved by, e.g.,
mycars <- mtcars
fcts <- list(function(.) {. - mean(.)}, abs, sqrt)
mycars$mpg %<>% freduce(fcts)
mycars$disp %<>% freduce(fcts)
mycars
I'm new working with spark. I would like to multiply a large number of columns of a spark dataframe by values in a vector. So far with mtcars I used a for loop and mutate_at as follows:
library(dplyr)
library(rlang)
library(sparklyr)
sc1 <- spark_connect(master = "local")
mtcars_sp = sdf_copy_to(sc1, mtcars, overwrite = TRUE)
mtcars_cols = colnames(mtcars_sp)
mtc_factors = 0:10 / 10
# mutate 1 col at a time
for (i in 1:length(mtcars_cols)) {
# set equation and print - use sym() convert a string
mtcars_eq = quo( UQ(sym(mtcars_cols[i])) * mtc_factors[i])
# mutate formula - LHS resolves to a string, RHS a quosure
mtcars_sp = mtcars_sp %>%
mutate(!!mtcars_cols[i] := !!mtcars_eq )
}
dbplyr::sql_render(mtcars_sp)
mtcars_sp
This works ok with mtcars. However, it results in nested SQL queries being sent to spark, as shown by the sql_render, and breaks down with many columns. Can dplyr be used to instead send a single SQL query in this case?
BTW, I'd rather not transpose the data as it would be too expensive. Any help would be much appreciated!
In general you can use great answer by Artem Sokolov
library(glue)
mtcars_sp %>%
mutate(!!! setNames(glue("{mtcars_cols} * {mtc_factors}"), mtcars_cols) %>%
lapply(parse_quosure))
However if this is input for MLlib algorithms then ft_vector_assembler combined with ft_elementwise_product might be a better fit:
scaled <- mtcars_sp %>%
ft_vector_assembler(mtcars_cols, "features") %>%
ft_elementwise_product("features", "features_scaled", mtc_factors)
The result can be separated (I wouldn't recommend that if you're going with MLlib) into individual columns with sdf_separate_column:
scaled %>%
select(features_scaled) %>%
sdf_separate_column("features_scaled", mtcars_cols)
I am doing a filter() using %in% but the way dplyr translates the query seems incorrect. In fact, the %in% operator works fine with more than one value, but it doesn't when only a single element is present.
In my original scenario the filtering values are dynamic, thus I would like to have a function that works in both cases.
my_db <- src_mysql(dbname = "dplyr",
host = "dplyr.csrrinzqubik.us-east-1.rds.amazonaws.com",
port = 3306,
user = "dplyr",
password = "dplyr")
tbl(my_db, "dplyr") %>% filter(carrier %in% c("UA","AA")) #works
tbl(my_db, "dplyr") %>% filter(carrier %in% c("UA")) #doesn't work
My question is a duplicate of multiple selectInput values create unexpected dplyr (postgres) behavior. Seems like this issue is well-known too
I can't offer any insights into why your code fails. But until someone can provide a better solution, here is a simple work-around that provides "a function that works in both cases".
my.carriers <- c("UA","AA")
my.carriers <- c("UA")
if (length(my.carriers)>1) {
tbl(my_db, "dplyr") %>% filter(carrier %in% my.carriers)
} else {
tbl(my_db, "dplyr") %>% filter(carrier == my.carriers)
}
Putting together some of the suggestions, the best approach for my scenario it's probably gonna be the one below. The reason why I don't like nesting the filter() in a if statement, is that I have multiple filter from menu items of a shiny app. Thus, manipulating the variable at the source saves me a lot of typing.
a <- c("UA")
b <- if(length(a)>1) a else c(a,"")
tbl(my_db, "dplyr") %>%
filter(carrier %in% b)
Or
a <- c("UA")
varToFilterFor <- rep(a ,2)
tbl(my_db, "dplyr") %>%
filter(carrier %in% varToFilterFor)