I am using dplyr package and want to create variable only if condition exists. The function below performs recoding and works fine. I want this line of code !!var := ifelse(is.na(as.numeric(!!var)), 0, !!var) run only if the variable has missing values. I also need to check whether the variable name is valid or not.
recoding <- function(df, var, interval) {
var <- enquo(var)
var2 <- paste0(as_label(var), '_group')
df <- df %>% mutate(
!!var := ifelse(is.na(as.numeric(!!var)), 0, !!var),
!!var2 := Hmisc::cut2(!!var, cuts = interval)
)
message(var2, " variable has been created")
return(df)
}
mtcars %>% recoding(mpg, c(0,20,50,Inf))
Answering here, as this will be too long for a comment.
I think I understand why you want to run the ifelse line only when there is NAs in the variable, to save time/resources. But it will be an overhead. You will be testing for NAs in the variable twice, first to check if there are any NAs, and after that to check which are NA to change them.
# lets check if the var has NAs (as per #akrun and #Gregor comments)
if (sum(is.na(!!var)) > 0) { # just now you just check all the var values to see if there is NAs
df <- df %>% mutate(
!!var := ifelse(is.na(as.numeric(!!var)), 0, !!var), # And here you are checking again
!!var2 := Hmisc::cut2(!!var, cuts = interval)
)
}
The code as it is is perfect. You only check it once and if there are any NA the change will be done. If not, the variable will be the same. I know it's counterintuitive, but trying to perform the operation only in NAs-containing variables will not save you time or resources, in fact it will "double" them.
About the var name checking, try
# checking if var is valid
if (!as_label(var) %in% names(df)) {
stop(as_label(var), ' is not present in df')
}
just before the the data frame modification step. That way the function will throw and error if the variable var is nor in the data frame.
Related
I'm comparing the created date value at each row with the minimum removed date value for that user until row-1 index at which the loop currently stands
I tried to create a function which would be passed with 1:i-1 range of data and the current loop's user id.
minif<- function(data,user){
data<-as.data.frame(data)
user<-as.numeric(unlist(user))
aggregate(data$removed_at,list(user=data$user_id),min)
}
for(i in 2:nrow(mydf1)){
if (mydf1$created_at[i]<=minif(mydf1[1:i-1,],as.numeric(mydf1$user_id[i]))){
mydf1$test[i]=1
}
else
mydf1$test[i]=0
}
The expected output should have been 1 for the rows that meet the criteria, however it gives me an error:
(list) object cannot be coerced to type 'double'
An option would be
library(dplyr)
mydf %>%
group_by(user_id) %>%
mutate(test = as.integer(created_at <= cummin(lag(removed_at,
default = first(removed_at)))))
I have a Spark DataFrame with an ID column called "userid" that I am manipulating using sparklyr. Each userid can have anywhere from one row of data up to hundreds of rows of data. I am applying a function to each userid group which condenses the number of rows it contains based on certain event criteria. Something like
sdf %>%
group_by(userid) %>%
... %>% # using dplyr::filter and dplyr::mutate
ungroup()
I would like to wrap this function in an error handler such as purrr::possibly so that computation will not be interrupted if an error occurs in a single group.
So far, I have had the most success using the replyr package. Specifically, replyr::gapply "partitions from by values in grouping column, applies a generic transform to each group and then binds the groups back together." There are two methods for partitioning the data: "group_by" and "extract". The authors only recommend using "extract" in the case that the number of groups is 100 or less, but the "group_by" method does not work as I'd expect:
library(sparklyr)
library(dplyr)
library(replyr) # replyr::gapply
library(purrr) # purrr::possibly
sc <- spark_connect(master = "local")
# Create a test data frame to use gapply on.
test_spark <- tibble(
userid = c(1, 1, 2, 2, 3, 3),
occurred_at = seq(1, 6)
) %>%
sdf_copy_to(sc, ., "test_spark")
# Create a data frame that purrr::possibly should return in case of error.
default_spark <- tibble(userid = -1, max = -1, min = -1) %>%
sdf_copy_to(sc, ., "default_spark")
#####################################################
# Method 1: gapply with partitionMethod = "group_by".
#####################################################
# Create a function which may throw an error. The group column, userid, is not
# included since gapply( , partitionMethod = "group_by") creates it.
# - A print statement is included to show that when gapply uses "group_by", the
# function is only called once.
fun_for_groups <- function(sdf) {
temp <- sample(c(1,2), 1)
print(temp)
if (temp == 2) {
log("a")
} else {
sdf %>%
summarise(max = max(occurred_at),
min = min(occurred_at))
}
}
# Wrap the risk function to try and handle the error gracefully.
safe_for_groups <- purrr::possibly(fun_for_groups, otherwise = default_spark)
# Apply the safe function to each userid using gapply and "group_by".
# - The result is either a) only the default_spark data frame.
# b) the result expected if no error occurs in fun_for_groups.
# I would expect the answer to have a mixture of default_spark rows and correct rows.
replyr::gapply(
test_spark,
gcolumn = "userid",
f = safe_for_groups,
partitionMethod = "group_by"
)
#####################################################
# Method 2: gapply with partitionMethod = "extract".
#####################################################
# Create a function which may throw an error. The group column, userid, is
# included since gapply( , partiionMethod = "extract") doesn't create it.
# - Include a print statement to show that when gapply uses partitionMethod
# "split", the function is called for each userid.
fun_for_extract <- function(df) {
temp <- sample(c(1,2), 1)
print(temp)
if (temp == 2) {
log("a")
} else {
df %>%
summarise(max = max(occurred_at),
min = min(occurred_at),
userid = min(userid))
}
}
safe_for_extract <- purrr::possibly(fun_for_extract, otherwise = default_spark)
# Apply that function to each userid using gapply and "split".
# - The result dataframe has a mixture of "otherwise" rows and correct rows.
replyr::gapply(
test_spark,
gcolumn = "userid",
f = safe_for_extract,
partitionMethod = "extract"
)
How bad of an idea is it to use gapply when the grouping column has millions of values? Is there an alternative to the error handling strategies presented above?
replyr::gapply() is just a thin wrapper on top of dplyr (and in this case sparklyr).
For the grouped mode- the result can only be correct if no group errors out, as the calculation is issued all at once. This is the most efficient mode, but can't really achieve any sort of error handling.
For the extract mode- it might be possible to add error handling, but the current code does not have it.
As the replyr author I would actually suggest looking into sparklyr's spark_apply() method. replyr's gapply was designed when spark_apply() was not available in sparklyr (and also when binding lists of data was also not available in sparklyr).
Also replyr is mostly in "maintenance mode" (patching issues for clients who used it in larger projects), and probably not a good choice for new projects.
I'm trying to understand how SE works in dplyr so I can use variables as inputs to these functions. I'm having some trouble with understanding how this works across the different functions and when I should be doing what. It would be really good to understand the logic behind this.
Here are some examples:
library(dplyr)
library(lazyeval)
a <- c("x", "y", "z")
b <- c(1,2,3)
c <- c(7,8,9)
df <- data.frame(a, b, c)
The following is exactly why i'd use SE and the *_ variant of a function. I want to change the name of what's being mutated based on another variable.
#Normal mutate - copies b into a column called new
mutate(df, new = b)
#Mutate using a variable column names. Use mutate_ and the unqouted variable name. Doesn't use the name "new", but use the string "col.new"
col.name <- "new"
mutate_(df, col.name = "b")
#Do I need to use interp? Doesn't work
expr <- interp(~(val = b), val = col.name)
mutate_(df, expr)
Now I want to filter in the same way. Not sure why my first attempt didn't work.
#Apply the same logic to filter_. the following doesn't return a result
val.to.filter <- "z"
filter_(df, "a" == val.to.filter)
#Do I need to use interp? Works. What's the difference compared to the above?
expr <- interp(~(a == val), val = val.to.filter)
filter_(df, expr)
Now I try to select_. Works as expected
#Apply the same logic to select_, an unqouted variable name works fine
col.to.select <- "b"
select_(df, col.to.select)
Now I move on to rename_. Knowing what worked for mutate and knowing that I had to use interp for filter, I try the following
#Now let's try to rename. Qouted constant, unqouted variable. Doesn't work
new.name <- "NEW"
rename_(df, "a" = new.name)
#Do I need an eval here? It worked for the filter so it's worth a try. Doesn't work 'Error: All arguments to rename must be named.'
expr <- interp(~(a == val), val = new.name)
rename_(df, expr)
Any tips on best practice when it comes to using variable names across the dplyr functions and when interp is required would be great.
The differences here are not related to which dplyr verb you are using. They are related to where you are trying to use the variable. You are mixing whether the variable is used as a function argument or not, and whether it should be interpreted as a name or as a character string.
Scenario 1:
You want to use your variable as an argument name. Such as in your mutate example.
mutate(df, new = b)
Here new is the name of a function argument, it is left of a =. The only way to do this is to use the .dots argument. Like
col.name <- 'new'
mutate_(df, .dots = setNames(list(~b), col.name))
Running just setNames(list(~b), col.name) shows you how we have an expression (~b), which is going right of the =, and the name is going left of the =.
Scenario 2:
You want to give only a variable as a function argument. This is the simplest case. Let's again use mutate(df, new = b), but in this case we want b to be variable. We could use:
v <- 'b'
mutate_(df, .dots = setNames(list(v), 'new'))
Or simply:
mutate_(df, new = b)
Scenario 3
You want to do some combinations of variable and fixed things. That is, your expression should only be partly variable. For this we use interp. For example, what if we would like to do something like:
mutate(df, new = b + 1)
But being able to change b?
v <- 'b'
mutate_(df, new = interp(~var + 1, var = as.name(v)))
Note that we as.name to make sure that we insert b into the expression, not 'b'.
When writing functions it is important to check for the type of arguments. For example, take the following (not necessarily useful) function which is performing subsetting:
data_subset = function(data, date_col) {
if (!TRUE %in% (is.character(date_col) | is.expression(date_col))){
stop("Input variable date is of wrong format")
}
if (is.character(date_col)) {
x <- match(date_col, names(data))
} else x <- match(deparse(substitute(date_col)), names(data))
sub <- data[,x]
}
I would like to allow the user to provide the column which should be extracted as character or expression (e.g. a column called "date" vs. just date). At the beginning I would like to check that the input for date_col is really either a character value or an expression. However, 'is.expression' does not work:
Error in match(x, table, nomatch = 0L) : object '...' not found
Since deparse(substitute)) works if one provides expressions I thought 'is.expression' has to work as well.
What is wrong here, can anyone give me a hint?
I think you are not looking for is.expression but for is.name.
The tricky part is to get the type of date_col and to check if it is of type character only if it is not of type name. If you called is.character when it's a name, then it would get evaluated, typically resulting in an error because the object is not defined.
To do this, short circuit evaluation can be used: In
if(!(is.name(substitute(date_col)) || is.character(date_col)))
is.character is only called if is.name returns FALSE.
Your function boils down to:
data_subset = function(data, date_col) {
if(!(is.name(substitute(date_col)) || is.character(date_col))) {
stop("Input variable date is of wrong format")
}
date_col2 <- as.character(substitute(date_col))
return(data[, date_col2])
}
Of course, you could use if(is.name(…)) to convert only to character when date_col is a name.
This works:
testDF <- data.frame(col1 = rnorm(10), col2 = rnorm(10, mean = 10), col3 = rnorm(10, mean = 50), rnorm(10, mean = 100))
data_subset(testDF, "col1") # ok
data_subset(testDF, col1) # ok
data_subset(testDF, 1) # Error in data_subset(testDF, 1) : Input variable date is of wrong format
However, I don't think you should do this. Consider the following example:
var <- "col1"
data_subset(testDF, var) # Error in `[.data.frame`(data, , date_col2) : undefined columns selected
col1 <- "col2"
data_subset(testDF, col1) # Gives content of column 1, not column 2.
Though this "works as designed", it is confusing because unless carefully reading your function's documentation one would expect to get col1 in the first case and col2 in the second case.
Abusing a famous quote:
Some people, when confronted with a problem, think “I know, I'll use non-standard evaluation.” Now they have two problems.
Hadley Wickham in Non-standard evaluation:
Non-standard evaluation allows you to write functions that are extremely powerful. However, they are harder to understand and to program with. As well as always providing an escape hatch, carefully consider both the costs and benefits of NSE before using it in a new domain.
Unless you expect large benefits from allowing to skip the quotes around the name of the column, don't do it.
I'm working with dplyr and created code to compute new data that is plotted with ggplot.
I want to create a function with this code. It should take a name of a column of the data frame that is manipulated by dplyr. However, trying to work with columnnames does not work. Please consider the minimal example below:
df <- data.frame(A = seq(-5, 5, 1), B = seq(0,10,1))
library(dplyr)
foo <- function (x) {
df %>%
filter(x < 1)
}
foo(B)
Error in filter_impl(.data, dots(...), environment()) :
object 'B' not found
Is there any solution to use the name of a column as a function argument?
If you want to create a function which accepts the string "B" as an argument (as in you question's title)
foo_string <- function (x) {
eval(substitute(df %>% filter(xx < 1),list(xx=as.name(x))))
}
foo_string("B")
If you want to create a function which accepts captures B as an argument (as in dplyr)
foo_nse <- function (x) {
# capture the argument without evaluating it
x <- substitute(x)
eval(substitute(df %>% filter(xx < 1),list(xx=x)))
}
foo_nse(B)
You can find more information in Advanced R
Edit
dplyr makes things easier in version 0.3. Functions with suffixes "_" accept a string or an expression as an argument
foo_string <- function (x) {
# construct the string
string <- paste(x,"< 1")
# use filter_ instead of filter
df %>% filter_(string)
}
foo_string("B")
foo_nse <- function (x) {
# capture the argument without evaluating it
x <- substitute(x)
# construct the expression
expression <- lazyeval::interp(quote(xx < 1), xx = x)
# use filter_ instead of filter
df %>% filter_(expression)
}
foo_nse(B)
You can find more information in this vignette
I remember a similar question which was answered by #Richard Scriven. I think you need to write something like this.
foo <- function(x,...)filter(x,...)
What #Richard Scriven mentioned was that you need to use ... here. If you type ?dplyr, you will be able to find this: filter(.data, ...) I think you replace .data with x or whatever. If you want to pick up rows which have values smaller than 1 in B in your df, it will be like this.
foo <- function (x,...) filter(x,...)
foo(df, B < 1)