R - Removing all outliers from a data set - r

Question

I'd like to make a function that removes all outliers from a data set. I've read a lot of Stack Overflow articles about this, so I am aware of the dangers of removing outliers. Here's what I have so far:
# Remove outliers from a column
remove_outliers <- function(x, na.rm = TRUE, ...) {
qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)
H <- 1.5 * IQR(x, na.rm = na.rm)
y <- x
y[x < (qnt[1] - H)] <- NA
y[x > (qnt[2] + H)] <- NA
y
}
# Removes all outliers from a data set
remove_all_outliers <- function(df){
# We only want the numeric columns
a<-df[,sapply(df, is.numeric)]
b<-df[,sapply(df, !is.numeric)]
a<-lapply(a,function(x) remove_outliers(x))
d<-merge(a,b)
d
}
There are a few things wrong with this that I know of, but please correct me if anything could be handled better.
!is.numeric() is not a thing, How should I accomplish this?
I have allso tried is.numeric==FALSE
is.numeric() converts factors into ints. How do I prevent this?
Did I do lapply right?
Is there a better / easier way to perform the remove_outliers function than separating the data set, performing it, then merging it back with the non-numeric set?

Answer 1

Factors are ints, just not atomic ints.
# Remove outliers from a column
remove_outliers <- function(x, na.rm = TRUE, ...) {
qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)
H <- 1.5 * IQR(x, na.rm = na.rm)
y <- x
y[x < (qnt[1] - H)] <- NA
y[x > (qnt[2] + H)] <- NA
y
}
You can replace the columns by index so you don't need to create separate data sets. Just make sure you pass the same data to lapply, eg, you don't want to do data[, 1:3] <- lapply(data, FUN) which I have done many times.
# Removes all outliers from a data set
remove_all_outliers1 <- function(df){
# We only want the numeric columns
df[,sapply(df, is.numeric)] <- lapply(df[,sapply(df, is.numeric)], remove_outliers)
df
}
Similar to above (and slightly easier I think), you can pass the entire data set to lapply. Also making sure not to
data <- lapply(data, if (x) something else anotherthing)
or
data[] <- lapply(data, if (x) something)
Which are also mistakes I have made many times
remove_all_outliers2 <- function(df){
df[] <- lapply(df, function(x) if (is.numeric(x))
remove_outliers(x) else x)
df
}
## test
mt <- within(mtcars, {
mpg <- factor(mpg)
gear <- letters[1:2]
})
head(mt)
identical(remove_all_outliers1(mt), remove_all_outliers2(mt))
# [1] TRUE
Your ideas can work with a few minor adjustments. !is.numeric can work as either Negate(is.numeric) or the more verbose function(x) !is.numeric(x) or !sapply(x, is.numeric). Generally, function(function) doesn't work in r out of the box.
# Removes all outliers from a data set
remove_all_outliers <- function(df){
# We only want the numeric columns
## drop = FALSE in case only one column for either
a<-df[,sapply(df, is.numeric), drop = FALSE]
b<-df[,sapply(df, Negate(is.numeric)), drop = FALSE]
## note brackets
a[]<-lapply(a, function(x) remove_outliers(x))
## stack them back together, not merge
## you could merge if you had a unique id, one id per row
## then make sure the columns are returned in the original order
d<-cbind(a,b)
d[, names(df)]
}
identical(remove_all_outliers2(mt), remove_all_outliers(mt))
# [1] TRUE