Removing Values in DataFrame Based On Condition R - r

Question

I have a data frame of 4 columns (magnified for this example). Most columns have outliers which are significantly larger than the other values in the data frame. For example: A column (with a maximum value of 99), has outliers with 96, 97, 98, 99. These outliers signify essentially "no response". This obviously heavily skews the data, thus they must be removed. I want to remove the outliers, but each variable has a different maximum value (and different set of outliers) and some have decimals.
96, 97, 98, 99 must be removed ONLY from the columns that have those as reserve values. So the function must know which columns have each specific classification of reserve values. More below.
The issue is that, I do not want to "remove from all columns" the reserve values as some values may mean something else in another column. For example removing 996 in one column could mean something of significance in another column, such as hourly wage/week.
It get tricky as some have decimals like hours worked/week. For example. 37.5 hours worked per week would have reserve values of 999.6, 999.7, 999.8, 999.9.
This length would be classified as 5.1.
I need to remove these reserve values from the data frame, but they must first match the corresponding reserve value length. Since each column has a different reserve value, the column names of the data frame should correspond to a specific reserve value.
df <- data.frame("children#" = c(1,5,0,2,10),
"annual income" = c(700000.00,50000.65,30000.45,1000000.59,9999999.96),
"hour wage"= c(25.65,9999999.99,50.23,1000.72,65.16),
"hours worked/week" = c(148.5,77.0,64.2,25.9,999.7))
Max length of children# is 2
Max length of annual income is 10.2 (10 total, 2 decimal)
Max length of hour wage is 10.2
Max length of hours worked/week is 5.1 (5 total, 1 decimal)
ALWAYS WILL BE 4 RESERVE VALUES
If max length = 2, remove reserve values: 96,97,98,99
If max length = 3, remove reserve values: 996, 997, 998, 999... and so forth with solid numbers
With decimals:
If max length = 5.1, remove reserve values: 999.6, 999.7, 999.8, 999.9.
If max length = 10.2, remove reserve values: 9999999.96, 9999999.97, 9999999.98, 9999999.99
Thus, I would like to figure out how to make a function that will
find max lengths
connect the corresponding max lengths with the correct reserve values
remove reserve values from data frame based on max lengths of each column
So far I have the max lengths of each column with the decimal points.
I just need some help with connecting it to the reserve values and getting those reserve values removed from the data frame.
If more info is required please comment as I will elaborate further if needed.
Code sample: For the reserve values I was thinking of creating a separate data frame and using that to remove the values. Other suggestions are welcome.
Find.Max.Length <- function(data){
# Check Max Length of each column
tmp <- data.frame(lapply(data, function(x) max(nchar(x, keepNA = F))))
tmp <- data.frame(t(tmp))
return(tmp)}
max.length <- Find.Max.Length(df)
Check.Decimal.Places <- function(x){
if((x %% 1) != 0){
nchar(strsplit(sub('0+$', '',as.character(x)), ".", fixed = TRUE)[[1]][[2]])
}else{
return(0)}
}
decimal <- data.frame(Check.Decimal.Places(df$random)) #<--- used to
initialize the variable before the loop
for(i in seq_along(df)){
decimal[i] <- data.frame(Check.Decimal.Places(df[[i]]))}
decimal<- data.frame(t(decimal))
rownames(decimal) <- names(df)
length.df <- cbind(max.length, decimal)
names(length.df) <- c("Max Length", "Decimal Place")
length.df$NewVariableLength <- paste0(length.df$`Max Length`, sep=
".",length.df$`Decimal Place`)
NOTE: Row names of length.df data frame match original data frame names. That can possibly be a way to link the two together?
There is probably a faster way to do this all, all suggestions are welcome.

Answer 1

edit: Now I understand what you mean with "reserve values" - answers from a survey that should not be counted (e.g. "I don't want to answer this question")
You have essentially three easy methods here without having to search of "integer length" or other overengineering:
Max values (i.e., remove the four highest values),
Manual thresholds (i.e., remove all values above X),
If-else logic (i.e., if answer == X, remove it).
Building the dataset
Your data did not correspond to your specifications ("always 4 outliers"), so I took the liberty to extend it.
df <- data.frame(
"children" = c(1, 0, 96, 2, 10, 99, 98, 99),
"annual_income" = c(700000.00, 50000.65, 30000.45, 1000000.59, 9999999.96, 9999999.97, 9999999.98, 9999999.99),
"hour_wage"= c(25.65, 9999999.99, 50.23, 9999999.98, 9999999.99, 9999999.98, 1000.72, 65.16),
"hours_worked_week" = c(148.5, 999.6, 77.0, 64.2, 999.9, 999.8, 25.9, 999.7)
)
df
children annual_income hour_wage hours_worked_week
1 1 700000.00 25.65 148.5
2 0 50000.65 9999999.99 999.6
3 96 30000.45 50.23 77.0
4 2 1000000.59 9999999.98 64.2
5 10 9999999.96 9999999.99 999.9
6 99 9999999.97 9999999.98 999.8
7 98 9999999.98 1000.72 25.9
8 99 9999999.99 65.16 999.7
1. Maximum-Values-Approach (obsolete after clarification)
Load libraries
library(dplyr)
library(magrittr)
Get the four outliers
children_out <- tail(sort(df$children), 4)
Replace outliers with NA
df[df$children %in% children_out,]
%<>% mutate(children = NA)
Check dataset
df
children annual_income hour_wage hours_worked_week
1 1 700000.00 25.65 148.5
2 0 50000.65 9999999.99 999.6
3 NA 30000.45 50.23 77.0
4 2 1000000.59 9999999.98 64.2
5 10 9999999.96 9999999.99 999.9
6 NA 9999999.97 9999999.98 999.8
7 NA 9999999.98 1000.72 25.9
8 NA 9999999.99 65.16 999.7
Caveat: This approach will work only if you always have four outliers for each column.
2. Manual thresholds
Load libraries
library(dplyr)
library(magrittr)
Exclude existing NA and replace anything that is 96 or above with NA
df[!is.na(df$children) & df$children >=96, ] %<>%
mutate(children = NA)
Check dataset
df
children annual_income hour_wage hours_worked_week
1 1 700000.00 25.65 148.5
2 0 50000.65 9999999.99 999.6
3 NA 30000.45 50.23 77.0
4 2 1000000.59 9999999.98 64.2
5 10 9999999.96 9999999.99 999.9
6 NA 9999999.97 9999999.98 999.8
7 NA 9999999.98 1000.72 25.9
8 NA 9999999.99 65.16 999.7
3. If-else logic
Load libraries
library(dplyr)
library(magrittr)
Save "reserved answers"
children_res <- c(96, 97, 98, 99)
Replace anything that is a reserved answer with NA (excluding existing NA is not needed here)
df[df$children %in% children_res, ] %<>%
mutate(children = NA)
Check dataset
df
children annual_income hour_wage hours_worked_week
1 1 700000.00 25.65 148.5
2 0 50000.65 9999999.99 999.6
3 NA 30000.45 50.23 77.0
4 2 1000000.59 9999999.98 64.2
5 10 9999999.96 9999999.99 999.9
6 NA 9999999.97 9999999.98 999.8
7 NA 9999999.98 1000.72 25.9
8 NA 9999999.99 65.16 999.7
4. edit: Combined approach 1&3
Load libraries
library(dplyr)
library(magrittr)
Get "reserved answers"
children_res <- tail(sort(unique(df$children)), 4)
Replace anything that is a reserved answer with NA (excluding existing NA is not needed here)
df[df$children %in% children_res, ] %<>%
mutate(children = NA)
Caveat: This approach will work only if you always have ALL reserved answers (e.g., 96, 97, 98, and 99) present in each column. This will NOT WORK if, by accident, nobody would answer "97".