R Removing duplicate columns based on variability - r

Question

I have a very large dataset with multiple duplicated column names (the values within the columns are different). I would like to remove the columns with duplicate names and lower variability. My problem is I have too many of these duplicate variables to do this manually. One path I am trying to to use read.csv(), which automatically adds '.1' to the duplicate column name, then make a vector of the variability of all the columns and try to work with that.
df<-data.frame("A"=c(1,5,10), "A.1"=c(2,2,2), "C"=c(1,5,10), "C.1"=c(2,2,2), "C.2"=c(2,5,10))
v<-lapply(df, function(x) var(x))
Is there a way to filter out duplicates based on variability when I am importing the dataset? Again, the biggest problem is that I have too many duplicates to do this manually. Thanks in advance!

Answer 1

Combining techniques from base R and tidyverse:
# calculate variance for each column
dvar <- apply(df, 2, var)
library(tidyverse)
# create data frame with column names
# "grouped" column names and variance
# find column with highest variance
keep_names <- data.frame(names = names(dvar),
grouping = gsub("[[:punct:]][0-9]", "", names(dvar)),
vals = dvar) %>%
group_by(grouping) %>%
slice_max(vals) %>%
pull(names)
# pull data
df[keep_names]
# A C
# 1 1 1
# 2 5 5
# 3 10 10

Answer 2

df <- data.frame("A"=c(1,5,10), "A.1"=c(2,2,2), "C"=c(1,5,10), "C.1"=c(2,2,2), "C.2"=c(2,5,10))
df["var",] <-apply(df, 2, var)
nrow(df)
df <- df[,which(df["var",] < 20)]
df
Imagine "20" being your threshold. I used apply here an appended it to the dataframe.
A.1 C.1 C.2
1 2 2 2.00000
2 2 2 5.00000
3 2 2 10.00000
var 0 0 16.33333

Answer 3

I would like to propose a new version using exponential search and data.table power.
This a function I implemented in dataPreparation package.
The function
dataPreparation::which_are_bijection
which_are_in_double(df)
Which return 3 and 4 the columns that are duplicated in your example
Build a data set with wanted dimensions for performance tests
df<-data.frame("A"=c(1,5,10), "A.1"=c(2,2,2), "C"=c(1,5,10), "C.1"=c(2,2,2), "C.2"=c(2,5,10))
for (i in 1:20){
df = rbind(df, df)
}
Which result in a data.frame of size (3 145 728, 5)
The benchmark
To perform the benchmark, I use the library rbenchmark which will reproduce each computations 100 times
benchmark(
which_are_in_double(df, verbose=FALSE),
apply(df, 2, var) == 0
)
test replications elapsed relative
2 apply(df, 2, var) == 0 100 38.298 3.966
1 which_are_in_double(df, verbose = FALSE) 100 9.656 1.000
So which are bijection is 4 time faster than other proposed solution. The nice thing is that the bigger the data.frame the performance will be even more interesting.