compare variables and remove one with lowest value R [closed] - r

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I have a dataframe of the correlation between 45 variables, and have added the the random forest importance value given to each by the 'varImp' function (I ran a random forest training model with this data.
I would like to run through each column, and wherever a variable has a correlation over .8 (in absolute terms), remove either that row variable or that column variable, whichever has the lower 'varImp' importance. I would also like to remove the same variable from the column/row (since it's a correlation matrix, all variables show up in both a row and a column).
For example, roll_belt and max_picth_belt have a correlation of ~.97, and because roll_belt has a value of 3.77 compared to max_picth_belt's 3.16, I would like to delete max_pitch_belt both as a row, and as a column.
Thanks for your help!

Answer 1

I'm sure there must be a more straightforward way. Still, my code does the job.
Assume, we've loaded your dataset into an object called df (I do not include the code to get your data as it is not relevant).
First, it's handy to I split the data itself and the value column that is used for testing feature importance. New object called test.value is the 46-th column.
test.value <- df$value
df <- df[,-ncol(df)] # remove the last column from the dataset
Now we are ready to start.
The framework. We need to identify the numbers of rows/columns to remove from the dataset. So we will:
go column by column
identify the positions of all correlates bigger than 0.8
compare feature importance one by one in a nested loop
record the row/column numbers that should be removed in an object
remove
finally, remove the chosen rows/columns
The code is:
remove <- c() # a vector to store features to be removed
for(i in 1:ncol(df)){
coli <- df[,i] # pick up i-th column
highcori <- coli>.8 & coli!=1 # logical vector of cors > 0.8
# go further only if there are cors > 0.8
if(sum(highcori,na.rm = T)>0){
posi <- which(highcori) # identify positions of cors > 0.8
# compare feature importance one by one
for(k in 1:length(posi)){
remi <- ifelse(test.value[i]>test.value[posi[k]],posi[k],i)
remove <- c(remove,remi) # store the less valued feature
}
}
}
remove <- sort(unique(remove)) # keep only unique entries
df.clean <- df[-remove,-remove] # finally, clean the dataset
That's it.
UPDATE
For those who can provide a better solution, here are the data in an easily readable form, cor.remove.RData
OR
if you prefer dput
dput.df.txt
dput.test.value.txt
I would be interested to see a better way of solving the task.