I want to have the year column selected too in the output but I dont understand how to add another column in the select statement. The column I want to select is a simple column named "Year".
Following is the code I am trying to use but it gives me an error. The select_if condition is correct because if I do not add the "select("Year")" line the code works fine.
Please help
Subset <-df %>%
select("Year") & select_if(grepl("Apple", names(.)))
library(dplyr)
mtcars %>%
select(cyl, contains("gea")) %>%
head()
cyl gear
Mazda RX4 6 4
Mazda RX4 Wag 6 4
Datsun 710 4 4
Hornet 4 Drive 6 3
Hornet Sportabout 8 3
Valiant 6 3
Related
I want to make a subset of my data, by selecting columns, like below:
select(df, col1, col2, col3, col4)
But sometimes I have a slightly different data set, with only col1, col2 and col4.
How can I use select(), and If a column doesn't exist, it just continues without giving an error?
So it would give a dataset with col1, col2 and col4 (and skip col3). If I just run the above select() line, I get this error:
Error in overscope_eval_next(overscope, expr) : object 'col3' not found
df[, names(df) %in% c('col1', 'col2', 'col3', 'col4')]
You can use the one_of() select helper from dplyr and pass the column names as strings. It will just issue a warning for columns that don't exist.
library(dplyr)
select(mtcars, one_of(c("mpg", "disp", "foo")))
#> Warning: Unknown variables: `foo`
#> mpg disp
#> Mazda RX4 21.0 160.0
#> Mazda RX4 Wag 21.0 160.0
#> Datsun 710 22.8 108.0
#> Hornet 4 Drive 21.4 258.0
#> Hornet Sportabout 18.7 360.0
This question already has answers here:
how to drop columns by passing variable name with dplyr?
(6 answers)
Closed 5 years ago.
I'm interested in simplifying the way that I can remove columns with dplyr (version >= 0.7). Let's say that I have a character vector of names.
drop <- c("disp", "drat", "gear", "am")
Selecting Columns
With the current version version of dplyr, you can perform a selection with:
dplyr::select(mtcars, !! rlang::quo(drop))
Or even easier with base R:
mtcars[, drop]
Removing Columns
Removing columns names is another matter. We could use each unquoted column name to remove them:
dplyr::select(mtcars, -disp, -drat, -gear, -am)
But, if you have a data.frame with several hundred columns, this isn't a great solution. The best solution I know of is to use:
dplyr::select(mtcars, -which(names(mtcars) %in% drop))
which is fairly simple and works for both dplyr and base R. However, I wonder if there's an approach which doesn't involve finding the integer positions for each column name in the data.frame.
Use modify_atand set columns to NULL which will remove them:
mtcars %>% modify_at(drop,~NULL)
# mpg cyl hp wt qsec vs carb
# Mazda RX4 21.0 6 110 2.620 16.46 0 4
# Mazda RX4 Wag 21.0 6 110 2.875 17.02 0 4
# Datsun 710 22.8 4 93 2.320 18.61 1 1
# Hornet 4 Drive 21.4 6 110 3.215 19.44 1 1
# Hornet Sportabout 18.7 8 175 3.440 17.02 0 2
# Valiant 18.1 6 105 3.460 20.22 1 1
# ...
Closer to what you were trying, you could have tried magrittr::extract instead of dplyr::select
extract(mtcars,!names(mtcars) %in% drop) # same output
You can use -one_of(drop) with select:
drop <- c("disp", "drat", "gear", "am")
select(mtcars, -one_of(drop)) %>% names()
# [1] "mpg" "cyl" "hp" "wt" "qsec" "vs" "carb"
one_of evaluates the column names in character vector to integers, similar to which(... %in% ...) does:
one_of(drop, vars = names(mtcars))
# [1] 3 5 10 9
which(names(mtcars) %in% drop)
# [1] 3 5 9 10
I have several similar data frames with many columns in common. I would like to select and rename a subset of those columns from any table.
library(tidyverse)
mtcars %>%
select(my_mpg = mpg,
cylinders = cyl,
gear)
Is it possible to do something like
my_select_rename <- c("my_mpg"="mpg","cylinders"="cyl","gear")
mtcars %>%
select_(.dots = my_select_rename)
but using the tidyeval framework instead?
I think you want:
my_select <- c("mpg","cyl","gear")
my_select_rename <- c("my_mpg","cylinders","gear")
mtcars %>%
select_at(vars(my_select)) %>%
setNames(., my_select_rename)
my_mpg cylinders gear
Mazda RX4 21.0 6 4
Mazda RX4 Wag 21.0 6 4
Datsun 710 22.8 4 4
Hornet 4 Drive 21.4 6 3
Hornet Sportabout 18.7 8 3
lionel's answer to this question group_by by a vector of characters using tidy evaluation semantics provides the answer
mtcars %>%
select(!!! rlang::syms(my_select_rename))
I have a pretty big data.table (500 x 2000), and I need to find out if any of the columns are duplicates, i.e., have the same values for all rows. Is there a way to efficiently do this within the data.table structure?
I have tried a naive two loop approach with all(col1 == col2) for each pair of columns, but it takes too long. I have also tried converting it to a data.frame and using the above approach, and it still takes quite a long time.
My current solution is to convert the data.table to a matrix and use the apply() function as:
similarity.matrix <- apply(m, 2, function(x) colSums(x == m)))/nrow(m)
However, the approach forces the modes of all elements to be the same, and I'd rather not have that happen. What other options do I have?
Here is a sample construction for the data.table:
m = matrix(sample(1:10, size=1000000, replace=TRUE), nrow=500, ncol=2000)
DF = as.data.frame(m)
DT = as.data.table(m)
Following the suggestion of #Haboryme*, you can do this using duplicated to find any duplicated vectors. duplicated usually works rowwise, but you can transpose it with t() just for finding the duplicates.
DF <- DF[ , which( !duplicated( t( DF ) ) ) ]
With a data.table, you may need to add with = FALSE (I think this depends on the version of data.table you're using).
DT <- DT[ , which( !duplicated( t( DT ) ) ), with = FALSE ]
*#Haboryme, if you were going to turn your comment into an answer, please do and I'll remove this one.
Here's a different approach, where you hash each column first and then call duplicated.
library(digest)
dups <- duplicated(sapply(DF, digest))
DF <- DF[,which(!dups)]
Depending on your data this might be a faster way.
I am using mtcars for a reproducible result:
library(data.table)
library(digest)
# Create data
data <- as.data.table(mtcars)
data[, car.name := rownames(mtcars)]
data[, car.name.dup := car.name] # create a duplicated row
data[, car.name.not.dup := car.name] # create a second duplicated row...
data[1, car.name.not.dup := "Moon walker"] # ... but change a value so that it is no longer a duplicated column
data contains now:
> head(data)
mpg cyl disp hp drat wt qsec vs am gear carb car.name car.name.dup car.name.not.dup
1: 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Mazda RX4 Moon walker
2: 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 Mazda RX4 Wag Mazda RX4 Wag Mazda RX4 Wag
3: 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 Datsun 710 Datsun 710 Datsun 710
4: 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 Hornet 4 Drive Hornet 4 Drive Hornet 4 Drive
5: 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 Hornet Sportabout Hornet Sportabout Hornet Sportabout
6: 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 Valiant Valiant Valiant
Now find the duplicated colums:
# create a vector with the checksum for each column (and keep the column names as row names)
col.checksums <- sapply(data, function(x) digest(x, "md5"), USE.NAMES = T)
# make a data table with one row per column name and hash value
dup.cols <- data.table(col.name = names(col.checksums), hash.value = col.checksums)
# self join using the hash values and filter out all column name pairs that were joined to themselves
dup.cols[dup.cols,, on = "hash.value"][col.name != i.col.name,]
Results in:
col.name hash.value i.col.name
1: car.name.dup 58fed3da6bbae3976b5a0fd97840591d car.name
2: car.name 58fed3da6bbae3976b5a0fd97840591d car.name.dup
Note: The result still contains both directions (col1 == col2 and col2 == col1) and should be deduplicated ;-)
If I have a sample data frame like mtcars, and I want to find the difference between mtcars$qsec for all rows, I can do diff(mtcars$qsec). But is there a simple way to make diff(mtcars$qsec) a new column in the original mtcars data frame? I'm finding it difficult because there's one less row in diff(mtcars$qsec) than the rest of mtcars.
> head(mtcars,3)
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Here are two approaches. Both put an NA in the first row of diff_qsec and put diff(qsec) in the remaining rows:
library(dplyr)
mtcars %>% mutate(diff_qsec = qsec - lag(qsec)) # dplyr has its own version of lag
transform(mtcars, diff_qsec = c(NA, diff(qsec)))
Also, on the general issue of padding see: How can I pad a vector with NA from the front?
You could use the base function within() like so:
mtcars <- within(mtcars, difference <- c(NA,diff(qsec)))
This creates a column called "difference" with the first element NA and the rest calculated by diff(qsec).
You could create more columns at the same time by wrapping commands in {}, such as:
mtcars <- within(mtcars, {difference <- c(NA,diff(qsec))
multiple <- qsec*2})
Note that you must use <- for the assignment and not =.