R SQL templating with glue_sql, ability to dynamically drop where clause - r

TLDR
I would like to be able to template SQL queries and run them in R. The glue package and DBI work great, but I can't figure out a way to template statements. In other words, is there a way to do something like this (borrowing from jinja):
SELECT * FROM mtcars
{% if length( {make} ) > 0 %}
WHERE make IN( {make*}
{% end %}
Additional Detail
DBI and glue work great for a single use case, but often I want to reuse the same general SQL code with a few different variations of WHERE clauses and things like that. Often I want the WHERE to be "off". in some of the use cases and not in others (e.g. for WHERE IN() it defaults to all values, for WHERE x >= y it doesn't apply the conditional at all, etc.).
The only solution I can find is to evaluate inputs in R as discussed here, and then pass a default vector or the input. This approach works in some use cases and not at all in others. I think it makes it harder to generalize and has a performance hit in my most common use case - when I want a query with a parameter that passes values to a WHERE IN() clause, but defaults to all values. If the table is evolving (i.e. all values changes over time) then I need to first run a query to get all values, then input them if the user doesn't provide values. That can be expensive on larger tables and prohibitive if it's in a user experience (shiny).
library(DBI)
library(glue)
library(dplyr, warn.conflicts = F)
# Setup local DB ####
con <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")
mtcars_df <- tibble::rownames_to_column(mtcars, var = "make")
str(mtcars_df)
#> 'data.frame': 32 obs. of 12 variables:
#> $ make: chr "Mazda RX4" "Mazda RX4 Wag" "Datsun 710" "Hornet 4 Drive" ...
#> $ mpg : num 21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
#> $ cyl : num 6 6 4 6 8 6 8 4 4 6 ...
#> $ disp: num 160 160 108 258 360 ...
#> $ hp : num 110 110 93 110 175 105 245 62 95 123 ...
#> $ drat: num 3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
#> $ wt : num 2.62 2.88 2.32 3.21 3.44 ...
#> $ qsec: num 16.5 17 18.6 19.4 17 ...
#> $ vs : num 0 0 1 1 0 1 0 1 1 1 ...
#> $ am : num 1 1 1 0 0 0 0 0 0 0 ...
#> $ gear: num 4 4 4 3 3 3 3 4 4 4 ...
#> $ carb: num 4 4 1 1 2 1 4 2 2 4 ...
DBI::dbWriteTable(con, "mtcars", mtcars_df)
# Example query ####
sql <- glue::glue_sql("SELECT * FROM mtcars WHERE make IN( {make*} )", make = c("Fiat X1-9", "Datsun 710"), .con = con)
DBI::dbGetQuery(con, sql)
#> make mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
#> 2 Fiat X1-9 27.3 4 79 66 4.08 1.935 18.90 1 1 4 1
# Templating ####
sql <- "SELECT * FROM mtcars WHERE make IN( {make*} )"
sql_template <- tempfile(fileext = ".sql")
readr::write_file(sql, sql_template)
read_sql <- function(file, ..., .con, .envir = parent.frame()){
sql <- readr::read_file(file)
sql <- glue::glue_sql(sql, ..., .con = .con, .envir = .envir)
}
# SQL files can be templated and called from R
sql <- read_sql(sql_template, make = c("Fiat X1-9", "Datsun 710"), .con = con)
DBI::dbGetQuery(con, sql)
#> make mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
#> 2 Fiat X1-9 27.3 4 79 66 4.08 1.935 18.90 1 1 4 1
# All {values} must be provided, errors out
sql <- read_sql(sql_template, .con = con)
#> Error in eval(parse(text = text, keep.source = FALSE), envir): object 'make' not found
# Doesn't return anything
sql <- read_sql(sql_template, make = DBI::SQL(""), .con = con)
print(sql)
#> <SQL> SELECT * FROM mtcars WHERE make IN( )
DBI::dbGetQuery(con, sql)
#> [1] make mpg cyl disp hp drat wt qsec vs am gear carb
#> <0 rows> (or 0-length row.names)
# Can't make the entire where clause a parameter either without doing a lot of escapes and basically defeating the purppose of glue
sql <- glue::glue_sql("SELECT * FROM mtcars {makes}", makes = "WHERE make IN('Fiat X1-9', 'Datsun 710')", .con = con)
print(sql)
#> <SQL> SELECT * FROM mtcars 'WHERE make IN(''Fiat X1-9'', ''Datsun 710'')'
DBI::dbGetQuery(con, sql)
#> make mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
#> 2 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
#> 3 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
#> 4 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
#> 5 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
#> 6 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
#> 7 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
#> 8 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
#> 9 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
#> 10 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
#> 11 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
#> 12 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
#> 13 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
#> 14 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
#> 15 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
#> 16 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
#> 17 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
#> 18 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
#> 19 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
#> 20 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
#> 21 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
#> 22 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
#> 23 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
#> 24 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
#> 25 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
#> 26 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
#> 27 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
#> 28 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
#> 29 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
#> 30 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
#> 31 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
#> 32 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2
# Get all values first
all_makes <- DBI::dbGetQuery(con, "SELECT DISTINCT make FROM mtcars") %>% dplyr::pull(make)
sql <- read_sql(sql_template, make = all_makes, .con = con)
DBI::dbGetQuery(con, sql)
#> make mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
#> 2 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
#> 3 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
#> 4 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
#> 5 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
#> 6 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
#> 7 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
#> 8 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
#> 9 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
#> 10 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
#> 11 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
#> 12 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
#> 13 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
#> 14 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
#> 15 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
#> 16 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
#> 17 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
#> 18 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
#> 19 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
#> 20 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
#> 21 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
#> 22 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
#> 23 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
#> 24 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
#> 25 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
#> 26 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
#> 27 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
#> 28 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
#> 29 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
#> 30 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
#> 31 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
#> 32 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2
# Templating with a conditional####
sql <- "SELECT * FROM mtcars WHERE cyl >= {cyl} "
sql_template <- tempfile(fileext = ".sql")
readr::write_file(sql, sql_template)
read_sql <- function(file, ..., .con, .envir = parent.frame()){
sql <- readr::read_file(file)
sql <- glue::glue_sql(sql, ..., .con = .con, .envir = .envir)
}
# No way to use the all values approach since it's a one sided conditional
sql <- read_sql(sql_template, cyl = 8, .con = con)
DBI::dbGetQuery(con, sql)
#> make mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
#> 2 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
#> 3 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
#> 4 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
#> 5 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
#> 6 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
#> 7 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
#> 8 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
#> 9 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
#> 10 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
#> 11 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
#> 12 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
#> 13 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
#> 14 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8

You can use multiple arguments to glue::glue_sql:
# con <- dbConnect(...)
make <- c()
glue::glue_sql(
"select * from mtcars",
if (length(make)) " where make in ({make*})" else "",
.con = con)
# <SQL> select * from mtcars
make <- c("Fiat X1-9", "Datsun 710")
glue::glue_sql( # unchanged
"select * from mtcars",
if (length(make)) " where make in ({make*})" else "",
.con = con)
# <SQL> select * from mtcars where make in ('Fiat X1-9', 'Datsun 710')

Using jinjar v0.2.0:
query <- "SELECT * FROM mtcars
{% if length(make) > 0 -%}
WHERE make IN ({{ quote_sql(make) }})
{%- endif %}"
jinjar::render(query, make = c()) |> writeLines()
#> SELECT * FROM mtcars
jinjar::render(query, make = c("Fiat X1-9", "Datsun 710")) |> writeLines()
#> SELECT * FROM mtcars
#> WHERE make IN ('Fiat X1-9', 'Datsun 710')
Created on 2022-06-24 by the reprex package (v2.0.1)

Related

Is there a tidy way to mutate cluster predictions/assignments for methods not covered in broom?

It is a reference to this:
Alternatives to kmeans() for spotting small clusters for k=2
In the comments it is suggested something like this:
cc = ClusterR::GMM(mtcars, gaussian_comps = 2); predict(cc, mtcars)
But assume I have a tibble database db, I would like a code working like:
db %>%
mutate(cluster = clustering_function(selection())) -> db
Whereas selection is a selector of columns, and the clustering function put together the clustering methods and the assignment of predicted values.
I think your best bet in this situation is bind_cols():
library(tidyverse)
library(ClusterR)
#> Loading required package: gtools
cluster_fit <- ClusterR::GMM(mtcars, gaussian_comps = 2)
bind_cols(
mtcars,
.preds = predict(cluster_fit, mtcars)
)
#> mpg cyl disp hp drat wt qsec vs am gear carb .preds
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 1
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 1
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 1
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 2
#> Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 1
#> Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 2
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 1
#> Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 1
#> Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 1
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 1
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 2
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 2
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 2
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4 2
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4 2
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4 2
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 1
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 1
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 1
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 1
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 2
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 2
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4 2
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 2
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 1
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 1
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 1
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 2
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 1
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8 2
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2 1
Created on 2022-12-01 with reprex v2.0.2

Calculate row sums exclude the first n columns

I need to calculate row sums for a data frame except for the first 5 columns. The output will consist of these first 5 columns and the row sums.
I tried this:
df1$rowsums <- rowSums(df1[,-c(1:5)], na.rm= T)
But I get this error message:
Error in rowSums(df1[, c(1:5)], na.rm = T) : 'x' must be numeric
without data my guess is, that the columns you are using are not numeric. Then it will be hard to calculate the rowsum. Make sure, that columns you use for summing (except 1:5) are indeed numeric, then the following code should work:
library(tidyverse)
df2 <- df1[,-c(1:5)] %>%
rowwise() %>%
mutate(rowsum = sum(c_across(everything()), na.rm = T))
df_result <- cbind(df1[,c(1:5)], df2$rowsum)
EDIT: I added na.rm = T (dont know if necessary). And you might want to rename the resulting "df2$rowsum" column of the resulting df_result dataframe this can be done using
df_result <- df_result %>% rename(rowsum_name = "df2$rowsum")
You could select the columns except the first 5 by -c(1:5) and use rowSums like this (I use mtcars as an example):
library(dplyr)
mtcars %>%
mutate(rowsums = select(., -c(1:5)) %>%
rowSums(na.rm = TRUE))
#> mpg cyl disp hp drat wt qsec vs am gear carb rowsums
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 28.080
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 28.895
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 27.930
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 27.655
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 25.460
#> Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 28.680
#> Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 26.410
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 30.190
#> Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 33.050
#> Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 30.740
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 31.340
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 27.470
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 27.330
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 27.780
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4 30.230
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4 30.244
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4 29.765
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 28.670
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 28.135
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 28.735
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 27.475
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 25.390
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 25.735
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4 26.250
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 25.895
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 27.835
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 26.840
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 27.413
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 27.670
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 30.270
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8 32.170
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2 29.380
Created on 2022-07-09 by the reprex package (v2.0.1)

Cleaner way to reshape dataframe with many variables in R [duplicate]

This question already has answers here:
Reshaping data.frame from wide to long format
(8 answers)
Closed 1 year ago.
So I want to reshape my wide format dataframe, and my varying variables range from item01-item21. Is there a shortcut that i can use instead of manually typing item01, item02, etc? This is the code I have now:
RTdataLong <- reshape(RTdata,direction="long",idvar=c("ID","Language","Gender"), varying=list(c("Time01","Time02","Time03","Time04","Time05","Time06","Time07","Time08","Time09","Time10","Time11","Time12","Time13","Time14", "Time15","Time16","Time17","Time18","Time19","Time20","Time21")))
It works, but I would really appreicate it if someone could give me some tips on how to do this more efficiently.
You don't provide an exmaple data but you could use tidyr::pivot_longer like so:
library(tidyverse)
mtcars
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
#> Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
#> Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
#> Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
#> Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2
mtcars %>%
pivot_longer(cols = 1:11, names_to = "col_name")
#> # A tibble: 352 x 2
#> col_name value
#> <chr> <dbl>
#> 1 mpg 21
#> 2 cyl 6
#> 3 disp 160
#> 4 hp 110
#> 5 drat 3.9
#> 6 wt 2.62
#> 7 qsec 16.5
#> 8 vs 0
#> 9 am 1
#> 10 gear 4
#> # ... with 342 more rows
Created on 2021-09-15 by the reprex package (v2.0.1)

How to compare two columns with assertr

I want to assert that one column in my data is always greater than the other column using the assertr package. As an example let's day that mtcars mpg should always be greater than cyl. Here is what I've tried but it throws an error. Am I making a simple mistake?
library(assertr)
greater_than <- function(x, y){if(x <= y) return(FALSE)}
assert(mtcars, greater_than, x = mpg, y = cyl)
> Error in improper.predicate(x) : argument "y" is missing, with no default
I don't think you want assert - instead I think you want assert_rows. That means you need a row reduction function (takes a row and results in a single value) in addition to the predicate function. Here the reduction function just finds the difference between the first two columns of a data frame. Then the last argument in assert_rows tells it to essentially use a data frame only consisting of mpg and cyl (in that order) for passing to the row reduction function.
I will say, the documentation is not great for this package. I had to go to their GitHub and then consult the code of assert_rows directly to come up with this answer.
library(assertr)
greater_than_0 <- function(x){if(x <= 0) return(FALSE)}
row_redux <- function(df){df[[1]] - df[[2]]}
assert_rows(mtcars, row_redux, greater_than_0, c(mpg, cyl))
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
#> Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
#> Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
#> Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
#> Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2
Created on 2019-09-17 by the reprex package (v0.3.0)

Modifying dataframe within a list-column

I have a dataframe with a list-column which itself contains dataframes (see below). Essentially, I am trying to add values from another column in the parent dataframe into the smaller dataframe by creating another column.
This is a simplified example- my real application is more complex.
library(tidyverse)
# What I am trying to do: add column "a" to dataframe within the list column
add_column(mtcars, a = 1)
#> mpg cyl disp hp drat wt qsec vs am gear carb a
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 1
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 1
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 1
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 1
#> Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 1
#> Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 1
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 1
#> Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 1
#> Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 1
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 1
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 1
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 1
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 1
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4 1
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4 1
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4 1
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 1
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 1
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 1
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 1
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 1
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 1
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4 1
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 1
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 1
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 1
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 1
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 1
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 1
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8 1
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2 1
Then create list-column:
(df <- tibble(data = rep(list(mtcars), times = 3), a = 1:3))
#> # A tibble: 3 x 2
#> data a
#> <list> <int>
#> 1 <data.frame [32 x 11]> 1
#> 2 <data.frame [32 x 11]> 2
#> 3 <data.frame [32 x 11]> 3
But this doesn't work:
df %>%
rowwise() %>%
modify_at("data", ~ add_column(., a = a))
# Error in eval_tidy(xs[[i]], unique_output): object 'a' not found
We may use
df %>% mutate(data = data %>% map2(a, ~add_column(.x, a = .y)))
In this way we start by mutating a column as usual, but then recognising that it's a list we use map2 along with the a column.

Resources