dplyr works differently for dataframe and database - how to resolve? - r

Question

Apparently, dplyr requires different implementation for dataframes and databases.
This is not the first time I encounter this.
The example code is below. The purpose of the code is to remove Inf values from the database.
library(RSQLite)
library(DBI)
# dataframe
data <- data.frame(x = c(rep(1,2), rep(Inf, 3), rep(1, 5)),
y = c(rep(2,5), rep(Inf, 5)),
z = 1:10)
# database
db <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")
copy_to(db, data, name = "data", overwrite = TRUE)
data_db <- tbl(db, "data")
# WORKS for dataframe:
data %>% filter_at(c("x", "y"), all_vars(base::is.finite(.)))
# DOES NOT WORK for database
data_db %>% filter_at(c("x", "y"), all_vars(base::is.finite(.)))
The last line returns an error:
Error in eval_bare(call, env) : object 'x' not found
Is there a good overview of the differences required in implementations of dplyr for dataframes vs. databases?
Please help to have the code above work for the database case.
Thank you

Answer 1

Looking through dbplyr's Function translation vignette, it doesn't look like is.finite() is mentioned, and indeed we can verify that it doesn't know how to translate is.infinite to the SQL command ISFINITE().
dbplyr::translate_sql(is.infinite(x))
# <SQL> is.infinite(`x`)
In this case, as per the Writing SQL with dbplyr vignette, you can use the sql command, something like:
## write idea, but probably won't work
data_db %>% filter(across(c("x", "y"), sql(NOT ISFINITE(.))))
Though that admittedly looks like it has a low chance of working because of the . in the SQL. I'm not sure how well across() (or the older filter_at) dplyr functions play with the sql() function. You may need to write out the columns you want to filter on:
data_db %>% filter(sql(NOT ISFINITE(x)), sql(NOT ISFINITE(y)))
Is there a good overview of the differences required in implementations of dplyr for dataframes vs. databases?
The vignettes mentioned above are both good reading on this.

Answer 2

You could turn it into a proper data.frame or tibble:
data_db <- tbl(db, "data") %>% as_tibble
# WORKS for dataframe:
data %>% filter_at(c("x", "y"), all_vars(base::is.finite(.)))
# DOES NOT WORK for database
data_db %>% filter_at(c("x", "y"), all_vars(base::is.finite(.)))
Output:
> data %>% filter_at(c("x", "y"), all_vars(base::is.finite(.)))
x y z
1 1 2 1
2 1 2 2
> data_db %>% filter_at(c("x", "y"), all_vars(base::is.finite(.)))
# A tibble: 2 x 3
x y z
<dbl> <dbl> <int>
1 1 2 1
2 1 2 2
You could use as.data.frame instead of as_tibble if that is more to your liking.

Answer 3

I have figured it out (almost).
The problem with SQLite is that it does not have function ISFINITE causing Error: no such function: ISFINITE (as I posted in the comment to Gregor's answer).
My solution is to use basic math logic to define Inf:
> data_db %>% filter(sql("x * x != x or x = 0 or x = 1"),
+ sql("y * y != y or y = 0 or y = 1"))
# Source: lazy query [?? x 3]
# Database: sqlite 3.33.0 [:memory:]
x y z
<dbl> <dbl> <int>
1 1 2 1
2 1 2 2
The only thing is that I could not figure out how to make this work with filter(across(c("x", "y"))