How do I refer to a specific DataFrame in a spark pipeline? - r

Question

Suppose I have two spark DataFrames with the same features in Spark and I want to build a pipeline to cross validate them both. How can I refer to each table within a pipeline? I use sparklyr in R to do this, but I guess it should be the same with pyspark.
First I can use the following code to build a linear regression and cross evaluate it using ml_cross_validator()
suppressMessages(library(sparklyr))
suppressMessages(library(tidyverse))
sc <- spark_connect(master = "local")
copy_to(sc, mtcars, "mtcars")
mtcars <- tbl(sc, "mtcars")
pipeline <- ml_pipeline(sc) %>%
ft_r_formula(mpg ~ .) %>%
ml_linear_regression()
grid <- list(linear_regression = list(reg_param = 0))
cv <- ml_cross_validator(
sc,
estimator = pipeline, # use our pipeline to estimate the model
estimator_param_maps = grid, # use the params in grid
evaluator = ml_regression_evaluator(sc, metric_name = "rmse"), # how to evaluate the CV
num_folds = 2, # number of CV folds
seed = 2018
)
cv_model <- ml_fit(cv, mtcars)
cv_model$avg_metrics_df
#> rmse reg_param_1
#> 1 3.997882 0
Created on 2019-09-13 by the reprex package (v0.3.0)
But if I add another table with the same features:
mtcars_sample <- sdf_sample(mtcars, fraction = 0.8) %>%
sdf_register("mtcars_sample")
How can I refer to it within the pipeline?
─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
setting value
version R version 3.6.0 (2019-04-26)
os macOS Mojave 10.14.6
system x86_64, darwin15.6.0
ui RStudio
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz Europe/Stockholm
date 2019-09-13