How to add multiple calculated columns to a SparkDataFrame using SparkR? - r

Question

Now I'm stuck on a rather basic case, but I can't find a clever solution with SparkR ...
From N columns in my SparkDataFrame, I need to create N new calculated columns.
df <- data.frame(V1 = base::sample(1:10,5),
V2 = base::sample(1:10,5),
V3 = base::sample(1:10,5),
V4 = base::sample(1:10,5),
V5 = base::sample(1:10,5),
X = runif(n = 5, min = 0, max = 5))
sdf <- createDataFrame(df)
sdf <- withColumn(sdf, "V1_X", column("X") / column("V1"))
sdf <- withColumn(sdf, "V2_X", column("X") / column("V2"))
sdf <- withColumn(sdf, "V3_X", column("X") / column("V3"))
sdf <- withColumn(sdf, "V4_X", column("X") / column("V4"))
sdf <- withColumn(sdf, "V5_X", column("X") / column("V5"))
Basically, I want to apply a function to my vector/list of columns names.
Easy in R. In SparkR, I am able to lapply a function, but I modify the original columns. Something escapes me...!?
Thanks !

Answer 1

Maybe you can consider the following approach
library(SparkR)
df <- data.frame(V1 = base::sample(1:10,5),
V2 = base::sample(1:10,5),
V3 = base::sample(1:10,5),
V4 = base::sample(1:10,5),
V5 = base::sample(1:10,5),
X = runif(n = 5, min = 0, max = 5))
sdf <- createDataFrame(df)
col_Names <- colnames(sdf)
nb_Col_Ratio <- length(col_Names) - 1
for(i in 1 : nb_Col_Ratio)
{
new_Col_Name <- paste(col_Names[i], "X")
sdf <- withColumn(sdf, new_Col_Name, column("X") / column(col_Names[i]))
}
print(as.data.frame(sdf))

Answer 2

I usually prefer doing these kind of tasks using SparkR::select command, instead of looping over multiple withColumn statements. The same logic can be used in Python using Python's list comprehension. See the modified version of your code below:
df <- data.frame(V1 = base::sample(1:10,5),
V2 = base::sample(1:10,5),
V3 = base::sample(1:10,5),
V4 = base::sample(1:10,5),
V5 = base::sample(1:10,5),
X = runif(n = 5, min = 0, max = 5))
sdf <- SparkR::createDataFrame(df)
# sdf <- SparkR::withColumn(sdf, "V1_X", SparkR::column("X") / SparkR::column("V1"))
# sdf <- SparkR::withColumn(sdf, "V2_X", SparkR::column("X") / SparkR::column("V2"))
# sdf <- SparkR::withColumn(sdf, "V3_X", SparkR::column("X") / SparkR::column("V3"))
# sdf <- SparkR::withColumn(sdf, "V4_X", SparkR::column("X") / SparkR::column("V4"))
# sdf <- SparkR::withColumn(sdf, "V5_X", SparkR::column("X") / SparkR::column("V5"))
loop_on_cols <- SparkR::colnames(sdf)[SparkR::colnames(sdf)!="X"]
sdf2 <- SparkR::select(
sdf,
c(
SparkR::colnames(sdf),
lapply(
loop_on_cols,
function(c) {
SparkR::alias(SparkR::column("X")/SparkR::column(c), paste0(c,"_X"))
}
)
)
)
SparkR::head(sdf2)