Unable to install spark with sparklyr in Dockerfile - r

Question

We are trying to build our own docker image to use R and the tidyverse with Spark. However, we are getting an error in the build when trying to install Spark.
Here is our Dockerfile:
# start with the most up-to-date tidyverse image as the base image
FROM rocker/tidyverse:latest
# install openjdk 8 (Java)
RUN apt-get update \
&& apt-get install -y openjdk-8-jdk
# Install sparklyr
RUN install2.r --error --deps TRUE sparklyr
# Install spark
RUN Rscript -e 'sparklyr::spark_install("2.4.3")'
RUN mv /root/spark /opt/ && \
chown -R rstudio:rstudio /opt/spark/ && \
ln -s /opt/spark/ /home/rstudio/
RUN install2.r --error --deps TRUE DBI
RUN install2.r --error --deps TRUE RPostgres
RUN install2.r --error --deps TRUE dbplyr
We are using Docker compose up to build and then create the container.
When building, it throws the error:
=> ERROR [4/8] RUN Rscript -e 'sparklyr::spark_install("2.3.0")' 62.0s
------
> [4/8] RUN Rscript -e 'sparklyr::spark_install("2.3.0")':
#7 61.85 Error in download.file(installInfo$packageRemotePath, destfile = installInfo$packageLocalPath, :
#7 61.85 download from 'https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz' failed
#7 61.85 Calls: <Anonymous>
#7 61.85 Execution halted
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c Rscript -e 'sparklyr::spark_install("2.3.0")']: exit code: 1
We have also tried running it as:
RUN R -e 'sparklyr::spark_install("2.4.3")'
Instead of:
RUN Rscript -e 'sparklyr::spark_install("2.4.3")'
but it stil throws an error. We have also tried installing different versions of Spark to see if that would work but still no luck. Does anyone know why I am getting this error and how I can properly install spark with sparklyr in docker? Thank you.