Parallelise a for loop inside an R-script using slurm - r

Question

I have thousands of data frames and I want to parallelise their analysis into slurm.
Here I am providing a simplified example:
I have an Rscript that I call: test.R
test.R contains these commands:
library(tidyverse)
df1 <- tibble(col1=c(1,2,3),col2=c(4,5,6))
df2 <- tibble(col1=c(7,8,9),col2=c(10,11,12))
files <- list(df1,df2)
for(i in 1:length(files)){
df3 <- as.data.frame(files[1]) %>%
summarise(across(everything(), list(mean=mean,sd=sd)))
write.table(df3, paste0("df",i))
}
Created on 2022-04-15 by the reprex package (v2.0.1)
I want to parallelise the for loop and analyse each data frame as a separate job.
Any help, guidance, and tutorials are appreciated.
Would the array command help?
#!/bin/bash
### Job name
#SBATCH --job-name=parallel
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --time=10:00:00
#SBATCH --mem=16G
#SBATCH --array=1-2
module load R/4.1.3
Rscript test.R $SLURM_ARRAY_TASK_ID

Answer 1

If the dataframes are stored in separate files with numerical identifiers, you can create a list of these files as a bash variable and submit the array of identifying numbers in your sbatch command. The $SLURM_ARRAY_TASK_ID can be used as an input argument for your R code to point to that specific dataframe's file.
Just for the purpose of the following example, say the dataframes are stored as .csv files with names dataframe_1.csv, dataframe_2.csv, ... dataframe_100. Your command to run the parallel jobs would be something like this:
sbatch -a 1-100 jobscript.sh
And jobscript.sh would resemble your sample code in your question:
#!/bin/bash
### Job name
#SBATCH --job-name=parallel
#SBATCH --nodes=1
#SBATCH --time=10:00:00
#SBATCH --mem=16G
module load R/4.1.3
Rscript test.R "dataframe_${SLURM_ARRAY_TASK_ID}.csv"
Note that you may need to break up these sbatch jobs to allow for processing thousands of dataframes. If a file name has four digits in its numeric ID, you can prepend the first digit before the ${SLURM_ARRAY_TASK_ID} in the last line of the jobscript.sh:
Rscript test.R "dataframe_1${SLURM_ARRAY_TASK_ID}.csv"
or if you want to avoid making multiple scripts, pass that number as an argument to jobscript.sh:
sbatch -a 1-100 jobscript.sh 1
#!/bin/bash
### Job name
#SBATCH --job-name=parallel
#SBATCH --nodes=1
#SBATCH --time=10:00:00
#SBATCH --mem=16G
prepend=$1
module load R/4.1.3
Rscript test.R "dataframe_${prepend}${SLURM_ARRAY_TASK_ID}.csv"