R package or technique to save progress in parallel computing job - r

Question

I am trying to schedule the following R script (exposure_train.R):
.libPaths("/gscratch/csde/mienkoja/rpackages")
library(Rmpi)
library(missForest)
library(snow)
library(doSNOW)
# load the data prepared above
load("/home/mienkoja/pse_rodis/dat_clean.rds")
# get rid of garbage from memory
gc()
# read node information from the system environment
nodefile <- Sys.getenv("PBS_NODEFILE")
# assign node information to a nodes object
nodes <- readLines(nodefile)
# create a cluster
cl <- makeMPIcluster(length(nodes), includemaster=TRUE)
# register the cluster
registerDoSNOW(cl)
rf_random <- missForest(dat_clean, maxiter = 10, ntree = 100, parallelize = "forests")
stopCluster(cl)
save(rf_random
,file = "/home/mienkoja/pse_rodis/rf_random_out.rds")
I am working on a cluster with several hundred nodes - each of which typically has 16 processor cores and 128GB of memory. All the nodes run CentOS 6 Linux, and they are tied together by the Moab cluster software. I am scheduling the job with a TORQUE scheduler using the following bash file
#!/bin/bash
### User specs
## Name the job "hyak_train"
#PBS -N hyak_train
## Request 16 CPUs (cores) on 2 nodes, 32 total cores
## If the job doesn't finish in 1 day, cancel it
#PBS -l nodes=2:ppn=16,pmem=2gb,feature=16core,walltime=24:00:00
## Put the output from jobs into the below directory
#PBS -o /gscratch/csde/mienkoja/exposure_train_outn_2p16
## Put both the stderr and stdout into a single file
#PBS -k oe
#PBS -j oe
#PBS -d /home/mienkoja/pse_rodis
## Send an email when the job is aborted, begins, or terminates
#PBS -m abe
### Standard specs
HYAK_NPE=$(wc -l < $PBS_NODEFILE)
HYAK_NNODES=$(uniq $PBS_NODEFILE | wc -l )
HYAK_TPN=$((HYAK_NPE/HYAK_NNODES))
NODEMEM=`grep MemTotal /proc/meminfo | awk '{print $2}'`
NODEFREE=$((NODEMEM-2097152))
MEMPERTASK=$((NODEFREE/HYAK_TPN))
ulimit -v $MEMPERTASK
export MX_RCACHE=0
### Modules
module load r_3.2.4
module load icc_14.0.3-ompi_1.8.3
### App
Rscript exposure_train.R
I am getting the following output using the script above:
Warning message:
package ‘Rmpi’ was built under R version 3.3.2
Loading required package: foreach
Loading required package: iterators
Warning messages:
1: package ‘doMPI’ was built under R version 3.3.2
2: package ‘foreach’ was built under R version 3.3.2
3: package ‘iterators’ was built under R version 3.3.2
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Warning message:
package ‘randomForest’ was built under R version 3.3.2
Attaching package: ‘snow’
The following object is masked from ‘package:doMPI’:
sinkWorkerOutput
Warning message:
package ‘snow’ was built under R version 3.3.2
Warning message:
package ‘doSNOW’ was built under R version 3.3.2
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 114354 6.2 350000 18.7 350000 18.7
Vcells 13378510 102.1 17463238 133.3 13379451 102.1
[n0729:14816] [[43070,1],0] FORKING HNP: orted --hnp --set-sid --report-uri 17 --singleton-died-pipe 18 -mca state_novm_select 1 -mca ess_base_jobid 2822635520
32 slaves are spawned successfully. 0 failed.
missForest iteration 1 in progress...Warning message:
package ‘Rmpi’ was built under R version 3.3.2
Loading required package: foreach
Loading required package: iterators
Warning messages:
1: package ‘doMPI’ was built under R version 3.3.2
2: package ‘foreach’ was built under R version 3.3.2
3: package ‘iterators’ was built under R version 3.3.2
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Warning message:
package ‘randomForest’ was built under R version 3.3.2
Attaching package: ‘snow’
The following object is masked from ‘package:doMPI’:
sinkWorkerOutput
Warning message:
package ‘snow’ was built under R version 3.3.2
Warning message:
package ‘doSNOW’ was built under R version 3.3.2
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 114354 6.2 350000 18.7 350000 18.7
Vcells 13378510 102.1 17463238 133.3 13379451 102.1
[n0829:32101] [[43995,1],0] FORKING HNP: orted --hnp --set-sid --report-uri 17 --singleton-died-pipe 18 -mca state_novm_select 1 -mca ess_base_jobid 2883256320
32 slaves are spawned successfully. 0 failed.
missForest iteration 1 in progress...Warning message:
package ‘Rmpi’ was built under R version 3.3.2
Loading required package: foreach
Loading required package: iterators
Warning messages:
1: package ‘doMPI’ was built under R version 3.3.2
2: package ‘foreach’ was built under R version 3.3.2
3: package ‘iterators’ was built under R version 3.3.2
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Warning message:
package ‘randomForest’ was built under R version 3.3.2
Attaching package: ‘snow’
The following object is masked from ‘package:doMPI’:
sinkWorkerOutput
Warning message:
package ‘snow’ was built under R version 3.3.2
Warning message:
package ‘doSNOW’ was built under R version 3.3.2
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 114354 6.2 350000 18.7 350000 18.7
Vcells 13378510 102.1 17463238 133.3 13379451 102.1
[n0829:32615] [[43481,1],0] FORKING HNP: orted --hnp --set-sid --report-uri 17 --singleton-died-pipe 18 -mca state_novm_select 1 -mca ess_base_jobid 2849570816
32 slaves are spawned successfully. 0 failed.
missForest iteration 1 in progress...Warning message:
package ‘Rmpi’ was built under R version 3.3.2
Loading required package: foreach
Loading required package: iterators
Warning messages:
1: package ‘doMPI’ was built under R version 3.3.2
2: package ‘foreach’ was built under R version 3.3.2
3: package ‘iterators’ was built under R version 3.3.2
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Warning message:
package ‘randomForest’ was built under R version 3.3.2
Attaching package: ‘snow’
The following object is masked from ‘package:doMPI’:
sinkWorkerOutput
Warning message:
package ‘snow’ was built under R version 3.3.2
Warning message:
package ‘doSNOW’ was built under R version 3.3.2
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 114354 6.2 350000 18.7 350000 18.7
Vcells 13378510 102.1 17463238 133.3 13379451 102.1
[n0768:11458] [[54933,1],0] FORKING HNP: orted --hnp --set-sid --report-uri 17 --singleton-died-pipe 18 -mca state_novm_select 1 -mca ess_base_jobid 3600089088
32 slaves are spawned successfully. 0 failed.
missForest iteration 1 in progress...done!
missForest iteration 2 in progress...Warning message:
package ‘Rmpi’ was built under R version 3.3.2
Loading required package: foreach
Loading required package: iterators
Warning messages:
1: package ‘doMPI’ was built under R version 3.3.2
2: package ‘foreach’ was built under R version 3.3.2
3: package ‘iterators’ was built under R version 3.3.2
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Warning message:
package ‘randomForest’ was built under R version 3.3.2
Attaching package: ‘snow’
The following object is masked from ‘package:doMPI’:
sinkWorkerOutput
....
This output repeats itself several times, but the point is that I get 22 successful iterations (i.e. missForest iteration 1 in progress...done!) over the course of the day that I have the job scheduled.
The problem is that I am scheduling the job in a backfill queue and the job gets preempted (at least) once every 4 hours. This means that I never get beyond the second iteration (i.e. missForest iteration 2 in progress...) because the job starts over every 4 hours.
While all iterations are not created equal, since I only want maxiter = 10 (see R script above) it seems that one day is enough time to run the job providing I can build some sort of checkpointing scheme (saving my progress) into the R script.
The source of the missForest function is available in the package repo. The relevant foreach loop (based on my parameterization of parallelize = "forests”) is pasted below.
foreach(xntree=idiv(ntree, chunks=getDoParWorkers()),
.combine='combine', .multicombine=TRUE,
.packages='randomForest') %dopar% {
randomForest(
x = obsX,
y = obsY,
ntree = xntree,
mtry = mtry,
replace = replace,
classwt = if (!is.null(classwt)) classwt[[varInd]] else
rep(1, nlevels(obsY)),
cutoff = if (!is.null(cutoff)) cutoff[[varInd]] else
rep(1/nlevels(obsY), nlevels(obsY)),
strata = if (!is.null(strata)) strata[[varInd]] else obsY,
sampsize = if (!is.null(sampsize)) sampsize[[varInd]] else
if (replace) nrow(obsX) else ceiling(0.632*nrow(obsX)),
nodesize = if (!is.null(nodesize)) nodesize[2] else 5,
maxnodes = if (!is.null(maxnodes)) maxnodes else NULL)
}
It seems like the solution is just to re-write the missForest function and write the relevant information from each iteration to an appropriately indexed file so that I can pick up where I left off on the next preemption.
QUESTION: Are there any R packages/techniques available to simplify the implementation of a checkpointing scheme like this?