R microbenchmark: How to pass same argument to evaluated functions? - r

Question

I'd like to evaluate the time to extract data from a raster time series using different file types (geotiff, binary) or objects (RasterBrick, RasterStack). I created a function that will extract the time series from a random point of the raster object and I then use microbenchmark to test it.
Ex.:
# read a random point from a raster stack
sample_raster <- function(stack) {
poi <- sample(ncell(stack), 1)
raster::extract(stack, poi)
}
# opening the data using different methods
data_stack <- stack(list.files(pattern = '3B.*tif'))
data_brick <- brick('gpm_multiband.tif')
bench <- microbenchmark(
sample_stack = sample_raster(data_stack),
sample_brick = sample_raster(data_brick),
times = 10
)
boxplot(bench)
# this fails because sampled point is different
bench <- microbenchmark(
sample_stack = sample_raster(data_stack),
sample_brick = sample_raster(data_brick),
times = 10,
check = 'equal'
)
I included a sample of my dataset here
With this I can see that sampling on RasterBrick is faster than stacks (R Raster manual also says so -- good). The problem is that I'm sampling at different points at each evaluated expression. So I can't check if the results are the same. What I'd like to do is sample at the same location (poi) on both objects. But have the location be different for each iteration. I tried to use the setup option in microbenchmark but from what I figured out, the setup is evaluated before each function is timed, not once per iteration. So generating a random poi using the setup will not work.
Is it possible to pass the same argument to the functions being evaluated in microbenchmark?
Result
Solution using microbenchmark
As suggested (and explained bellow), I tried the bench package with the press call. But for some reason it was slower than setting the same seed at each microbenchmark iteration, as suggested by mnist. So I ended up going back to microbenchmark. This is the code I'm using:
library(microbenchmark)
library(raster)
annual_brick <- raster::brick('data/gpm_tif_annual/gpm_2016.tif')
annual_stack <- raster::stack('data/gpm_tif_annual/gpm_2016.tif')
x <- 0
y <- 0
bm <- microbenchmark(
ext = {
x <- x + 1
set.seed(x)
poi = sample(raster_size, 1)
raster::extract(annual_brick, poi)
},
slc = {
y <- y + 1
set.seed(y)
poi = sample(raster_size, 1)
raster::extract(annual_stack, poi)
},
check = 'equal'
)
Solution using bench::press
For completeness sake, this was how I did, using the bench::press. In the process, I also separated the code for selecting the random cell from the point sampling function. So I can time only the point sampling part of the code. Here is how I'm doing it:
library(bench)
library(raster)
annual_brick <- raster::brick('data/gpm_tif_annual/gpm_2016.tif')
annual_stack <- raster::stack('data/gpm_tif_annual/gpm_2016.tif')
bm <- bench::press(
pois = sample(ncell(annual_brick), 10),
mark(
iterations = 1,
sample_brick = raster::extract(annual_brick, pois),
sample_stack = raster::extract(annual_stack, pois)
)
)

Answer 1

My approach would be to set the same seats for each option in microbenachmark but change them prior to each function call. See the output and how the same seats are used for both calls eventually
x <- 0
y <- 0
microbenchmark::microbenchmark(
"checasdk" = {
# increase seat value by 1
x <- x + 1
print(paste("1", x))
set.seed(x)},
"check2" = {
y <- y + 1
print(paste("2", y))
set.seed(y)
}
)

Answer 2

If I understand correctly, the OP has two requirements:
The same data points should be sampled when timing the two expressions in order to check the results are identical.
In addition, timing of the two expressions is to be repeated for different data points sampled.
Using the same random numbers
As suggested by Roman, set.seed() can be used to set the seed values for R's random number generator. If the same parameter is used, the sequence of generated random numbers will be the same.
sample_raster() can be modified to ensure that the random number generator will be initiliased for each call.
sample_raster <- function(stack) {
set.seed(1L)
poi <- sample(ncell(stack), 1)
raster::extract(stack, poi)
}
This will met requirement 1 but not requirement 2 as the same data samples will be used for all repetitions.
Different random numbers in repetitions
The OP has asked:
Is it possible to pass the same argument to the functions being
evaluated in microbenchmark?
One possibility is to use for or lapply() to loop over a sequence of seed values as suggested in answers to a similar question.
In this case, I suggest to use the bench package for benchmarking. It has a press() function which runs bench::mark() across a grid of parameters.
For this, sample_raster() gets a second parameter:
sample_raster <- function(stack, seed) {
set.seed(seed)
poi <- sample(ncell(stack), 1L)
# cat(substitute(f), s, poi, "\n") # just to check, NOT to use for timings
raster::extract(stack, poi)
}
The timings are executed for different seeds as given in vector seed_vec.
library(bench)
bm <- press(
seed_vec = 1:10,
mark(
iterations = 1L,
sample_stack = sample_raster(data_stack, seed_vec),
sample_brick = sample_raster(data_brick, seed_vec)
)
)
Note that the length of seed_vec determines the number of repetitions with different poi, now. The iterations parameter to mark() specifies how often the timings are to be repeated for the same seed / poi.
The results can be plotted using
library(ggplot2)
autoplot(bm)
or summarized using
library(dplyr)
bm %>%
group_by(expression = expression %>% as.character()) %>%
summarise(median = median(median), n_itr = n())