Using a custom R generator function with fit_generator (Keras, R) - r

Question

I'd like to train a convolutional network to solve a multi-class, multi-label problem on image data. Due to the nature of the data, and for reasons I'll spare you, it would be best if I could use a custom R generator function to feed to the fit_generator command, instead of its built-in image_data_generator and flow_images_from_directory commands (which I was successfully able to get working, just not for this particular problem).
Here (https://www.rdocumentation.org/packages/keras/versions/2.2.0/topics/fit_generator) it says that I can do just that, without giving any examples. So I tried the following. Here is an extremely stripped down example of what I'm trying to do (this code is entirely self contained):
library(keras)
library(reticulate) #for py_iterator function
play.network = keras_model_sequential() %>%
layer_dense(units = 10, activation = "relu", input_shape = c(10)) %>%
layer_dense(units = 1, activation = "relu")
play.network %>% compile(
optimizer = "rmsprop",
loss = "mse"
)
mikes.custom.generator.function = function() #generates a 2-list of a random 1 x 10 array, and a scalar
{
new.func = function()
{
arr = array(dim = c(1,10))
arr[,] = sample(1:10, 10, replace = TRUE)/10
return(list(arr,runif(1)))
}
}
mikes.custom.iterator = py_iterator(mikes.custom.generator.function()) #creates a python iterator object
generator_next(mikes.custom.iterator) #correctly returns a 2-member list consisting of a 1 x 10 array, and a scalar
generator_next(mikes.custom.iterator)[[1]] #a 1 x 10 array
generator_next(mikes.custom.iterator)[[2]] #a scalar
#try to fit with "fit_generator":
play.network %>% fit_generator( #FREEZES.
mikes.custom.iterator,
steps_per_epoch = 1,
epochs = 1
)
The thing freezes at training time, without giving me an error message or anything. I also tried it with a custom image data generator for my original problem, same result.
Note that this network trains just fine if I just use fit and input the training data manually:
play.network %>% fit(generator_next(mikes.custom.iterator)[[1]],generator_next(mikes.custom.iterator)[[2]], epochs = 1, batch_size = 1)
#trains just fine
I think I know the problem, but I don't know the solution. If you ask it for the class of my custom iterator, it gives
class(mikes.custom.iterator)
[1] "python.builtin.iterator" "rpytools.generator.RGenerator" "python.builtin.object"
whereas if I build an iterator using the builtin image_data_generator and flow_images_from_directory commands, it gives
train_datagen <- image_data_generator(rescale = 1/255)
class(train_datagen)
[1] "keras.preprocessing.image.ImageDataGenerator" "keras_preprocessing.image.ImageDataGenerator" "python.builtin.object"
train_generator <- flow_images_from_directory(
train_dir,
train_datagen,
....
)
class(train_generator)
[1] "python.builtin.iterator" "keras_preprocessing.image.DirectoryIterator" "keras_preprocessing.image.Iterator" "tensorflow.python.keras.utils.data_utils.Sequence" "python.builtin.object"
So my guess is that train_datagen and/or train_generator have attributes that mikes.custom.iterator does not, and fit_generator is trying to call upon mikes.custom.iterator using functions other than the basic generator_next (which is in theory all it should really need). But I don't know what they may be, or how to build mikes.custom.iterator correctly, even after searching for two hours online.
Help anyone?

Answer 1

In R, you can build an iterator using <<- operator. This is very helpful to build a custom generator function; and it is compatible with Keras' fit_generator() function.
Some minimal example:
# example data
data <- data.frame(
x = runif(80),
y = runif(80),
z = runif(80)
)
# example generator
data_generator <- function(data, x, y, batch_size) {
# start iterator
i <- 1
# return an iterator function
function() {
# reset iterator if already seen all data
if ((i + batch_size - 1) > nrow(data)) i <<- 1
# iterate current batch's rows
rows <- c(i:min(i + batch_size - 1, nrow(data)))
# update to next iteration
i <<- i + batch_size
# create container arrays
x_array <- array(0, dim = c(length(rows), length(x)))
y_array <- array(0, dim = c(length(rows), length(y)))
# fill the container
x_array[1:length(rows), ] <- data[rows, x]
y_array[1:length(rows), ] <- data[rows, y]
# return the batch
list(x_array, y_array)
}
}
# set-up a generator
gen <- data_generator(
data = data.matrix(data),
x = 1:2, # it is flexible, you can use the column numbers,
y = c("y", "z"), # or the column name
batch_size = 32
)
From above function, you can simply check the resulting arrays by calling the generator:
gen()
Or you could also test the generator using a simple Keras model:
# import keras
library(keras)
# set up a simple keras model
model <- keras_model_sequential() %>%
layer_dense(32, input_shape = c(2)) %>%
layer_dense(2)
model %>% compile(
optimizer = "rmsprop",
loss = "mse"
)
# fit using generator
model %>% fit_generator(
generator = gen,
steps_per_epoch = 100, # will auto-reset after see all sample
epochs = 10
)
I have to admit that the process is a little bit complex and requires extensive programming. You should check this featured blog post by François Chollet himself, or kerasgenerator package that I develop personally.

Answer 2

sampling_generator <- function(X_data, Y_data, batch_size) {
function() {
rows <- sample(1:nrow(X_data), batch_size, replace = TRUE)
list(X_data[rows,], Y_data[rows,])
}
}
model %>%
fit_generator(sampling_generator(X_train, Y_train, batch_size = 128),
steps_per_epoch = nrow(X_train) / 128, epochs = 10)
I found this answer in R keras FAQs which seems to work
https://keras.rstudio.com/articles/faq.html#how-can-i-use-keras-with-datasets-that-dont-fit-in-memory