Updater does not work with dictionary of optimizers - dictionary

Question

I have been trying to modify the cifer example code of Chainer so that it works with multiple optimizers.
As written in page 946 of Chainer Documentation (Release 7.0.0b3), I thought that the function, training.updaters.ParallelUpdater, can take not only an optimizer object but also a dictionary that maps strings to optimizers.
So, I put three optimizer objects in a dictionary, setup them with a model, and put the dictionary into the ParallelUpdate function as an argument. (see the code below)
optimizers = {}
optimizers['SGD'] = chainer.optimizers.SGD(args.learnrate)
optimizers['Adam'] = chainer.optimizers.Adam()
optimizers['RMSProp'] = chainer.optimizers.RMSprop()
.
.
for key in optimizers.keys() :
optimizers[key].setup(model)
.
.
updater = training.updaters.ParallelUpdater(
train_iter, optimizer=optimizers, devices = devices
)
However, the program gives an error message as follows.
File "C:\WPy64-3720\python-3.7.2.amd64\lib\site-packages\chainer\training\updaters\parallel_updater.py", line 82, in __init__
models = {'main': optimizer.target}
AttributeError: 'dict' object has no attribute 'target'
I understand that 'dict' object is not an optimizer object, but I thought that the description in the document implies that the optimizers are automatically exracted from the dictionary.
Does anybody know what is wrong with my code or how to make the updater work with multiple optimizers?

Answer 1

To use multiple optimizers, you need to override the update_core method and provide custom logic to update with these optimizers. The feature of accepting a dict of optimizers is designed for optimizing multiple different objectives with separate optimizers (e.g. GANs) rather than applying different update rules for parameters of a single objective.
Note that it means you eventually need to implement the parallel update by yourself because ParallelUpdater.update_core, which you need to override, actually implement parallelization. I strongly recommend ChainerMN, which cleanly decouple the optimization logic and parallel training.