Passing multiple config groups - fb-hydra

Question

In my config.yaml, how to pass two datasets for e.g. cifar and cinic at once? Can I pass a multiple config groups to the defaults list?
This is for the case when I want to train my model on a mix of datasets, but I do not want to create a config group for every possible combination.
├── config.yaml
└── dataset
├── cifar.yaml
└── imagenet.yaml
└── cinic.yaml
What I tried is as follows:
dataset:
- cifar
- cinic
which resulted in following error:
Could not load train_dataset/['cifar', 'cinic']. Available options 'cifar ... '

Answer 1

Currently config groups are mutually exclusive.
Support for this is planned for Hydra 1.1. See issue 499.
One possible workaround is to put everything in the config and to use interpolation:
all_datasets:
imagenet:
name: imagenet
cifar10:
name: cifar10
datasets:
- ${all_datasets.imagenet}
- ${all_datasets.cifar10}
This way you can override dataset to a different list of datasets from the command line (with the interpolation).
If you want to simplify the usage at the expense of some additional code, you can do something like:
all_datasets:
...
datasets_list:
- imagenet
- cifar10
datasets: []
#hydra.main(config_path="conf", config_name="config")
def my_app(cfg: DictConfig) -> None:
for ds in cfg.datasets_list:
cfg.datasets.append(cfg.all_datasets[ds])
if __name__ == "__main__":
my_app()
I didn't test this but I hope you get the idea.