specializing configuration with files instead of variables in Hydra config - fb-hydra

Question

I'd like to use specialized configuration as per Hydra documentation in Common Patterns -> Specializing Configuration. The difference is that my specialized configuration is in a file, not just one variable. In the example below I want to choose transform based on the model and the dataset. The configs for different transforms are in files. This would work if I specified all the transform configuration in dataset_model/cifar10_alexnet.yaml file, but that would defeat the purpose because I can't reuse the transform config in this case. Alsewhere in Hydra if you specify the name of the file it would automatically pick up the config in that file, but it does not seem to work in the specialized configuration.
I've modified the example in documentation as follows:
config.yaml:
defaults:
- dataset: cifar10
- model: alexnet
- transform: crop
- dataset_model: ${defaults.0.dataset}_${defaults.1.model}
optional: true
Added directory called transform and two files inside that directory:
crop.yaml:
# #package _group_
type: crop
test1: 7
resize.yaml:
# #package _group_
type: resize
test1: 50
changed file dataset_model/cifar10_alexnet.yaml:
# #package _global_
model:
num_layers: 5
transform: resize
Everything else is exactly as per the documentation. When I run this I get an exception:
Traceback (most recent call last):
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config
ret = OmegaConf.merge(cfg, loaded_cfg)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 321, in merge
target.merge_with(*others[1:])
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 331, in merge_with
self._format_and_raise(key=None, value=None, cause=e)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/base.py", line 101, in _format_and_raise
type_override=type_override,
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
_raise(ex, cause)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 610, in _raise
raise ex # set end OC_CAUSE=1 for full backtrace
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 329, in merge_with
self._merge_with(*others)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 347, in _merge_with
BaseContainer._map_merge(self, other)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 296, in _map_merge
dest.__setitem__(key, src_value)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 262, in __setitem__
self._format_and_raise(key=key, value=value, cause=e)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/base.py", line 101, in _format_and_raise
type_override=type_override,
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
_raise(ex, cause)
File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 610, in _raise
raise ex # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ValidationError:
full_key: transform
reference_type=Optional[Dict[Union[str, Enum], Any]]
object_type=dict
So, the question is - is this functionality supported and if it is, what am i doing wrong?

Answer 1

Your config is trying to merge the string "resize" into a dictionary like:
transform:
type: crop
test1: 7
This is not something you can do.
You are not explaining what you are trying to do very well, but my guess is that you want to compose a different transform based on selected dataset.
Hydra 1.1 will add support for recursive defaults list which will probably allow you to do what you want.
This is the doc for the new defaults list. You can install this version as a pre-release (see primary project readme).