I've been trying to make this work for a while, without much luck:
file_show:
path: /{user}/file/{group}/{file}
controller: Acme\Controller\File::show
requirements:
group: .+
file: .+
# /john/file/acme/group/test/file.zip
# user: john
# group: acme/group
# file: test/file.zip
Does the Symfony Router support multiple params with multiple slashes like this?
No symfony does not support multiple params with slashes straight after each other as ist can not know where one parameter ends and the next one begins.
# /john/file/acme/group/test/file.zip
# user: john
# group: acme/group
# file: test/file.zip
could also be
# /john/file/acme/group/test/file.zip
# user: john
# group: acme
# file: group/test/file.zip
You can work around this by using a different sign between the params and not allow that sign inside the params. docs
Related
Here is my current config structure
hydra/
pipeline/
common/
feature.yaml
stage/
train.yaml
with the following files:
train.yaml
# #package _global_
defaults:
- _self_
- ../pipeline/common#train: feature
- ../pipeline/common#val: feature
train:
conf:
split: train
val:
conf:
split: val
pipeline:
- ${oc.dict.values: train.steps}
- ${oc.dict.values: val.steps}
feature.yaml
conf:
split: train
steps:
tabular:
name: "${conf.split}-tabular
class: FeatureGeneration
dataset:
datasources: [ "${conf.split}_split" ]
What I've accomplished:
I've been able to figure out how to use the config group multiple times utilizing the defaults in train.yaml.
What I'm stuck on:
I'm getting an error: InterpolationKeyError 'conf.split' not found
I do realize that imports are absolute. If I put #package common.feature at the beginning of feature.yaml I can import conf.split via common.feature.conf.split, but is there not a cleaner way? I tried relative imports but got the same error.
I can't seem to override conf.split from train.yaml. You can see where I set train.conf.split and val.conf.split but these do not get propagated. What I need to be able to do is have each instance of the config group utilize a different conf.split value. This is the biggest issue I'm facing.
What I've referenced so far:
The following resources have gotten me to where I am so far, but am still having trouble with what's listed above.
Hydra : how to assign config files from same group to two different fields
https://hydra.cc/docs/advanced/overriding_packages/
https://hydra.cc/docs/patterns/extending_configs/
Interpolation is not import and it's evaluated at when you access the config node. At that point your config is already composed so it should be straight forward to use either absolute interpolation (the default) or relative based on the structure of your final config.
Hard to be 100% sure, but I suspect this problem is because your defaults list has _self_ at the beginning. This means that the content of the config with containing the defaults list is overridden by what comes after in the defaults list.
Try to move _self_ to the end:
# #package _global_
defaults:
- ../pipeline/common#train: feature
- ../pipeline/common#val: feature
- _self_
#...
I have some config file, model/foo.yaml:
# #package _global_
# foo.yaml
MODEL:
BACKBONE:
OUT_FEATURES: [c4, c5]
HEAD:
IN_FEATURES: ${MODEL.BACKBONE.OUT_FEATURES}
There are no issues with variable interpolation when I point to this config in the defaults-list of another config, eg buzz.yaml, except when I also override the package like so:
# buzz.yaml
defaults:
- model#foo_head: foo
Attempting to compose buzz.yaml, you will get an error like:
omegaconf.errors.InterpolationKeyError: Interpolation key 'MODEL.BACKBONE.OUT_FEATURES' not found
Can variable interpolation not be used in configs when packaging?
Yes. OmegaConf supports relative interpolation.
MODEL:
BACKBONE:
OUT_FEATURES: [c4, c5]
HEAD:
IN_FEATURES: ${..BACKBONE.OUT_FEATURES}
I strongly recommend that you read the docs of OmegaConf.
I would like to get a list of configs as a (default) value entry
and use a structured schema to validate the input list.
E.g., in trainer.yaml:
defaults:
- callbacks:
- checkpointer
- early_stopping
In callbacks/checkpointer.yaml and callbacks/early_stopping.yaml I have a link to appropriate structured configs as default values, e.g.:
# callbacks/checkpointer.yaml
defaults:
- /trainer_lib/callbacks/base_checkpointer#_here_
The structured schema:
#dataclass
class CheckpointerConfig:
_target_: str = "some_library_class"
data_dir: str = "folder"
#dataclass
class TrainerConfig:
callbacks: List[Any] = MISSING
and config store:
cs = ConfigStore.instance()
cs.store(group="trainer_lib/callbacks", name="base_checkpointer", node=CheckpointerConfig)
I am not sure what is the correct syntax (what I tried fails) to accomplish this. I get an omegaconf.errors.ConfigTypeError: Cannot merge DictConfig with ListConfig.
Is there a way to accomplish this? Thanks.
Discussion on this topic in this Hydra issue.
Are you on Hydra 1.0? This is actually supported in Hydra 1.1. Here is the documentation: https://hydra.cc/docs/next/patterns/select_multiple_configs_from_config_group
Using salt i want to find the attribute(key) and replace it with value based on specific stanza. The attribute(key) is present in multiple times in a file under different stanzas. I want to find my attribute under specific stanza and replace with value.
Example:
output.kafka:
# Boolean flag to enable or disable the output module.
enabled:
I need to find enabled: under output.kafka: and replace it with value. The enabled: attribute present multiple times in my file.
Thanks
Bala.
Salt has a few commands like file.line, file.replace and file.blockreplace that can modify an existing file, but I highly recommend managing the whole file using file.managed. It makes for a less brittle experience.
Here's an example based off your question:
Pillar top file:
cat /srv/pillar/top.sls
base:
'*':
- common
'minion01':
- minion01kafkasettings
Set our pillar data:
cat /srv/pillar/minion01kafkasettings.sls
kafka_output: True
Here's our filebeat template:
cat /srv/salt/filebeat.tmpl
output.kafka:
# Boolean flag to enable or disable the output module.
enabled: {{ pillar.get('kafka_output', True) }}
Here's the filebeat Salt sls file:
cat /srv/salt/filebeat.sls
the_filebeat_file:
file.managed:
- name: /etc/filebeat/filebeat.yml
- template: jinja
- user: root
- group: root
Then we can run the following:
Refresh our pillar data
salt 'minion01' saltutil.refresh_pillar
Then apply the sls file:
salt 'minion01' state.sls filebeat
I have another theory using file.seralize that might work but not in its current state, Maybe Dave could help.
{% set json_data = salt.cp.get_file_str('/etc/filebeat/filebeat.yml') | load_yaml %}
{% do json_data.update({'enabled': pillar.get('kafka_output', True) }) %}
update_config:
file.serialize:
- name: /etc/filebeat/filebeat.yml
- user: root
- group: root
- mode: 644
- formatter: yaml
- dataset: |
{{ json_data | yaml(False)| indent(8) }}
This state should load the whole configuration file then you can modify any of its values based on your pillar setting using the do statement in your case it could be
{% do json_data.update({'enabled': pillar.get('kafka_output', True) }) %}
The config file is populated but not as exepcted as the result will be as following:
'enabled: true
status: active
'
Note there are quotes and the yaml is not intended correctly, is there another way to make it work ? I will update this answer if I found any new results
I have a yaml file like this:
local: &local
image: xxx
# *tons of config*
ci:
<<: *local
image: # delete
build: .
I want ci to inherit all values from local, except the image.
Is there a way to "delete" this value?
No there isn't a way to mark a key for deletion in a YAML file. You can only overwrite existing values.
And the latter is what you do, you associate the empty scalar as value to the key image as if you would have written:
image: null # delete
There are two things you can do: post-process or make a base mapping in your YAML file.
If you want to post-process, you associate a special unique value to image, or a specially tagged object, and after loading recursively walk over the tree to remove key-value pairs with this special value. Whether you can already do this during parsing, using hooks or overwriting some of its methods, depends on the parser.
Using a base mapping requires less work, but is more intrusive to the YAML file:
localbase: &lb
# *tons of config*
local: &local
image: xxx
ci:
<<: *lb
build: .
If you do the former you should note that if you use a parsers that preserve the "merge-hierarchy" on round-tripping (like my ruamel.yaml parser can do) it is not enough to delete the key-value pair, in that case the original from local would come back. Other parsers that simply resolve this at load time don't have this issue.
For properties that accept a list of values, you can send [] as value.
For example in docker-compose you don't want to inherit ports:
service_1: &service_1
# some other properties.
ports:
- "49281:22"
- "8876:8000"
# some other properties
image: some_image:latest
service_2:
<<: *service_1
ports: [] # it removes ports values.
image: null # it removes image value.