I'm attempting to provide an environment variable in a config YAML file as such:
resources:
be_warehouse:
config:
conn_str:
env: DB_CONN_STR
analytics_warehouse:
config:
conn_str:
env: WH_DB_CONN_STR
but I am receiving the following error:
Invalid scalar at path root:resources:analytics_warehouse:config:conn_str. Value "{'env': 'WH_DB_CONN_STR'}" of type "<class 'dict'>" is not valid for expected type "String".
I have seen this syntax used in this official example. Am I missing something obvious?
The env: ENV_VAR support is available for config schema which is typed as StringSource. If these are #resources you are creating you just need to declare config_schema={'conn_str': StringSource} instead of just using str.
https://docs.dagster.io/_apidocs/config#dagster.StringSource
Related
So, I have a hydra model config (autoencoder.yaml) defined as:
_target_: student.models.AutoEncoder
defaults:
- dataloader: msa
- encoder: default_encoder
- decoder: vae_decoder
- scheduler: null
- optimizer: adam
- preprocessing: null
batch_size: 256
Now in the encoder folder, I have the following YAML configs:
- default_encoder.yaml
- vae_encoder.yaml
and my base config file is as:
# #package _global_
defaults:
- _self_
- model: autoencoder.yaml
# enable color logging
- override hydra/hydra_logging: colorlog
- override hydra/job_logging: colorlog
work_dir: ${hydra:runtime.cwd}
seed: null
name: "default"
Now, I can call this as is with:
python myapp.py seed=42 #works
but when I do something like:
python myapp.py ++model.encoder=vae_encoder
It comes with the error:
Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance
How can I just replace the underlying object through composition with hydra? Basically, when I do this or ++model.encoder=vae_encoder, it replaces this as a string rather than referring to the yaml file
When modifying the defaults list, use a slash ('/') instead of a period ('.') as the separator for path components:
python myapp.py model/encoder=vae_encoder
instead of
python myapp.py model.encoder=vae_encoder
Suppose I have FOO=dGVzdA== in my .env file and then I try to load this binary environment variable in my YAML configuration.
foo: !!binary '%env(FOO)%'
This errors out because it tries to decode %env(FOO)% verbatim as if it were base64-encoded. That is, it does not substitute the environment variable when prefixed with !!binary. So then, how does one actually use a binary environment variable?
It seems the correct way to express this is:
foo: '%env(base64:FOO)%'
My .env file has the following entries, but FOO is not listed when I run bin/console debug:container --env-vars. Note however that $_ENV['FOO'] exists when I dump the variable.
FOO=1
AUTH0_CLIENT_ID=clientid
AUTH0_CLIENT_SECRET=secret
AUTH0_DOMAIN=myapp.us.auth0.com
What determines if an env var defined in .env will be available in the container?
Not really sure if this is worthy of an answer but I suppose it might help.
The Symfony .env files are really just one possible sources of $_ENV variables. There are lots of other env variables floating around and of course in production, you might not use .env at all.
So rather than save access to all env variables, the Symfony configuration system only saves those that are actually used. So in this case:
# config/services.yaml
parameters:
foo: '%env(resolve:FOO)%'
Will result in:
bin/console debug:container --env-vars
APP_SECRET n/a "84dc6de50e6f2f7af3db3f78f886840f"
DATABASE_URL n/a "mysql://db_user:db_password#127.0.0.1:3306/db_name?serverVersion=5.7"
FOO n/a "1"
MAILER_DSN n/a n/a
VAR_DUMPER_SERVER "127.0.0.1:9912" n/a
For a fresh 5.1 project.
Off-topic but vaguely interesting to me at least, the above command also generates a warning
[WARNING] The following variables are missing:
* MAILER_DSN
MAILER_DSN is commented out in the default .env file. So I guess it is possible to use env values during configuration even if none are defined at compile time. Good way to check for spelling errors I guess.
apparently, Symfony cast env variable is not working on symfony 4.0.
I have this configuration:
cache:
session:
enabled: "%env(bool:SESSION_CACHE_ENABLED)%"
But I get this error:
Invalid type for path "cache.session.enabled". Expected boolean, but got string.
What is my problem? I'm using symfony version 4.0
Thanks
EDIT
Probably, it is a problem of the plugins. That's what I think: Symfony 4 is now based on .env config variable, that are STRING as default; to handle this, S4 is able to use "casting" env var
'%env(bool:myvar)%'
And it works; if you do a var_dump within a controller, you can see that the variable is a boolean.
Most of current plugins, also those who supports S4, are not able to use this syntax, so, they see that variable as STRING, and the validator return an error.
These plugins should be fixed or, actually, I can duplicate the .yml file on each package/{env}/ dir with separated configuration ( the situation that I would avoid with .env )
The problem is fixed in symfony 4.1 https://github.com/symfony/symfony/issues/22151
I'm deploying a Symfony 2.8 application to Elastic Beanstalk and the DB parameters are read from ENV. The EB already has the RDS env set, like RDS_HOSTNAME, RDS_DB_NAME etc, but Symfony needs to read these parameters from env variables prefixed with SYMFONY__. Is there a way to map existing RDS_ variables to the SYMFONY__ ones? I tried with an env.config file placed in the .ebextensions dir, with the following content (sample):
option_settings:
- namespace: aws:elasticbeanstalk:application:environment
option_name: SYMFONY__ENV__DATABASE__HOST
value: "$RDS_HOSTNAME"
I know I could duplicate the values in the env.config, but I rather not.
Thanks!
You could try Fn::GetOptionSetting:
option_settings:
- namespace: aws:elasticbeanstalk:application:environment
option_name: SYMFONY__ENV__DATABASE__HOST
value:
"Fn::GetOptionSetting":
Namespace: "aws:elasticbeanstalk:application:environment"
OptionName: "RDS_HOSTNAME"
DefaultValue: ""
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions-functions.html#ebextensions-functions-getoptionsetting