extend list from default config in Hydra - fb-hydra

I have a list that is defined in my defaults
configuration file base_list:
list:
- 1
- 2
I know I can override the list values in the config file:
defaults:
- base_list
list:
- 3
- 4
which results
list:
- 3
- 4
However, I look for a way to extend the list, and the desired output is:
list:
- 1
- 2
- 3
- 4
Any idea how to do this?

This is not supported directly.
However, you can achieve the desired behavior using the built-in OmegaConf resolvers oc.dict.*.
Those resolvers allow you to access the keys or values of a config node as a list:
cfg = OmegaConf.create(
{
"workers": {
"node3": "10.0.0.2",
"node7": "10.0.0.9",
},
"nodes": "${oc.dict.keys: workers}",
"ips": "${oc.dict.values: workers}",
}
)
# Keys are copied from the DictConfig:
show(cfg.nodes)
# -> type: ListConfig, value: ['node3', 'node7']
# Values are dynamically fetched through interpolations:
show(cfg.ips)
# -> type: ListConfig, value: ['${workers.node3}', '${workers.node7}']
assert cfg.ips == ["10.0.0.2", "10.0.0.9"]
With this, you can compose a dictionary and have a node that access the values or keys as if they are a list.

Related

HOT template for cinder volume with or without volume_type

I am trying to write a HOT template for Openstack volume, and need to have the volume_type as a parameter. I also need to support a case when the parameter is not given, and default to the Cinder default volume type.
First attempt was to pass null to the volume_type , hoping it would give the default volume type. However no matter what I pass (null, ~, default, "" ) , seems there is no way to get the default volume type.
type: OS::Cinder::Volume
properties:
name: test
size: 1
volume_type: { if: ["voltype_given" , {get_param:[typename]} , null] }
Is there any way to get the default volume type , when you have the "volume_type" property defined?
Alternatively, is there any way to have the "volume_type" property itself behind a conditional? I tried several ways, but no luck. Something like:
type: OS::Cinder::Volume
properties:
if: ["voltype_given" , [ volume_type: {get_param:[typename]} ] , ""]
name: test
size: 1
ERROR: TypeError: : resources.kk-test-vol: : 'If' object is not iterable
Could you do something like this?
---
parameters:
typename:
type: string
conditions:
use_default_type: {equals: [{get_param: typename}, '']}
resources:
MyVolumeWithDefault:
condition: use_default_type
type: OS::Cinder::Volume
properties:
name: test
size: 1
MyVolumeWithExplicit:
condition: {not: use_default_type}
type: OS::Cinder::Volume
properties:
name: test
size: 1
volume_type: {get_param: typename}
# e.g. if you need to refer to the volume from another resource
MyVolumeAttachment:
type: OS::Cinder::VolumeAttachment
properties:
instance_uid: some-instance-uuid
volume_id:
if:
- use_default_type
- get_resource: MyVolumeWithDefault
- get_resource: MyVolumeWithExplicit

Select version string from JSON array and increment it by one using jq

Bash script find a a tags in ECR repo:
aws ecr describe-images --repository-name laplacelab-backend-repo
\ --query 'sort_by(imageDetails,& imagePushedAt)[*]'
\--output json | jq -r '.[].imageTags'
Output:
[
"v1",
"sometag",
...
]
How I can extract the version number? v<number> can contain the only version tag. I need to get a number and increment version for the set to var. If output of sort_by(imageDetails,& imagePushedAt)[*] is empty JSON arr instead
[
{
"registryId": "057296704062",
"repositoryName": "laplacelab-backend-repo",
"imageDigest": "sha256:c14685cf0be7bf7ab1b42f529ca13fe2e9ce00030427d8122928bf2d46063bb7",
"imageTags": [
"v1"
],
"imageSizeInBytes": 351676915,
"imagePushedAt": 1593514683.0
}
]
Set 2
No one repo sort_by(imageDetails,& imagePushedAt)[*] return [] set 1.
As a result, I try to get var VERSION with next version for an update or 1 if the repo is empty.
You could use the select() function on the imageTags array and get only the tag starting with v and increment it.
jq '( .[].imageTags[] | select(startswith("v")) | ltrimstr("v") | tonumber | .+1 ) // 1'
For other cases like the tags array being empty or containing null strings (error case), the value defaults to 1
For storing into the variable e.g. say version (avoid using uppercase variable names from a user scripts), use command substitution. See How do I set a variable to the output of a command in Bash?
version=$( <your-pipeline> )
Note: This does not work well with version strings following Semantic versioning RFC, e.g. as v1.2.1 as jq does not have a library to parse them.

Parsing JSON dict of CloudFormation parameters for '--parameter-overrides'

I'm using AWS CloudFormation at the moment, and I need to parse out parameters due to differences between stack creation and deployment. Command aws cloudformation create accepts a JSON file, but aws cloudformation deploy only accepts inlined application parameters of Key=Value type.
I have this JSON file:
[
{
"ParameterKey": "EC2KeyPair",
"ParameterValue": "$YOUR_EC2_KEY_PAIR"
},
{
"ParameterKey": "SSHLocation",
"ParameterValue": "$YOUR_SSH_LOCATION"
},
{
"ParameterKey": "DjangoEnvVarDebug",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DEBUG"
},
{
"ParameterKey": "DjangoEnvVarSecretKey",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_SECRET_KEY"
},
{
"ParameterKey": "DjangoEnvVarDBName",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_NAME"
},
{
"ParameterKey": "DjangoEnvVarDBUser",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_USER"
},
{
"ParameterKey": "DjangoEnvVarDBPassword",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_PASSWORD"
},
{
"ParameterKey": "DjangoEnvVarDBHost",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_HOST"
}
]
And I want to turn it into this:
'EC2KeyPair=$YOUR_EC2_KEY_PAIR SSHLocation=$YOUR_SSH_LOCATION DjangoEnvVarDebug=$YOUR_DJANGO_ENV_VAR_DEBU
G DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY DjangoEnvVarDBName=$YOUR_DJANGO_ENV_VAR_DB_NAME D
jangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER DjangoEnvVarDBPassword=$YOUR_DJANGO_ENV_VAR_DB_PASSWORD Dj
angoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST'
This would be the equivalent Python code:
thing = json.load(open('stack-params.example.json', 'r'))
convert = lambda item: f'{item["ParameterKey"]}={item["ParameterValue"]}'
>>> print(list(map(convert, thing)))
['EC2KeyPair=$YOUR_EC2_KEY_PAIR', 'SSHLocation=$YOUR_SSH_LOCATION', 'DjangoEnvVarDebug=$YOUR_DJANGO_ENV_V
AR_DEBUG', 'DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY', 'DjangoEnvVarDBName=$YOUR_DJANGO_ENV_
VAR_DB_NAME', 'DjangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER', 'DjangoEnvVarDBPassword=$YOUR_DJANGO_EN$
_VAR_DB_PASSWORD', 'DjangoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST']
>>> ' '.join(map(convert, thing))
'EC2KeyPair=$YOUR_EC2_KEY_PAIR SSHLocation=$YOUR_SSH_LOCATION DjangoEnvVarDebug=$YOUR_DJANGO_ENV_VAR_DEBU
G DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY DjangoEnvVarDBName=$YOUR_DJANGO_ENV_VAR_DB_NAME D
jangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER DjangoEnvVarDBPassword=$YOUR_DJANGO_ENV_VAR_DB_PASSWORD Dj
angoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST'
I have this little snippet:
$ cat stack-params.example.json | jq '.[] | "\(.ParameterKey)=\(.ParameterValue)"'
"EC2KeyPair=$YOUR_EC2_KEY_PAIR"
"SSHLocation=$YOUR_SSH_LOCATION"
"DjangoEnvVarDebug=$YOUR_DJANGO_ENV_VAR_DEBUG"
"DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY"
"DjangoEnvVarDBName=$YOUR_DJANGO_ENV_VAR_DB_NAME"
"DjangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER"
"DjangoEnvVarDBPassword=$YOUR_DJANGO_ENV_VAR_DB_PASSWORD"
"DjangoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST"
But I'm not sure how to join the strings together. I was looking at reduce but I think it only works on lists, and streams of strings aren't lists. So I'm thinking the correct approach is to convert the key : value association into 'key=value' strings within the list, then join altogether, though I have trouble working with the regex. Does anybody have any tips?
The goal as exemplified by the illustrative output seems highly dubious, but it can easily be achieved using the -r command-line option together with the filter:
map("\(.ParameterKey)=\(.ParameterValue)") | "'" + join(" ") + "'"
Footnote
I was looking at reduce but I think it only works on lists, and streams of strings aren't lists.
To use reduce on a list, say $l, you could simply use [] as in:
reduce $l[] as $x (_;_)

Self-referential values in an R config file

Using the config package, I'd like elements to reference other elements,
like how path_file_a references path_directory.
config.yml file in the working directory:
default:
path_directory : "data-public"
path_file_a : "{path_directory}/a.csv"
path_file_b : "{path_directory}/b.csv"
path_file_c : "{path_directory}/c.csv"
# recursive : !expr file.path(config::get("path_directory"), "c.csv")
Code:
config <- config::get()
config$path_file_a
# Returns: "{path_directory}/a.csv"
glue::glue(config$path_file_a, .envir = config)
# Returns: "data-public/a.csv"
I can use something like glue::glue() on the value returned by config$path_file_a.
But I'd prefer to have the value already substituted so config$path_file_a contains the actual value (not the template for the value).
As you might expect, uncommenting the recursive line creates an endless self-referential loop.
Are there better alternatives to glue::glue(config$path_file_a, .envir = config)?
I came across the same problem and I've written a wrapper around config and glue.
The package is called gonfig and has been submitted to CRAN.
With it you would have:
config.yml
default:
path_directory : "data-public"
path_file_a : "{path_directory}/a.csv"
path_file_b : "{path_directory}/b.csv"
path_file_c : "{path_directory}/c.csv"
And in your R script:
config <- gonfig::get()
config$path_file_c
#> "data-public/c.csv"

Change ossec(wazuh) agent profiles via saltstack

I'm trying to modify the <config-profile> section of a ossc.conf file, including a grains content.
something like:
ossec-profiles:
- profile1
- profile2
and I want to modify the section <config-profile> from
<config-profile>centos, centos7</config-profile>
to
<config-profile>centos, centos7, profile1, profile2</config-profile>
in the ossec.conf file
Any idea?
This can be done by using file.replace module which makes you able to change a text in a file based on a pattern. So in your case you can do the following:
You need to select the pattern as regex group so you can use it later as shown below
configure_ossec:
file.replace:
- name: /path/to/ossec.conf
- pattern: '((<config-profile>.*?)[^<]*)'
- repl: {{ '\\1, ' + pillar['ossec-profiles'] | join(', ') }}
Or you might use this pattern to match only whatever inside config-profile tags then you will be able to call it again in the repl parameter:
(?<=<config-profile>)(.*)(?=<\/config-profile>)
Note: As pillar['ossec-profiles'] should return a list of profiles
then you have to use the join filter in order to separate the values
with comma as a delimiter
And finally the output expected to be something like this:
Changes:
----------
diff:
---
+++
## -1 +1 ##
-<config-profile>centos, centos7</config-profile>
+<config-profile>centos, centos7, profile1, profile2</config-profile>

Resources