Make non-existent pillar cause warning or error - salt-stack

Is there any way to cause rendering of a salt state to output a warning or error when a requested pillar is not found?
I'm setting up Saltstack for a system with a lot of components, and we use pillars to make sure different Salt states have the same values where appropriate. We also aim to keep all values that will differ from installation to installation in pillars. So there are many pillar files and the total amount of pillar variables is slightly unhinged. When we're installing this system again we'll have to make sure that all all pillars are defined and have an appropriate value. In this process it would greatly help to have warnings or errors when pillars are undefined, instead of having to grep for "None" through the entire minion, then working out where things went wrong. Is there any way to accomplish such warnings or errors?
We can't possibly be the only company with a complicated salt installation.
SOLUTION:
As linked in the approved answer, the solution is to add the following to /etc/salt/minion:
pillar_raise_on_missing: True
PILLAR_RAISE_ON_MISSING: True
Remember to restart salt-minion

You can make sure a pillar return an error if not defined like that:
{%- set port = pillar['db']['host'] %}
Or you can specify default value if there is none.
{%- set host = salt['pillar.get']('db:host', 'localhost') %}
If the pillar is defined but without value, you can use the pillar module, to get an error.
Check here: https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.pillar.html#salt.modules.pillar.get
Attempt to retrieve the named value from pillar, if the named value is
not available return the passed default. The default return is an
empty string except opts['pillar_raise_on_missing'] is set to
True, in which case a KeyError will be raised.

The default behavior is to produce an error if a requested pillar value isn't present. The documentation recommends using pillar.get with a default value so that missing values get filled in, but it's not required. To get hard errors from states, do an attribute lookup on the pillar instead.
Suppose you have a pillar file that looks like this:
some_app:
opt1: 23
opt2: null
Here are some methods of accessing them and what they will do:
{{ pillar['some_app']['opt1'] }} # Prints 23
{{ pillar.some_app.opt1 }} # Alternate syntax, also prints 23
{{ pillar['some_app']['missing_opt'] }} # ERROR
{{ pillar.some_app.missing_opt }} # ERROR
{{ pillar.get('some_app:missing_opt', 17) }} # No error, prints 17
{{ pillar['some_app']['opt2'] }} # Also no error. I think it prints None.
The methods marked as 'error' will make the state fail noisily when it is run, which I gather is what you want.

Related

How to set salt cmd.run changed state based on stderr

In ansible you can register a variable to capture the output of running a command, specify a changed_when argument, and use that variable to determine if the state changed:
- name: Combine multiple conditions to override 'changed' result
ansible.builtin.command: /bin/fake_command
register: result
ignore_errors: True
changed_when:
- '"ERROR" in result.stderr'
- result.rc == 2
In salt I found that there is a stateful argument to cmd.run but the output appears that it needs to be in a specific json or key=value pair format to do the equivalent. In salt is there a way to do the equivalent of the ansible example to search for the existence of a specific string in the output to determine if the state changed?
You'd need to write a wrapper that returns the stateful-compatible response.
do the thing:
cmd.run:
- name: '/bin/fake_command 2>&1 | grep "ERROR" && echo "changed=true" || true'
- stateful: true
Though outputting "ERROR" seems an odd way for this command to indicate successful changes.
Alternatively, you can write an entirely custom state for more complex logic.

Filter Data in Oracle Goldengate Replicat Side

I have a requirement to setup CDC from Source(Oracle) to Target(BigQuery) using Goldengate.
I can have only option to filter data in replicat side based on Specific column name .
As per the below link :
https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/using-oracle-goldengate-parameter-files.html#GUID-7F405A81-B2D1-4072-B254-DC2B0EC56FBA
I have setup the replicat like below
REPLICAT RPOC
TARGETDB LIBFILE libggjava.so SET property=dirprm/bqpoc.props
SOURCEDEFS /app/oracle/ogg_bigdata/dirdef/poc.def
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 500
MAP ARADMINPI.TPOC, TARGET PRD.TPOCFL,KEYCOLS(ID),WHERE (NAME= ?SOUVIKPOC);
===================================
export SOUVIKPOC='Smith'
But I am getting below error
2020-02-19 05:47:37 ERROR OGG-01157 Error in WHERE clause for ARADMINPI.TPOC.
=============================
Is there anything I am doing wrong here?
For Parameter Substitution to work, you'll need to enclose ?SOUVIKPOC in quotes, like this:
MAP ARADMINPI.TPOC, TARGET PRD.TPOCFL,KEYCOLS(ID),WHERE (NAME= '?SOUVIKPOC');
There should also be additional information about the failure earlier in the report file.
Another Example Using #GETENV
Another option is to use the #GETENV function instead of Parameter Substitution. Here, the MAP statement uses a FILTER clause instead of the WHERE clause:
MAP ARADMINPI.TPOC, TARGET PRD.TPOCFL,KEYCOLS(ID),
FILTER (#STREQ(NAME, #GETENV('OSVARIABLE', 'SOUVIKPOC')));
Unless you set the SOUVIKPOC environment variable prior to running GGSCI (and executing START MGR), you need to add a SETENV statement to your parameter file:
SETENV (SOUVIKPOC = 'Smith')
Putting it all together:
REPLICAT RPOC
TARGETDB LIBFILE libggjava.so SET property=dirprm/bqpoc.props
SOURCEDEFS /app/oracle/ogg_bigdata/dirdef/poc.def
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 500
SETENV (SOUVIKPOC = 'Smith')
MAP ARADMINPI.TPOC, TARGET PRD.TPOCFL,KEYCOLS(ID),
FILTER (#STREQ(NAME, #GETENV('OSVARIABLE', 'SOUVIKPOC')));

salt attribute(key/value) replacement based on particular stanza

Using salt i want to find the attribute(key) and replace it with value based on specific stanza. The attribute(key) is present in multiple times in a file under different stanzas. I want to find my attribute under specific stanza and replace with value.
Example:
output.kafka:
# Boolean flag to enable or disable the output module.
enabled:
I need to find enabled: under output.kafka: and replace it with value. The enabled: attribute present multiple times in my file.
Thanks
Bala.
Salt has a few commands like file.line, file.replace and file.blockreplace that can modify an existing file, but I highly recommend managing the whole file using file.managed. It makes for a less brittle experience.
Here's an example based off your question:
Pillar top file:
cat /srv/pillar/top.sls
base:
'*':
- common
'minion01':
- minion01kafkasettings
Set our pillar data:
cat /srv/pillar/minion01kafkasettings.sls
kafka_output: True
Here's our filebeat template:
cat /srv/salt/filebeat.tmpl
output.kafka:
# Boolean flag to enable or disable the output module.
enabled: {{ pillar.get('kafka_output', True) }}
Here's the filebeat Salt sls file:
cat /srv/salt/filebeat.sls
the_filebeat_file:
file.managed:
- name: /etc/filebeat/filebeat.yml
- template: jinja
- user: root
- group: root
Then we can run the following:
Refresh our pillar data
salt 'minion01' saltutil.refresh_pillar
Then apply the sls file:
salt 'minion01' state.sls filebeat
I have another theory using file.seralize that might work but not in its current state, Maybe Dave could help.
{% set json_data = salt.cp.get_file_str('/etc/filebeat/filebeat.yml') | load_yaml %}
{% do json_data.update({'enabled': pillar.get('kafka_output', True) }) %}
update_config:
file.serialize:
- name: /etc/filebeat/filebeat.yml
- user: root
- group: root
- mode: 644
- formatter: yaml
- dataset: |
{{ json_data | yaml(False)| indent(8) }}
This state should load the whole configuration file then you can modify any of its values based on your pillar setting using the do statement in your case it could be
{% do json_data.update({'enabled': pillar.get('kafka_output', True) }) %}
The config file is populated but not as exepcted as the result will be as following:
'enabled: true
status: active
'
Note there are quotes and the yaml is not intended correctly, is there another way to make it work ? I will update this answer if I found any new results

SaltStack - How can I use grain values in pillar.get statement?

I've got a few configuration values in my application that are tied to ip, mac addr, server name, etc and am trying to come up with a modular state that will match on the correct value from my pillar with a pillar.get. I'm not finding any info on how to use grain values in pillar.get.
pillar example:
account:
qa:
server1:
appUsername: 'user2'
appPassword: 'password2'
server2:
appUsername: 'user2'
appPassword: 'password2'
prod:
server3:
appUsername: 'user3'
appPassword: 'password3'
server4:
appUsername: 'user4'
appPassword: 'password4'
Lines from my template:
keyUser={{ salt['pillar.get']('account:grains['env']:grains['id']:appUsername', 'default_user') }}
keyPass={{ salt['pillar.get']('account:grains['env']:grains['id']:appPassword', 'default_pass') }}
This just seems so natural, but whatever I try errors out, or will escape actual grain lookup and give me the defaults. I also can't find anything on google. Anybody have a solution? Should I dynamically set the appUsername and appPassword values on the pillar instead? I like the layout in the pillar as is, because it's a great easy to read lookup table, without a ton of conditional jinja.
First you can't just embed grains['env'] into the pillar lookup string- you'll need to concatenate. Second, your Jinja assignment looks wrong. Try this:
{% set keyUser = pillar.get('account:'~grains['env']~':'~grains['id']~':appUsername', 'default_user') %}
~ is the concatenate operator in Jinja.
Also, salt['pillar.get']('blah') is the same as pillar.get('blah').
However! It's difficult to be sure without the actual error and/or the full template.

How do I get the path to the directory containing the current sls file or template?

A specific case leading to a more general question: I have a salt formula that looks like this:
formula/
init.sls
some_other_state.sls
defaults.yaml
As part of both sls's behavior I want them to load defaults.yaml into a dict. I can do it this way in either of them:
{%- import_yaml formula/defaults.yaml as defaults %}
...but this hardcodes the location of the formula relative to the base of the salt tree, and will break if the tree is ever restructured and the formula moved to (say) a nested directory.
I tried this instead:
{%- import_yaml (slspath + "/defaults.yaml") as defaults %}
This works for init.sls but not for some_other_state.sls; the reason is that slspath expands to the containing directory for the former but the full path for the latter.
What I really want is some equivalent of slspath that always expands to "the directory containing the currently-running sls." I can then specify whatever path I need relative to that.
Does such a variable exist, and if so, what is it?
The variable you are looking for is tpldir which was added in 0c78d7dc.

Resources