The situation: We have multiple salt formulas with certain pillar-configured options that, in our environment, are identical. For example, they use the same URL for an upstream service. We would like to avoid duplicating these values in multiple pillar locations (we want a single point of truth), but we don't want to write the formulas in such a manner that they share pillar keys (orthoganality is good for the soul).
It seems to me that the right way to do this is to have one pillar file with the "shared" values and import them from there to the appropriate locations in the formula-specific pillar files. For example:
# pillar/shared.sls
upstream: https://example.com/youarehere
# pillar/formula1.sls
{%- from shared import upstream %}
formula1:
upstream_uri: {{ upstream }}
# pillar/formula2.sls
{%- from shared import upstream %}
formula2:
upstream_url: {{ upstream }}
# and so on...
Of course, that doesn't work as written. What's the correct way to do it?
Try this:
pillar/shared.sls
upstream: https://example.com/youarehere
pillar/formula1.sls
{% import_yaml "shared.sls" as defaults %}
formula1:
upstream_uri: {{ defaults.upstream }}
pillar/formula2.sls
{% import_yaml "shared.sls" as defaults %}
formula2:
upstream_url: {{ defaults.upstream }}
Related
I'm trying to setup users on hosts. The users are split to groups, e.g admins, users, etc...
I want to tag each host with a list of which groups users should be present on it. i.e ideally I would be able to do this from pillar:
base:
'*':
- user_groups: [admins]
'env: qa* AND role: frontend':
- user_groups: [users, ci]
Right now I have to prepare files with various permutations if I want to do this from pillar.
I was thinking I can have a single pillar with expressions like so:
server_user_groups:
'*': [admins]
'env: qa* AND role: frontend': [users, ci]
Then from my state I can iterate this structure:
{% for target, groups in pillar.server_user_groups if salt.check_target(target) %}
# Include this group of users on this host...
{% endfor %}
Is this doable?
Never mind, found it...
https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.match.html
{% for target, groups in pillar.server_user_groups if salt['match.compound'](target) %}
# Include this group of users on this host...
{% endfor %}
I created a complex state for API service, it involves git checkouts, python venv, uwsgi, nginx, etc etc. It works fine.
Now I would like to turn it into a template and execute it several times per minion, with variables supplied from pillar - i.e something like.
{% for apiserver in pillar.apiservers %}
include apiserver_template.sls, locals: apiserver.config
{% endfor %}
where apiserver_template will work with context supplied to it, with apiserver.config having all config data for each API instance. I know syntax is wrong but hopefully I am communicating the idea - ideally, something like executing ruby partials with supplying local variables.
How is it done properly in saltland?
It sounds to me like Jinja Macro is something you want to use for this. You can find more information about usage here: https://docs.saltstack.com/en/2015.8/topics/development/conventions/formulas.html#jinja-macros
In short what you will have in your case may look like:
{% macro api_server(git_repo, python_venv_path, python_venv_requirements) %}
{{python_venv_path}}:
virtualenv.managed:
- system_site_packages: False
- requirements: salt://{{python_venv_requirements}}
{{git_repo}}:
git.latest:
- name: {{git_repo}}
{% endmacro %}
Assuming you have a pillar apiservers where each api server has git_repo, python_venv_path and python_venv_requirements values, you can use the macro like this:
{% for server in salt.pillar.get('apiservers', []) %}
{{ api_server(server['git_repo'], server['python_venv_path'], server['python_venv_requirements']) }}
{% endfor %}
If you want - you can also put a macro in a separate state file and then import a marco as a regular salt resource.
Please also not that instead of pillar.apiservers I used salt.pillar.get('apiservers', []). This is a safer way to get data from pillar. If for some reason a pillar is unavailable - the later code will result in empty dict instead of failure in first case.
I'm having trouble using the salt mine in a pillar to dynamically create a list of hosts based on a grain value match. I don't get any error, I get no output for all hosts. Actually, I can't get any output for mine from the pillar at all even when using the example from the salt docs. I know it isn't an issue with my top file, because I can access all of the other pillar values. My test minion's mine.interval is set to 5. I've refreshed pillar data, and ran mine.update.
Here's an example of my pillar:
mine_functions:
network.ip_addrs: []
grains.item:
- host
- role
My template file that access the mine functions:
#I know this is writing the same list for each match, I'm just doing this for testing and I'll concat the results into a string when I know it works:
{% for host in salt['mine.get']('roles:web', 'grains.items:host', expr_form='grain') | dictsort() %}
serverList= {{ host }}
{% endfor %}
Output from CLI:
salt "server.domain.com" mine.get "*" "*"
server.domain.com:
----------
How do I get this to work? I get no errors, no output, it just runs smoothly, but nothing is written in the file and I get nothing from the command line. My goal here is to be able to dynamically build a list of servers that match a specific grain to set a configuration value in a config template. Am I down the wrong path here, is there a better way?
I'd recommend using mine.get directly in your sls file to get that list of hosts. I don't think there's any need to pass that through pillar data.
#Utah_Dave, thanks so much for the help both here and in IRC.
Posting this as an answer so anyone else searching for this gets a good example...
pillar:
mine_functions:
grains.items: []
template file:
{% set ft_hosts = [] %}
{% for grain_vals in salt['mine.get']('role:ps:ft:True', 'grains.items', expr_form='grain').items() %}
{% do ft_hosts.append(grain_vals[1]['host']) %}
{% endfor %}
ft.ps.server.hosts={{ ft_hosts|join('|') }}
I'm trying to have salt automatically deploy files to users' home directory (after creating them). The directory structure (under file_roots) is as follows:
users/
init.sls
user_list.jinja
files/
userX/
some_dir_I_want_deployed_to_userX_homedir/
some_script_I_want_deployed.sh
etc
user_list.jinja has a list of users. The idea is that if there's a directory for userX under files/, that directory subtree should be deployed to userX's home. However, I'd rather not have to create empty directories for userY etc if they don't have anything to deploy, so I'm trying to test existence of the directory to avoid an error.
Here's the relevant excerpt from users/init.sls:
{% from "users/user_list.jinja" import users with context %}
{% for name, user in users.items() %}
{% set files_path = '{0}/files/{1}'.format(salt['file.dirname'](tplpath), name) %}
{% if salt['file.directory_exists'](files_path) %}
{{ user.home }}:
file.recurse:
- source: salt://users/files/{{ name }}
- user: {{ name }}
- group: {{ name }}
{% endif %}
{% endfor %}
With a fair bit of debugging (which was rather necessary to figure out the above), I've worked out that this is a chicken-and-egg situation, namely:
The file.directory_exists test is run on the minion (that's fair)
The salt file-server seems to have an optimization whereby it only deploys to minion (to their local cache under /var/cache/salt/minion/file) items which are referenced in states (more likely, the minions only request stuff which they see referenced).
So unless the directory subtree for users/files/userX already exists on the minion, file.directory_exists returns False, which means the entire portion which starts with {{ user.home }} is suppressed during Jinja rendering; hence it's not referenced and the copy (to the minion's local cache) never occurs.
If I manually create an empty directory structure for users/files/userX on the minion, everything starts to work. This tells me my theory is at least partially correct.
I can "feel" I'm doing smth wrong here (the whole thing feels too procedural). What's the best approach to achieve this? The requirement itself doesn't seem too far-fetched.
The more salt-ish way to do this is to have some data in pillar data and check for the existence of that key. Something like user.enabled. But, that would require you to keep settings in 2 places, in pillar and in the file_roots.
You don't want to check for the existence of the directory on the minion server, you want to check for the existence of the file in your file roots.
Unfortunately, I don't think it's possible to check for the existence of a file under salt:// scheme. If I'm wrong, then all you have to do is replace your check for directory existence with the syntax to check for file_root file existence.
The more salt-ish approach is to define which users are enabled/disabled on each machine in pillar data, and use the user module to add them to the system.
https://github.com/saltstack-formulas/users-formula
You can add to the standard pillar that's given with the standard users formula and put a key that says to sync files
#pillar
users:
ausername:
fullname: A User
syncfiles: True
busername:
fullname: B User
syncfiles: False
#state
{% for name, user in pillar.get('users', {}).items() if user.absent is not defined or not user.absent %}
{% if user.syncfiles %}
/home/{{ user.username }}:
file.recurse:
- source: salt://users/files/{{ user.username }}
- user: {{ user.username }}
{% if user.prime_group %}
- group: {{ user.prime_group.name }}
{% endif %}
{% endif %}
{% endfor %}
Actually, the standard users-formula already handles pre-populating with files. I would just use that formula. I know it's keeping track of data in 2 places, but you get the bonus of leveraging an already built state file.
In Saltstack, I have the following use case:
There is a state redis.sls which can be included by other states. The result of redis.sls should be configured differently, depending on the state which included redis.sls.
For example:
redis.sls:
--------
{% if x==1 %}
#do something
{% else %}
#do something else
{% endif %}
state_a.sls
-----------
{% set x=1 %}
include:
- redis
state_b.sls
-----------
{% set x=2 %}
include:
- redis
But x is not recognized in *state_a* and *state_b*
I also tried setting a pillar value with something like this:
{{salt['pillar.set']('x', 1)}}
but that didn't work either.
Any other ideas?
I'd like to hear what the experts say but I have a similar use case. What I did was use the jinja template to extend a base template then my child templates populated the variables.
{% extends "base.template.sls" %}
{% block x %}1{% endblock %}
Only problem might be that you now have to call state_a and state_b separately but you can always put them in a comma separated list if you want both called.
Make your redis state a jinja macro.
redis.sls:
--------
{% macro redis(x) %}
{% if x==1 %}
#do something
{% else %}
#do something else
{% endif %}
{% endmacro %}
state_a.sls
-----------
{% from 'redis.sls' import redis %}
{{ redis(1) }}
state_b.sls
-----------
{% from 'redis.sls' import redis %}
{{ redis(2) }}
For clarity redis.sls should be renamed to redis.jinja here.
This, and many other ways of managing state customization is best explained in the Salt Formulas conventions guide. Specifically the part about Jinja macros
Note that your if x==1 logic can be probably avoided altogether, take a look at the 'better' version of haproxy example in the guide.
It looks like you want to parameterize a state based on either what depends on it, or where it is used. That sounds like whatever is setting the parameter(s) on which the redis.sls state is supposed to mutate, depends on a specific configuration of redis.
To me, that seems like there are more than one distinct states in which redis could be, and that some of your states depend on one state of redis, and others of your states depend on other states of redis.
So, give the installation of redis one state, and the specific configurations of redis would each get their own state. Your state_a could depend on redis_state_1 and your state_b would in turn depend on redis_state_2. Both redis_state_1 and redis_state_2 would depend on redis. It seems to me that the parameter passing you're asking about would be less explicit.
SALT.STATES.ENVIRON might work for you:
set_secret_key:
environ.setenv:
- name: SECRET_KEY
- value: ABC123!##abc
- update_minion: True
[..]
settings_secret_key:
file.replace:
- name: {{ salt['pillar.get']('data:source_folder') }}superlists/settings.py
- pattern: "SECRET_KEY =.+$"
- repl: 'SECRET_KEY = os.environ["SECRET_KEY"]'