If this explanation exists somewhere, I've spent 3 months trying to find it, and failed. I come from a Puppet background, however for various reasons I really want to try replacing it with Salt.
I've gotten a basic setup and I can code my own states and see them work without any issues. The documentation on this is pretty clear. Where I'm stuck is attempting to implement a community salt formula. I can include the formula with it's basic setup and they work fine, however I cannot figure out how to override the defaults from my pillar data. This seems to be where the Salt documentation is weakest.
The documentation states that you should check the pillar.example for how to configure the formula. The pillar.example gives the configuration part clearly, however nether the documentation or the pillar.example tell you how to include this into your pillar data.
In my case I'm trying to use the snmp-formula. I've got a basic setup for my salt file structure, which you can see from my file roots:
file_roots:
base:
- /srv/salt/base
- /srv/formulas/snmp-formula
Inside base I have two pillars:
base/
top.sls
common.sls
top.sls is very simple:
base:
'*':
- common
common.sls has all common config:
include:
- snmp
- snmp.conf
- snmp.trap
- snmp.conftrap
tcpdump:
pkg.latest:
- name: tcpdump
telnet:
pkg.latest:
- name: telnet
htop:
pkg.latest:
- name: htop
snmp:
conf:
location: 'Office'
syscontact: 'Example.com Admin <admin#example.com>'
logconnects: false
# vacm com2sec's (map communities into security names)
com2sec:
- name: mynetwork
source: 192.168.0.13/31
community: public
# vacm group's (map security names to group names)
groups:
- name: MyROGroup
version: v1
secname: mynetwork
- name: MyROGroup
version: v1c
secname: mynetwork
# vacm views (map mib trees to views)
views:
- name: all
type: included
oid: '.1'
# vacm access (map groups to views with access restrictions)
access:
- name: MyROGroup
context: '""'
match: any
level: noauth
prefix: exact
read: all
write: none
notify: none
# v3 users for read-write
rwusers:
- username: 'nagios'
passphrase: 'myv3password'
view: all
In common.sls I've included the snmp-formula and then followed the pillar.example from the formula to customize the configuration. However when I run a test with this I get the following error:
Data failed to compile:
----------
Detected conflicting IDs, SLS IDs need to be globally unique.
The conflicting ID is 'snmp' and is found in SLS 'base:common' and SLS 'base:snmp'
I'm not sure how to proceed with this. It seems like I would have to actually modify the community formula directly to achieve what I want, which seems like the wrong idea. I want to be able to keep the community formula up to date with it's repository and coming from the Puppet perspective, I should be overriding a modules defaults as I need, not modifying the modules directly.
Can someone please make the missing connection for me? How do I implement the pillar.example?
The Salt formula in question is here:
https://github.com/saltstack-formulas/snmp-formula
I have finally figured this out, and it was a problem with a fundamental misunderstanding of the differences between 'file_roots' and 'pillar_roots' as well as 'pillars' vs 'states'. I don't feel that the documentation is very clear in the Getting Started guide about these so I'll explain it, but first the answer.
ANSWER:
To implement the above pillar.example, simply create a dedicated snmp.sls file in your 'base' environment in your pillar data:
/srv/pillar/snmp.sls:
snmp:
conf:
location: 'Office'
syscontact: 'Example.com Admin <admin#example.com>'
logconnects: false
# vacm com2sec's (map communities into security names)
com2sec:
- name: mynetwork
source: 192.168.0.13/31
community: public
# vacm group's (map security names to group names)
groups:
- name: MyROGroup
version: v1
secname: mynetwork
- name: MyROGroup
version: v1c
secname: mynetwork
# vacm views (map mib trees to views)
views:
- name: all
type: included
oid: '.1'
mask: 80
# vacm access (map groups to views with access restrictions)
access:
- name: MyROGroup
context: '""'
match: any
level: noauth
prefix: exact
read: all
write: none
notify: none
# v3 users for read-write
rwusers:
- username: 'nagios'
passphrase: 'myv3password'
view: all
Your pillar_root must also include a top.sls (not to be confused with the top.sls in your file_roots for your states) like this:
/srv/pillar/top.sls
base:
'*':
- snmp
IMPORTANT: This directory and this top.sls for pillar data cannot exist or be included by your file_roots! This is where I was going wrong. For a complete picture, this is the the config I now have:
/etc/salt/master: (snippet)
file_roots:
base:
- /srv/salt/base
- /srv/formulas/snmp-formula
pillar_roots:
base:
- /srv/pillar
Inside /srv/salt/base I have a top.sls which includes a common.sls
for the 'base' environment. This is where the snmp-formula and it's states are included.
/srv/salt/base/top.sls:
base/
top.sls
common.sls
/srv/salt/base/common.sls:
include:
- snmp
- snmp.conf
- snmp.trap
- snmp.conftrap
tcpdump:
pkg.latest:
- name: tcpdump
telnet:
pkg.latest:
- name: telnet
htop:
pkg.latest:
- name: htop
Now the snmp parameter in the pillar data does not conflict with ID of the snmp state from the formula included by the state data.
Related
I am trying to configure an external pillar in github, but no matter what I cannot get the minions to successfully read top.sls. Below is my ext_pillar and pillar_roots config:
pillar_roots:
base:
- /srv/pillar
fileserver_backend:
- gitfs
- roots
gitfs_update_interval: 60
gitfs_base: main
gitfs_remotes:
- https://gituser:gittoken#github.com/gitaccount/saltstack.git:
- mountpoint: salt://
ext_pillar:
- git:
- main https://gituser:gittoken#github.com/gitaccount/saltpillar.git
I have the following in the root of my saltpillar repo:
top.sls:
base:
'*':
- data
data.sls:
info: some test data from remote pillar
Repos are accessible with the URIs provided. When I run salt '*' saltutil.refresh_pillar and then salt '*' pillar.items I get no results. However, I can put top.sls and data.sls directly into /srv/pillar and it works. I put the master in debug mode and don't see any errors running the commands. Any help is appreciated.
Does the following ext_pillar configuration fix your issue? I'm assuming your top.sls you posted is still in the main branch of your git repo.
ext_pillar:
- git:
- main https://gituser:gittoken#github.com/gitaccount/saltpillar.git
- env: base
Your top.sls must reference your actual branch name or you can add the env option to specify a different name.
https://docs.saltproject.io/en/latest/ref/pillar/all/salt.pillar.git_pillar.html
I'm not very successfully trying to figure out how to achieve an override of a group/package with and via a yaml file. Trying to explain my problem using the example (files and folder structre) from the hydra documentation https://hydra.cc/docs/tutorials/structured_config/schema/.
yaml.config as:
defaults:
- base_config # --> reference to dataclass
- db: base_mysql # --> reference to dataclass
- _self_ debug: true
gives the expected (print when running the myapp.py):
db:
driver: mysql
host: localhost
port: 3306
user: ???
password: ???
Using the yaml file instead instead of the base_mysql dataclass is also fine thus the yaml.config as:
defaults:
- base_config
- db: mysql # --> reads db/mysql.yaml
- _self_
debug: true
prints again as expected
db:
driver: mysql
host: localhost
port: 3306
user: omry
password: secret
Overriding individual fields is as well fine, e.g. with config.yaml like
defaults:
- base_config
- db: mysql
- _self_
debug: true
db:
password: UpdatedPassword
What I'm to able to figure out is how to override the full db group with a/via another yaml file - defining the structure via a dataclass and then override/set the values like:
defaults:
- base_config
- db: base_mysql # --> reference to dataclass to define the structure
- _self_
debug: true
db: mysql # --> mysql.yaml
throws the following error:
In 'config': Validation error while composing config:
Merge error: str is not a subclass of MySQLConfig. value: mysql
full_key:
object_type=Config
Searching the internet/stackoverflow already showed me that moving the self to the first position will get rid of the error - but then the composition order is "wrong".
Keeping the order as it is and using the mysql.yaml for an override works well - when done via commandline (python myapp.py db=mysql when the line "db:mysql" is not present), but for my usecase it much more convinient to handle it all via the yaml file(s).
Somehow I assume that the same functionality is available via CLI and files/code and that I just did not mange to figure out how it works.
(hydra version 1.1 in a conda environment with python 3.9)
Thank you very much in advance for any help that you can provide.
If understand correctly, you want to use the defaults list in your primary yaml file to merge together the base_mysql config with the mysql config. This will do the trick:
defaults:
- base_config
- db: [base_mysql, mysql]
- _self_
debug: true
Passing a list [base_mysql, mysql] of config names causes those configs base_mysql and mysql to be merged together. This is documented here -- see the "CONFIG_NAMES" alternative for specifying an option in the defaults list.
Note that passing the CLI override db=mysql (as in python myapp.py db=mysql) results in modification of the defaults list; the resulting defaults list will be the same as if you had used the following in your yaml file:
defaults:
- base_config
- db: mysql
- _self_
debug: true
You can pass a list [base_mysql, mysql] of config names at the CLI like this:
python my_app.py 'db=[base_mysql, mysql]'
I've gotten saltify to work on a fresh minion. I am able to specify a profile for the minion as well. However, I don't know how to assign custom grains to my minion during this process.
Here's my set up.
In /etc/salt/cloud.profiles.d/saltify.conf I have:
salt-this-webserver:
ssh_host: 10.66.77.99
ssh_username: opsuser
password: **********
provider: web-saltify-config
salt-this-fileserver:
ssh_host: 10.66.77.99
ssh_username: opsuser
password: **********
provider: file-saltify-config
In /etc/salt/cloud/cloud.providers I have:
web-saltify-config:
minion:
master: 10.66.77.44
driver: saltify
grains:
layers:
- infrastructure
roles:
- web-server
file-saltify-config:
minion:
master: 10.66.77.55
driver: saltify
grains:
layers:
- infrastructure
roles:
- file-server
When I run my command from my Salt master:
salt-cloud -p salt-this-fileserver slave-salttesting-01.eng.example.com
My /etc/salt/minion file on my minion looks like this:
grains:
salt-cloud:
driver: saltify
profile: salt-this-fileserver
provider: file-saltify-config:saltify
hash_type: sha256
id: slave-salttesting-01.eng.example.com
log_level: info
master: 10.66.77.99
I would really like it to also have:
grains:
layers:
- infrastructure
roles:
- file-server
I'd like for this to happen during the saltify stage rather than a subsequent step because it just fits really nicely into what I'm trying to accomplish at this step.
Is there a way to sprinkle some grains on my minion during "saltification"?
EDIT: The sync_after_install configuration parameter may have something to do with it but I'm not sure where to put my custom modules, grains, states, etc.
I found the grains from my cloud.providers file in /etc/salt/grains This appears to just work if you build your cloud.providers file in a similar fashion to the way I built mine (above).
I enabled debugging (in /etc/salt/cloud) and in the debugging output on the screen I can see a snippet of code that suggests that at some point a file named "grains" in the conf directory in git root may also be transferred over:
# Copy the grains file if found
if [ -f "$_TEMP_CONFIG_DIR/grains" ]; then
echodebug "Moving provided grains file from $_TEMP_CONFIG_DIR/grains to $_SALT_ETC_DIR/grains"
But, I am not sure because I didn't dig into it more since my grains are being sprinkled as I had hoped they would.
I have a repository with salt states for provisioning my cluster of servers in the cloud. Over time, I kept on adding more states - the .sls files - into this repo. Now im starting to struggle what is what and what is where.
I am wondering if there is a there is some software utility/package that will generate documentation off my states repository, preferably as html pages, so that I can browse them and see their interdependencies.
UPDATE:
The state sls files look like this:
include:
- states.core.pip
virtualenv:
pip.installed:
- require:
- sls: states.core.pip
virtualenvwrapper:
pip.installed:
- require:
- sls: states.core.pip
And another sls example:
{% set user_home = '/home/username' %}
my_executable_virtualenv:
virtualenv.managed:
- name: {{ user_home }}/.virtualenvs/my_executable_virtualenv
- user: username
- system_site_packages: False
- pip_pkgs:
- requests
- numpy
- pip_upgrade: True
- require:
- sls: states.core
my_executable_supervisor_entry:
file.managed:
- name: /etc/supervisor/conf.d/my_executable.conf
- source: salt://files/supervisor_config/my_executable.conf
- user: username
- group: username
- mode: 644
- makedirs: False
- require:
- sls: states.core
I did some research and found that salt stack has created one. It does work as HTML pages too. According to the documentation. If you have python installed installing Sphinx is as easy as doing
C:\> pip install sphinx
Salt-stacks docs on this can be found here. According to the docs making the HTML documentation is as easy as doing:
cd /path/to/salt/doc
make HTML
I hope this answer is what you were looking for!
This needs a custom plugin which needs to be written.
There is no plugins directly available to render sls files.
There are some plugins available for rendering YAML files, may be you can modify the same to suite your requirement.
You can use some of the functions in the state module to list all the everything in the highstate for a minion:
# salt-call state.show_states --out=yaml
local:
- ufw.package.install
- ufw.config.file
- ufw.service.enable
- ufw.service.reload
- ufw.config.services
- ufw.config.applications
- ufw.service.running
- apt.apt_conf
- apt.unattended
- cacerts
- kerberos
- network
- editor
- mounts
- openssh
- openssh.config_ini
- openssh.known_hosts
...
And then view the compiled data for each one (also works with states not in the highstate):
# salt-call state.show_sls editor --out=yaml
local:
vim-tiny:
pkg:
- installed
- order: 10000
__sls__: csrf.editor
__env__: base
editor:
alternatives:
- path: /usr/bin/vim.tiny
- set
- order: 10001
__sls__: csrf.editor
__env__: base
Or to get the entire highstate at once with state.show_highstate.
I'm not aware of any tools to build HTML documentation from that. You'd have to do that yourself.
To access all states (not just a particular highstate), you can use salt-run fileserver.file_list | grep '.sls$' to find every state, and salt-run state.orchestrate_show_sls to get the rendered data for each (though you may need to supply pillar data).
I'm trying SaltStack after using Puppet for a while, but I can't understand their use of the word "state".
My understanding is that, for example, a light switch has 2 possible states - on or off. When I write my SLS configuration I am describing what state a server should be in. When I ask SaltStack to provision a server I issue the command salt '*' state.highstate. I understand that a server can be in a highstate (as described in my config) or not. All good so far.
But this page describes other states. It describes lowstate, highstate and overstate (amongst others) as layers. Does this mean a server passes through several states to get to a highstate? Or all states are maintained simultaneously as layers? Or can I configure multiple possible states in my SLS and have SaltStack switch between them? Or are they just layers to SaltStack that have 'state' in the name and I'm confused?
I'm probably missing something obvious, if anyone can nudge me in the right direction I think a lot of the documentation will become clear to me!
Here, top.sls wihch contain,
# cat top.sls
base:
'*':
- httpd_require
and,
# cat httpd_require.sls
install_httpd:
pkg.installed:
- name: httpd
service.running:
- name: httpd
- enable: True
- require:
- file: install_httpd
file.managed:
- name: /var/www/html/index.html
- source: salt://index1.html
- user: root
- group: root
- mode: 644
- require:
- pkg: install_httpd
High state:
We can see all the aspects of high state system while working with state files( .sls), There are three specific components.
High data:
SLS file:
High State
Each individual State represents a piece of high data(pkg.installed:'s block), Salt will compile all relevant SLS inside the top.sls, When these files are tied together using includes, and further glued together for use inside an environment using a top.sls file, they form a High State.
# salt 'remote_minion' state.show_highstate --out yaml
remote_minion:
install_httpd:
__env__: base
__sls__: httpd_require
file:
- name: /var/www/html/index.html
- source: salt://index1.html
- user: root
- group: root
- mode: 644
- require:
- pkg: install_httpd
- managed
- order: 10002
pkg:
- name: httpd
- installed
- order: 10000
service:
- name: httpd
- enable: true
- require:
- file: install_httpd
- running
- order: 10001
First, an order is declared, All States that are set to be first will have their order adjusted accordingly. Salt will then add 10000 to the last defined number (which is 0 by default), and add any States that are not explicitly ordered.
Salt will also add some variables that it uses internally, to know which environment (__env__) to execute the State in, and which SLS file (__sls__) the State declaration came from, Remember that the order is still no more than a starting point; the actual High State will be executed based first on requisites, and then on order.
"In other words, "High" data refers generally to data as it is seen by the user."
Low States:
""Low" data refers generally to data as it is ingested and used by Salt."
Once the final High State has been generated, it will be sent to the State compiler. This will reformat the State data into a format that Salt uses internally to evaluate each declaration, and feed data into each State module (which will in turn call the execution modules, as necessary). As with high data, low data can be broken into individual components:
Low State
Low chunks
State module
Execution module(s)
# salt 'remote_minion' state.show_lowstate --out yaml
remote_minion:
- __env__: base
__id__: install_httpd
__sls__: httpd_require
fun: installed
name: httpd
order: 10000
state: pkg
- __env__: base
__id__: install_httpd
__sls__: httpd_require
enable: true
fun: running
name: httpd
order: 10001
require:
- file: install_httpd
state: service
- __env__: base
__id__: install_httpd
__sls__: httpd_require
fun: managed
group: root
mode: 644
name: /var/www/html/index.html
order: 10002
require:
- pkg: install_httpd
source: salt://index1.html
state: file
user: root
Together, all this comprises a Low State. Each individual item is a Low Chunk. The first Low Chunk on this list looks like this:
- __env__: base
__id__: install_httpd
__sls__: httpd_require
fun: installed
name: http
order: 10000
state: pkg
Each low chunk maps to a State module (in this case, pkg) and a function inside that State module (in this case, installed). An ID is also provided at this level (__id__). Salt will map relationships (that is, requisites) between States using a combination of State and __id__. If a name has not been declared by the user, then Salt will automatically use the __id__ as the name.Once a function inside a State module has been called, it will usually map to one or more execution modules which actually do the work.
salt '\*' state.highstate
'*' refers to all the minions connected to the master.
'state.highstate' is used to run all modules / scripts mentioned in top.sls defined in master
To invoke a specific module / script on all minions, use the following salt command where the state information is defined in state.sls for apache in the example given below.
salt '\*' state.sls apache
To invoke the above salt call only on a specific minion, use the below command.
salt 'minion-name' state.sls apache
I don't know all levels of state, but when you run :
salt '*' state.highstate
Saltstack apply the states you provide in /srv/salt/top.sls.
If you write nothing in it, you can't apply an highstate.
You can apply other state with this command :
salt '*' state.sls state.example
A highstate is just the collection of states that is applied to your server. There is a process in the background where Salt's "state compiler" goes through several stages preparing the data in order to produce the highstate, but you don't really need to worry about those.
Things like the lowstate can help with debugging, but aren't necessary for day to day usage. The highstate is only applied once.