I have quite extensive salt config and I want to be able to see what has changed. If I just run salt '*' state.highstate I got the whole list with things that were present and not changed - like 3 to 4 screens of log. But I'd really like to see only things that changed in the last job.
It doesn't have to work for the salt call, it can also employ salt-run jobs.lookup_jid.
You can set state_verbose: False in /etc/salt/master or /etc/salt/minion.
If you want to shorten the output to one line per state, set state_output: terse.
You can also pass these filters on command line:
salt --state-output=terse '*' state.highstate
If you only want to see changes, you can use state-output=changes or state-output=mixed. The latter one will show more information on a failure.
See the following answers fore more detail: basepi, psarossy
We've also added state_output: mixed which will give you the same output as terse, except if there's a failure, in which case it will give you the more verbose output.
To actually answer the question, yes, there is an output filter for changes only:
salt '*' state.highstate --state-output=changes
This will display one liners for things that are in the right state and the proper output for the changes. ie:
<...>
Name: /etc/sudoers - Function: file.managed - Result: Clean
Name: /etc/timezone - Function: file.managed - Result: Clean
Name: /etc/pki/tls/certs/logstash-forwarder.crt - Function: file.managed - Result: Clean
Name: /etc/init.d/logstash-forwarder - Function: file.managed - Result: Clean
----------
ID: /etc/logstash-forwarder
Function: file.managed
Result: True
Comment: File /etc/logstash-forwarder updated
Started: 14:14:28.580950
Duration: 65.664 ms
Changes:
----------
diff:
---
+++
## -1,6 +1,6 ##
{
"network": {
- "servers": [ "10.0.0.104:5000" ],
+ "servers": [ "10.0.0.72:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
Name: deb http://packages.elasticsearch.org/logstashforwarder/debian stable main - Function: pkgrepo.managed - Result: Clean
Name: logstash-forwarder - Function: pkg.installed - Result: Clean
<...>
There are 2 options, first is to change the state_output in master's configuration file, like mentioned in the accepted answer, and it also possible to override the state output in command line, like:
salt --state-output=mixed \* test.version
As of the following PR that was merged into Salt 2015.8.0 (https://github.com/saltstack/salt/pull/26962) it is now possible to toggle the state_verbose flag from command line when running highstate. This overrides the config you can set in /etc/salt/master that was mentioned in previous answers.
The following command should now display only the changes and errors from a highstate run salt '*' state.highstate --state-verbose=False
You can use the below to shorten the output in one line and then filter that output to show only the changes:
salt -v 'minion' state.highstate test=True --state-output=terse --state-verbose=False
Related
In ansible you can register a variable to capture the output of running a command, specify a changed_when argument, and use that variable to determine if the state changed:
- name: Combine multiple conditions to override 'changed' result
ansible.builtin.command: /bin/fake_command
register: result
ignore_errors: True
changed_when:
- '"ERROR" in result.stderr'
- result.rc == 2
In salt I found that there is a stateful argument to cmd.run but the output appears that it needs to be in a specific json or key=value pair format to do the equivalent. In salt is there a way to do the equivalent of the ansible example to search for the existence of a specific string in the output to determine if the state changed?
You'd need to write a wrapper that returns the stateful-compatible response.
do the thing:
cmd.run:
- name: '/bin/fake_command 2>&1 | grep "ERROR" && echo "changed=true" || true'
- stateful: true
Though outputting "ERROR" seems an odd way for this command to indicate successful changes.
Alternatively, you can write an entirely custom state for more complex logic.
Recently, when I load iTerm2 on my mac, it lists out the configuration of my profile (or possibly its something different).
This is what I see:
Last login: Tue Jun 21 19:54:38 on ttys000COLORFGBG='15;0'
COLORTERM=truecolor
COMMAND_MODE=unix2003
HOME=/Users/me
ITERM_PROFILE=Default
ITERM_SESSION_ID=w0t0p0:FD9764B1-4535-4FE2-932A-97AFD9D6C804
LANG=en_GB.UTF-8
LC_TERMINAL=iTerm2
LC_TERMINAL_VERSION=3.4.15
LOGNAME=me
OLDPWD=/Users/me
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PWD=/Users/me
SHELL=/bin/zsh
SHLVL=1
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.u21pRFo1wk/Listeners
TERM=xterm
TERM_PROGRAM=iTerm.app
TERM_PROGRAM_VERSION=3.4.15
TERM_SESSION_ID=w0t0p0:FD9764B1-4535-4FE2-932A-97AFD9D6C804
TMPDIR=/var/folders/dz/wvb1lxq56wg_wk3ppwqw4mdc0000gn/T/
USER=me
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.googlecode.iterm2
__CF_USER_TEXT_ENCODING=0x0:0:0
me#my-iMac ~ %
I don't really like this behaviour. I can't find where that has been set to happen. I can't see anything in .zshenv, .zshrc, .bashrc and can't see where in iTerm such a thing might be triggered. I've tried creating a new profile, but that also results in the same output.
Any other suggestions to how I can get back to the way it was a few weeks back where it just told me the last time I logged in?
If you add set -x to ~/.zshenv, the shell will print every line it runs with file names and line numbers. You can then find the offending command and remove or fix it. My best guess would be a call to export without any arguments: something like export $foo, where $foo is empty?
I have input files:
Bob_1.fastq.gz
Bob_2.fastq.gz
Bob_3.fastq.gz
Bob_4.fastq.gz
Ron_1.fastq.gz
Ron_2.fastq.gz
Ron_3.fastq.gz
Ron_4.fastq.gz
I am running demultiplexing and trimming steps in one snakefile, like this:
workdir: "/path/to/dir/"
(SAMPLES,) =glob_wildcards('/path/to/dir/raw/{sample}.fastq.gz')
rule all:
input:
expand("demulptiplex/{sample}.fastq.gz", sample=SAMPLES),
expand("trimmed/{sample}.trimmed.fastq.gz", sample=SAMPLES)
rule sabre:
input:
infile="/path/to/dir/raw/{sample}.fastq.gz",
barcodefile= "files/{sample}.txt"
output:
unknownfile=temp("demulptiplex/unknown_barcode_{sample}.fastq.gz"),
shell:
"""
/Tools/sabre-master2/sabre se -f {input.infile} -b {input.barcodefile} -u {output.unknownfile}
"""
rule trimmomatic_se:
input:
r="{sample}.fastq.gz"
output:
r="trimmed/{sample}.trimmed.fastq.gz"
threads: 10
shell:
"""java -jar /Tools/Trimmomatic-0.36/trimmomatic-0.36.jar SE -threads {threads} {input.r} {output.r} ILLUMINACLIP:/Tools/Trimmomatic-0.36/adapters/TruSeq3-SE.fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15 MINLEN:36"""
The demultiplex output files are like this:
Bob_1_CL1.fastq.gz.... Bob_1_CL345.fastq.gz
Bob_2_CL1.fastq.gz.... Bob_1_CL248.fastq.gz
Ron_1_dad1.fastq.gz... Ron_1_dad67.fastq.gz
and so on
So,if I do not specify the demultiplex output file the program would create it by itself. My problem is how to specify/introduce a new wildcard from the output of the previous rule in the next trimming step, as the wildcards are different from initial sample now.
Wildcards just need to be consistent in a rule, not across the workflow. The issue here is that you have a rule generating 'unknown' outputs that you need to process further. For that you need to use checkpoints.
Read through the second block of code about aggregating. Your checkpoint will be demultiplexing and if you don't have any other steps, all will be your aggregate step that calls checkpoints.demultiplex.get. If you search for checkpoint on stackoverflow you will find lots of examples; it's a hard feature to use at first!
I am trying to work through the Salt Formulas documentation and seem to be having a fundamental misunderstanding of what a salt formula really is.
Understandably, this question may seem like a duplicate of these questions, but due to my failing to grasp the basic concepts I'm also struggling to make use of the answers to these questions.
I thought, that a salt formula is basically just a package that implements extra functions, a lot like
#include <string.h>
in C, or
import numpy as np
in Python. Thus, I thought, I could download the salt-formula-linux to /srv/formulas/salt-formula-linux/, add that to file_roots, restart the master (all as per the docs), and then write a file like swapoff.sls containing
disable_swap:
linux:
storage:
swap:
file:
enabled: False
(the above is somewhat similar to the examples in the repo's root) in hope that the formula would then handle removing the swap entry from /etc/fstab and running swapoff -a for me. Needless to say, this didn't work, clearly because I'm not understanding what a salt formula is meant to be.
So, what is a salt formula and how do I use it? Can I make use of it as a library of functions too?
This answer might not be fully correct in all technicalities, but this is what solved my problem.
A salt formula is not a library of functions. It is, rather, a collection of state files. While often a state file can be very simple, such as some of my user defined
--> top.sls <--
base:
'*':
- docker
--> docker.sls <--
install_docker_1703:
pkgrepo.managed:
# stuff
pkg.installed:
- name: docker-ce
creating a state file like
--> swapoff.sls <--
disable_swap:
linux.storage.swap: # and so on
is, perhaps, not the way to go. Well, at least, maybe not for a beginner with lacking knowledge.
Instead, add an item to top.sls:
- linux.storage.swap
This is not enough, however. Most formulas (or the state files within them, if you will) are highly parametrizable, i.e. they're full of placeholders with variable names, such as {{ swap.device }}. If there's nothing to fill this gap, the state fill will not be able to do anything. These gaps are filled from pillars.
All that remains, is to create a file like swap.sls in /srv/pillar/ that would contain something like (as per the examples of that formula)
linux:
storage:
enabled: true
swap:
file:
enabled: true
engine: file
device: /swapfile
size: 1024
and also /srv/pillar/top.sls with
base:
'*':
- swap
Perhaps /srv/pillar should also be included in pillar_roots in /etc/salt/master.
So now /srv/salt/top.sls runs /srv/formulas/salt-formula-linux/linux/storage/swap.sls which using the guidance of /srv/pillar/top.sls pulls some parameters from /srv/pillar/swap.sls and enables a swapfile.
Using salt i want to find the attribute(key) and replace it with value based on specific stanza. The attribute(key) is present in multiple times in a file under different stanzas. I want to find my attribute under specific stanza and replace with value.
Example:
output.kafka:
# Boolean flag to enable or disable the output module.
enabled:
I need to find enabled: under output.kafka: and replace it with value. The enabled: attribute present multiple times in my file.
Thanks
Bala.
Salt has a few commands like file.line, file.replace and file.blockreplace that can modify an existing file, but I highly recommend managing the whole file using file.managed. It makes for a less brittle experience.
Here's an example based off your question:
Pillar top file:
cat /srv/pillar/top.sls
base:
'*':
- common
'minion01':
- minion01kafkasettings
Set our pillar data:
cat /srv/pillar/minion01kafkasettings.sls
kafka_output: True
Here's our filebeat template:
cat /srv/salt/filebeat.tmpl
output.kafka:
# Boolean flag to enable or disable the output module.
enabled: {{ pillar.get('kafka_output', True) }}
Here's the filebeat Salt sls file:
cat /srv/salt/filebeat.sls
the_filebeat_file:
file.managed:
- name: /etc/filebeat/filebeat.yml
- template: jinja
- user: root
- group: root
Then we can run the following:
Refresh our pillar data
salt 'minion01' saltutil.refresh_pillar
Then apply the sls file:
salt 'minion01' state.sls filebeat
I have another theory using file.seralize that might work but not in its current state, Maybe Dave could help.
{% set json_data = salt.cp.get_file_str('/etc/filebeat/filebeat.yml') | load_yaml %}
{% do json_data.update({'enabled': pillar.get('kafka_output', True) }) %}
update_config:
file.serialize:
- name: /etc/filebeat/filebeat.yml
- user: root
- group: root
- mode: 644
- formatter: yaml
- dataset: |
{{ json_data | yaml(False)| indent(8) }}
This state should load the whole configuration file then you can modify any of its values based on your pillar setting using the do statement in your case it could be
{% do json_data.update({'enabled': pillar.get('kafka_output', True) }) %}
The config file is populated but not as exepcted as the result will be as following:
'enabled: true
status: active
'
Note there are quotes and the yaml is not intended correctly, is there another way to make it work ? I will update this answer if I found any new results