Using Inline Yaml Formatting over multiple lines? - dictionary

I'm trying to create a YAML file for use with nose-testconfig with data like this:
"Computers":
"Brand: "Dell"
"Cost": 500
"IP": "194.66.82.11"
"Brand: "HP"
"Cost": 600
"IP": "194.66.82.13"
"Brand: "Asus"
"Cost": 550
"IP": "194.66.82.15"
The problem I'm having is that each of the repeated items just get overwritten. I need to have it be a list of lists of dictionaries, however I don't know how to make that work. I can't just use the inline formatting with brackets and braces because the actual data I'm working with has a lot more data in each dictionary, so the lines would be way too long.
Is there a way to format this in a way that keeps each list of dictionaries separate?

What I think you want is this:
Computers:
- Brand: Dell
Cost: 500
IP: 194.66.82.11
- Brand: HP
Cost: 600
IP: 194.66.82.13
- Brand: Asus
Cost: 550
IP: 194.66.82.15
This represents a mapping ("dictionary") with one key, "Computers," whose value is a sequence ("list") having three items, each of which is a mapping with the three keys "Brand," "Cost", and "IP."
In JavaScript, for example, it would deserialize to this structure:
{ Computers:
[ { Brand: "Dell",
Cost: 500,
IP: "192.66.82.11"
},
{ Brand: "HP",
Cost: 600,
IP: "192.168.82.13"
},
{ Brand: "Asus",
Cost: 550,
IP: "192.168.82.15"
}
]
}
P.S. You'll notice I removed the quotation marks. Quotation marks are an antipattern in YAML—there are very few situations where they're necessary (for example, if you wanted "500" to be deserialized as a string instead of a number) and they add a lot of line noise, defeating the purpose of YAML, which is ease of reading and editing.
P.P.S. A utility I find hugely useful is this Online YAML Parser, which will show you the result, in either JSON or Python notation, of any YAML input as you type.

What you should do if you are unsure about how your YAML for a certain construct should look is create the construction in your programming language of choice (for which there should be no such uncertainty), and then dump it. E.g. in Python:
import sys
import ruamel.yaml
data = dict(Computers=[
dict(Brand="Dell", Cost=500, IP="194.66.82.11"),
dict(Brand="HP", Cost=600, IP="194.66.82.13"),
dict(Brand="Asus", Cost=550, IP="194.66.82.15"),
])
yaml = YAML()
yaml.dump(data, sys.stdout)
gives you:
Computers:
- IP: 194.66.82.11
Brand: Dell
Cost: 500
- IP: 194.66.82.13
Brand: HP
Cost: 600
- IP: 194.66.82.15
Brand: Asus
Cost: 550
What you see is that quotes around simple scalars are unnecessary (but they are allowed according to the specification, so it is perfectly OK to put them in). You also see that you missed putting a list in your YAML file (which in block mode is indicated by the - dashed items).
Using this way it is easy to determine for yourself how your data should be formatted to be valid YAML. For some parsers you need to try with block or non-flow style formatting to get a better readable result, for ruamel.yaml (of which I am the author) block style is default as it is, IMO, more readable.
Please note that keys for mappings in YAML are by definition unordered, although ruamel.yaml can preserve these for you in roundtrip mode

Related

What *is* a salt formula, really?

I am trying to work through the Salt Formulas documentation and seem to be having a fundamental misunderstanding of what a salt formula really is.
Understandably, this question may seem like a duplicate of these questions, but due to my failing to grasp the basic concepts I'm also struggling to make use of the answers to these questions.
I thought, that a salt formula is basically just a package that implements extra functions, a lot like
#include <string.h>
in C, or
import numpy as np
in Python. Thus, I thought, I could download the salt-formula-linux to /srv/formulas/salt-formula-linux/, add that to file_roots, restart the master (all as per the docs), and then write a file like swapoff.sls containing
disable_swap:
linux:
storage:
swap:
file:
enabled: False
(the above is somewhat similar to the examples in the repo's root) in hope that the formula would then handle removing the swap entry from /etc/fstab and running swapoff -a for me. Needless to say, this didn't work, clearly because I'm not understanding what a salt formula is meant to be.
So, what is a salt formula and how do I use it? Can I make use of it as a library of functions too?
This answer might not be fully correct in all technicalities, but this is what solved my problem.
A salt formula is not a library of functions. It is, rather, a collection of state files. While often a state file can be very simple, such as some of my user defined
--> top.sls <--
base:
'*':
- docker
--> docker.sls <--
install_docker_1703:
pkgrepo.managed:
# stuff
pkg.installed:
- name: docker-ce
creating a state file like
--> swapoff.sls <--
disable_swap:
linux.storage.swap: # and so on
is, perhaps, not the way to go. Well, at least, maybe not for a beginner with lacking knowledge.
Instead, add an item to top.sls:
- linux.storage.swap
This is not enough, however. Most formulas (or the state files within them, if you will) are highly parametrizable, i.e. they're full of placeholders with variable names, such as {{ swap.device }}. If there's nothing to fill this gap, the state fill will not be able to do anything. These gaps are filled from pillars.
All that remains, is to create a file like swap.sls in /srv/pillar/ that would contain something like (as per the examples of that formula)
linux:
storage:
enabled: true
swap:
file:
enabled: true
engine: file
device: /swapfile
size: 1024
and also /srv/pillar/top.sls with
base:
'*':
- swap
Perhaps /srv/pillar should also be included in pillar_roots in /etc/salt/master.
So now /srv/salt/top.sls runs /srv/formulas/salt-formula-linux/linux/storage/swap.sls which using the guidance of /srv/pillar/top.sls pulls some parameters from /srv/pillar/swap.sls and enables a swapfile.

How to remove duplicate lines in YAML format configuration files?

I have a bunch of manifest/yaml files that may or may not have these key value pair duplicates:
...
app: activity-worker
app: activity-worker
...
I need to search through each of those files and find those duplicates so that I can remove one of them.
Note: I know that to replace a certain string (say, switch service: to app:) in all files of a directory (say, dev) I can run grep -l 'service:' dev/* | xargs sed -i "" 's/\service:/app:/g'. I'm looking for a relation between lines.
What you call YAML, is not YAML. The YAML specification
very explicitly states that
keys in a mapping must be unique, and your keys are not:
The content of a mapping node is an unordered set of key: value node
pairs, with the restriction that each of the keys is unique. YAML
places no further restrictions on the nodes. In particular, keys may
be arbitrary nodes, the same node may be used as the value of
several key: value pairs, and a mapping could even contain itself as
a key or a value (directly or indirectly).
On the other hand some libraries have implemented this incorrectly, choosing to overwrite
any previous value associated with a key, with a later value. In your case, since
the values are the same, which value would be taken doesn't really matter.
Also your block style representation is not the only way to represent key-value pairs of a
mapping in "YAML", these duplicates could also be represented in a mapping, as
{...., app: activity-worker, app: activity-worker, .... }
With the two occurences not necessarily being next to each, nor on the same line. The
following is also semantically equivalent "YAML" to your input:
{...., app: activity-worker, app:
activity-worker, .... }
If you have such faulty "YAML" files, the best way to clean them up is
using the round-trip capabilities of
ruamel.yaml (disclaimer: I
am the author of that package), and its ability to switch except/warn
on faulty input containing duplicate keys. You can install it for your
Python (virtual environment) using:
pip install ruamel.yaml
Assuming your file is called input.yaml and it contains:
a: 1 # some duplicate keys follow
app: activity-worker
app: activity-worker
b: "abc"
You can run the following one-liner:
python -c "import sys; from ruamel.yaml import YAML; yaml = YAML(); yaml.preserve_quotes=yaml.allow_duplicate_keys=True; yaml.dump(yaml.load(open('input.yaml')), sys.stdout)"
to get:
a: 1 # some duplicate keys follow
app: activity-worker
b: "abc"
and if your input were like:
{a: 1, app: activity-worker, app:
activity-worker, b: "abc"}
the output would be:
{a: 1, app: activity-worker, b: "abc"}

How to delete an inherit property from yaml config?

I have a yaml file like this:
local: &local
image: xxx
# *tons of config*
ci:
<<: *local
image: # delete
build: .
I want ci to inherit all values from local, except the image.
Is there a way to "delete" this value?
No there isn't a way to mark a key for deletion in a YAML file. You can only overwrite existing values.
And the latter is what you do, you associate the empty scalar as value to the key image as if you would have written:
image: null # delete
There are two things you can do: post-process or make a base mapping in your YAML file.
If you want to post-process, you associate a special unique value to image, or a specially tagged object, and after loading recursively walk over the tree to remove key-value pairs with this special value. Whether you can already do this during parsing, using hooks or overwriting some of its methods, depends on the parser.
Using a base mapping requires less work, but is more intrusive to the YAML file:
localbase: &lb
# *tons of config*
local: &local
image: xxx
ci:
<<: *lb
build: .
If you do the former you should note that if you use a parsers that preserve the "merge-hierarchy" on round-tripping (like my ruamel.yaml parser can do) it is not enough to delete the key-value pair, in that case the original from local would come back. Other parsers that simply resolve this at load time don't have this issue.
For properties that accept a list of values, you can send [] as value.
For example in docker-compose you don't want to inherit ports:
service_1: &service_1
# some other properties.
ports:
- "49281:22"
- "8876:8000"
# some other properties
image: some_image:latest
service_2:
<<: *service_1
ports: [] # it removes ports values.
image: null # it removes image value.

SaltStack - How can I use grain values in pillar.get statement?

I've got a few configuration values in my application that are tied to ip, mac addr, server name, etc and am trying to come up with a modular state that will match on the correct value from my pillar with a pillar.get. I'm not finding any info on how to use grain values in pillar.get.
pillar example:
account:
qa:
server1:
appUsername: 'user2'
appPassword: 'password2'
server2:
appUsername: 'user2'
appPassword: 'password2'
prod:
server3:
appUsername: 'user3'
appPassword: 'password3'
server4:
appUsername: 'user4'
appPassword: 'password4'
Lines from my template:
keyUser={{ salt['pillar.get']('account:grains['env']:grains['id']:appUsername', 'default_user') }}
keyPass={{ salt['pillar.get']('account:grains['env']:grains['id']:appPassword', 'default_pass') }}
This just seems so natural, but whatever I try errors out, or will escape actual grain lookup and give me the defaults. I also can't find anything on google. Anybody have a solution? Should I dynamically set the appUsername and appPassword values on the pillar instead? I like the layout in the pillar as is, because it's a great easy to read lookup table, without a ton of conditional jinja.
First you can't just embed grains['env'] into the pillar lookup string- you'll need to concatenate. Second, your Jinja assignment looks wrong. Try this:
{% set keyUser = pillar.get('account:'~grains['env']~':'~grains['id']~':appUsername', 'default_user') %}
~ is the concatenate operator in Jinja.
Also, salt['pillar.get']('blah') is the same as pillar.get('blah').
However! It's difficult to be sure without the actual error and/or the full template.

Is there a way to display only changes and errors

I have quite extensive salt config and I want to be able to see what has changed. If I just run salt '*' state.highstate I got the whole list with things that were present and not changed - like 3 to 4 screens of log. But I'd really like to see only things that changed in the last job.
It doesn't have to work for the salt call, it can also employ salt-run jobs.lookup_jid.
You can set state_verbose: False in /etc/salt/master or /etc/salt/minion.
If you want to shorten the output to one line per state, set state_output: terse.
You can also pass these filters on command line:
salt --state-output=terse '*' state.highstate
If you only want to see changes, you can use state-output=changes or state-output=mixed. The latter one will show more information on a failure.
See the following answers fore more detail: basepi, psarossy
We've also added state_output: mixed which will give you the same output as terse, except if there's a failure, in which case it will give you the more verbose output.
To actually answer the question, yes, there is an output filter for changes only:
salt '*' state.highstate --state-output=changes
This will display one liners for things that are in the right state and the proper output for the changes. ie:
<...>
Name: /etc/sudoers - Function: file.managed - Result: Clean
Name: /etc/timezone - Function: file.managed - Result: Clean
Name: /etc/pki/tls/certs/logstash-forwarder.crt - Function: file.managed - Result: Clean
Name: /etc/init.d/logstash-forwarder - Function: file.managed - Result: Clean
----------
ID: /etc/logstash-forwarder
Function: file.managed
Result: True
Comment: File /etc/logstash-forwarder updated
Started: 14:14:28.580950
Duration: 65.664 ms
Changes:
----------
diff:
---
+++
## -1,6 +1,6 ##
{
"network": {
- "servers": [ "10.0.0.104:5000" ],
+ "servers": [ "10.0.0.72:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
Name: deb http://packages.elasticsearch.org/logstashforwarder/debian stable main - Function: pkgrepo.managed - Result: Clean
Name: logstash-forwarder - Function: pkg.installed - Result: Clean
<...>
There are 2 options, first is to change the state_output in master's configuration file, like mentioned in the accepted answer, and it also possible to override the state output in command line, like:
salt --state-output=mixed \* test.version
As of the following PR that was merged into Salt 2015.8.0 (https://github.com/saltstack/salt/pull/26962) it is now possible to toggle the state_verbose flag from command line when running highstate. This overrides the config you can set in /etc/salt/master that was mentioned in previous answers.
The following command should now display only the changes and errors from a highstate run salt '*' state.highstate --state-verbose=False
You can use the below to shorten the output in one line and then filter that output to show only the changes:
salt -v 'minion' state.highstate test=True --state-output=terse --state-verbose=False

Resources