How to join two Salt pillar files and merge data? - salt-stack

Question

Is there any way to join two pillar files?
I have a users pillar. It's something like:
users:
joe:
sudouser: True
jack:
sudouser: False
Now I need different set of users for certain servers (ie. add some users to one server). So I create new pillar file:
users:
new_user:
sudouser: True
And assign this topfile to the server. But because the key is the same it would overwrite the first one. If I change it I would need to update the state file (which I really don't want). How should I approach this problem? Is there any way to tell salt to "merge" the files?

Answer 1

It is possible at least according to the latest Salt documentation about pillar (as of 5188d6c) which states:
With some care, the pillar namespace can merge content from multiple pillar files under a single key, so long as conflicts are avoided ...
I tested it under Salt Helium (2014.7.0) and it's working as expected.
Your Example
Pillar file user_set_a.sls:
users:
joe:
sudouser: True
jack:
sudouser: False
Pillar file user_set_b.sls:
users:
new_user:
sudouser: True
Run pillar.items to confirm that all users are merged under the same users key:
salt-call pillar.items
...
users:
----------
jack:
----------
sudouser:
False
joe:
----------
sudouser:
True
new_user:
----------
sudouser:
True
...
See also:
Example to include pillar files under sub-keys: https://serverfault.com/a/591501/134406

Answer 2

Short answer: you can't merge pillar data in this way.
Long answer: the pillar doesn't support the extend keyword the same way the state tree does, though there is some conversation on salt issue #3991. Unfortunately, there doesn't seem to be any real momentum with this at the moment and I'm not aware of any plans for this to be included in Helium.
Realistically, you'd be better off ensuring that your pillar data is distinct on a per-minion basis, and then you won't need to worry about collisions. You could optionally do something with YAML anchors and references, e.g.
# common/base users.sls
base_users: &base_users
user1:
foo: bar
user2:
baz: bat
# minion1.sls
{% include 'common/base_users.sls' %}
users:
<<: *base_users
user3:
qux: quux
# minion2.sls
{% include 'common/base_users.sls' %}
users:
<<: *base_users
user4:
corge: grault
Another potential (hacky) option is to use an external pillar module and do some sort of glob matching on pillar keys provided to the module, so you could basically have keys like merge-thing-abc123 and merge-thing-def456, using the merge prefix to group by thing and combine the data. I wouldn't really recommend this as it's a pretty blatant antipattern WRT pillar data (not to mention difficult to maintain).
For what it's worth, this is something that also frustrates me occasionally, but I end up deciding that some minimal data duplication is better than coming up with a workaround. Using the YAML references, this could potentially be a more agreeable option since technically you don't need to duplicate data, and is more easily maintainable. Granted, you end up polluting the pillar with extra unused keys (e.g. base_users), but in this particular case I'd consider that acceptable.
Hope this helps!
Edit: I may have spoke too soon; it looks as though includes are parsed prior to being injected into the including file, so anchors/references wouldn't work in that case. Looking into it, will update.
Edit 2: Just occurred to me that since both state and pillar files are essentially Python modules, they can be included with Jinja vs using pillar's include. So, instead of
include:
- common.base_users
you can do
{% include 'common/base_users.sls' %}
and then proceed to reference any anchors defined in the included document. Updated the original answer to illustrate this (verified to work).

Answer 3

The way I got around it is by changing the list values to a dict, for example
/srv/pillar/common/packages.sls
packages:
htop: { pkg=installed }
rsync: { pkg=removed }
wget: { pkg=installed }
/srv/pillar/servers/nycweb01.sls
packages:
nginx: { pkg=installed }
checking this servers pillar items you can see it combined the data from both the common pillar and per-node pillar,
salt-ssh nycweb01 pillar.items
nycweb01:
----------
packages:
----------
htop:
----------
pkg=installed:
None
nginx:
----------
pkg=installed:
None
rsync:
----------
pkg=removed:
None
wget:
----------
pkg=installed:
And from the state file, you can use both the pkg name and the pkg state(installed, removed, etc)