Combine nested dictionaries in ansible - dictionary

I have 2 different dictionaries that contains application information I need to join together.
landscape_dictionary:
{
"app_1": {
"Category": "application",
"SolutionID": "194833",
"Availability": null,
"Environment": "stage",
"Vendor/Manufacturer": null
},
"app_2": false
}
app_info_dictionary:
{
"app_1": {
"app_id": "6886817",
"owner": "owner1#nomail.com",
"prod": [
"server1"
],
"stage": []
},
"app_2": {
"app_id": "3415012",
"owner": "owner2#nomail.com",
"prod": [
"server2"
],
"stage": [
"server3"
]
}
}
This is the code I'm using to join both dictionaries
- set_fact:
uber_dict: "{{app_info_dictionary}}"
- set_fact:
uber_dict: "{{ uber_dict | default ({}) | combine(new_item, recursive=true) }}"
vars:
new_item: "{ '{{item.key}}' : { 'landscape': '{{landscape_dictionary[item.key]|default(false)}}' } }"
with_dict: "{{ uber_dict }}"
- debug:
msg: "{{item.key}}: {{item.value}}"
with_dict: "{{uber_dict}}"
If the value in the landscape_dictionary is false it will add it to the uber_dict without problems. But if the value contains information, it fails.
This is the error:
fatal: [127.0.0.1]: FAILED! => {"msg": "|combine expects dictionaries, got u\"{ 'app_1' : { 'landscape': '{u'Category': u'application', u'SolutionID': u'194820', u'Availability': None, u'Environment': 'stage', u'Vendor/Manufacturer': None}' } }\""}
What could be the problem?
Do I need to do an extra combine when I set the var in the set_fact?
Thanks

As #DustWolf notes in the comments,
For anyone from the Internet looking for the answer to: "How tp combine nested dictionaries in ansible", the answer is | combine(new_item, recursive=true)
This solves a closely related issue that has baffled myself and my team for months.
I will demonstrate:
Code:
---
- hosts: localhost
gather_facts: false
vars:
my_default_values:
key1: value1
key2:
a: 10
b: 20
my_custom_values:
key3: value3
key2:
a: 30
my_values: "{{ my_default_values | combine(my_custom_values, recursive=true) }}"
tasks:
- debug: var=my_default_values
- debug: var=my_values
Output:
ok: [localhost] =>
my_values:
key1: value1
key2:
a: 30
key3: value3
Note how key2 was completely replaced, thus losing key2.b
We changed this to:
my_values: "{{ my_default_values | combine(my_custom_values, recursive=true) }}"
Output:
my_values:
key1: value1
key2:
a: 30
b: 20
key3: value3

This syntax is not legal, or at the very least doesn't do what you think:
new_item: "{ '{{item.key}}' : { 'landscape': '{{landscape_dictionary[item.key]|default(false)}}' } }"
Foremost, ansible will only auto-coerce JSON strings into a dict, but you have used python syntax.
Secondarily, the way to dynamically construct a dict is not to use jinja2 to build up text but rather use the fact that jinja2 is almost a programming language:
new_item: "{{
{
item.key: {
'landscape': landscape_dictionary[item.key]|default(false)
}
}
}}"
Any time you find yourself with nested jinja2 interpolation blocks, that is a code smell that you are thinking about the problem too much as text (by nested, I mean {{ something {{nested}} else }})

Related

ansible json_query to with dicts containing key values as list

I have a below value in a variable allcsv
"msg": [
{
"added_bundle_images": [
"locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.8.3-10"
],
"index_image": "locuz.qe.pnq.local/eng-build/iib:297699",
"locuz_version": "v4.8"
},
{
"added_bundle_images": [
"locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.10.0-57"
],
"index_image": "locuz.qe.pnq.local/eng-build/iib:297697",
"locuz_version": "v4.9"
},
{
"added_bundle_images": [
"locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.7.7-14"
],
"index_image": "locuz.qe.pnq.local/eng-build/iib:297497",
"locuz_version": "v4.7"
},
{
"added_bundle_images": [
"locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.9.2-3"
],
"index_image": "locuz.qe.pnq.local/eng-build/iib:297495",
"locuz_version": "v4.9"
}
]
I have a ansible variable which contains one of the values from added_bundle_images. for example i have a variable called lastcsv which contains value "locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.7.7-14"
I am trying to write a json_query and also trying selectattr to get index_image if my key value matches "locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.7.7-14"
- name: set my variable
set_fact:
lastcsv: "locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.7.7-14"
- name: my debug2
debug: msg="{{ allcsv | selectattr('added_bundle_images','equalto', [lastcsv]) | list }}"
The output i see is :
TASK [common : my debug2] ***************************************************************************************************************************************************
ok: [foo.example.com] => {
"msg": []
}
Need help in constructing the query so that i get the index_image value when the added_bundle_image key has value "locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.7.7-14"
In allcsv, added_bundle_images is a list (denoted by []), so we can use contains match rather than equalto to see if the lastcsv element is in that list. Something like below:
- name: set my variable
set_fact:
lastcsv: "locuz.qe.pnq.local/eng-build/lzfirewallmod:v4.7.7-14"
- name: my debug2
debug:
msg: "{{ allcsv | selectattr('added_bundle_images', 'contains', lastcsv) | map(attribute='index_image') | list }}"
Produces:
TASK [my debug2] ********************************************************************************************************************************************
ok: [localhost] => {
"msg": [
"locuz.qe.pnq.local/eng-build/iib:297497"
]
}

ansible changing nested dict variable

After deployment of a VM with a DHCP IP I would like to get the IP and append it to the guests dictionary.
For the first VM (testvm2) the code perfoms as expected and updates the tempip variable for the testvm2 VM.
But with the second VM (testvm1), it updates the tempip variable of the first VM (testvm2) with the IP of the second VM (testvm1), and updates the tempip variable of the second VM (testvm1) with the code of the variable '{{ tempip_reg.stdout_lines }}'
Can anyone explain to me why this happens?
I would appreciate the help.
I copied all the relevant code and output below:
guests dictionary:
---
guests:
testvm1:
mem: 512
cpus: 1
clone: template-centos
vmid: 102
tempip:
testvm2:
mem: 1536
cpus: 2
clone: template-centos
vmid: 102
tempip:
Ansible Playbook that starts the task:
---
- name: Provision VMs
hosts: infra
become: true
vars_files:
- group_vars/vms.yml
- group_vars/vars.yml
tasks:
- include_tasks: roles/tasks/provision-tasks.yml
with_dict: "{{ guests }}"
Ansible Tasks:
- name: Run proxmox-get-ip-linux.sh script to register DHCP IP of VM
script: proxmox-get-ip-linux.sh "{{ item.key }}"
register: tempip_reg
- name: temporary IP of VM "{{ item.key }}"
debug:
var: tempip_reg
- name: current host in item.key
set_fact:
current_host: "{{ item.key }}"
- name: current_host variable set to
debug:
var: current_host
- name: append item.value.tempip with the DHCP IP of the VM registered in
last task
set_fact:
guests: "{{ guests|combine({ current_host: {'tempip': '{{ tempip_reg.stdout_lines }}' }}, recursive=True) }}"
- name: temporary IP of "{{ item.key }}"
debug: var=guests
Result first VM:
"tempip_reg": {
"stdout": "192.168.1.21\r\n",
"stdout_lines": [
"192.168.1.21"
}
"current_host": "testvm2"
"guests": {
"testvm1": {
"clone": "template-centos",
"cpus": 1,
"ip": "192.168.1.60",
"mem": 512,
"tempip": null,
"vmid": 102
},
"testvm2": {
"clone": "template-centos",
"cpus": 2,
"ip": "192.168.1.61",
"mem": 1536,
"tempip": [
"192.168.1.21"
],
"vmid": 102
}
}
Result 2nd VM:
"tempip_reg": {
"stdout": "192.168.1.22\r\n",
"stdout_lines": [
"192.168.1.22"
}
"current_host": "testvm1"
"guests": {
"testvm1": {
"clone": "template-centos",
"cpus": 1,
"ip": "192.168.1.60",
"mem": 512,
"tempip": "{{ tempip_reg.stdout_lines }}",
"vmid": 102
},
"testvm2": {
"clone": "template-centos",
"cpus": 2,
"ip": "192.168.1.61",
"mem": 1536,
"tempip": [
"192.168.1.22"
],
"vmid": 102
}
}
TL;DR
Using Ansible code, you are trying to implement what Ansible already does for you.
Your attempts superimpose with built-in functionality and you get results which look nondeterministic.
Explanation:
The main problem with your code is a completely unnecessary loop declared with with_dict: "{{ guests }}" which causes to include the file 4 times.
It runs 4 times because you change the guests dictionary, which it loops over inside the included tasks-file.
In effect you get something which looks like an nondeterministic result.
The second problem is a trivial one: you always replace the value of tempip with a string {{ tempip_reg.stdout_lines }}.
Now, because of the unnecessary with_dict loop over a dictionary which you dynamically change, and because Jinja2 uses lazy variable evaluation, strings from previous iterations are interpreted as templates and get evaluated with incorrect values in subsequent iterations.
The last iteration leaves the string {{ tempip_reg.stdout_lines }} intact.
You also define and print two different facts.
What you should do:
You should not declare arbitrary iterations at all. Ansible implements a loop for all hosts itself. That is, if you declare a task:
- include_tasks: roles/tasks/provision-tasks.yml
the file will be included for each of the hosts in infra group (twice in your example).
You seem to want to have a single copy of your data structure with updated values for each VM.
At the same time, you create a fact, which is a separate data object maintained for each host separately.
So you should refer to and modify (combine) a single fact - you can do it for example on localhost.
You should structure your code like this:
---
- name: Provision VMs
hosts: infra
become: true
vars_files:
- group_vars/vms.yml
- group_vars/vars.yml
tasks:
- include_tasks: roles/tasks/provision-tasks.yml
- debug:
var: hostvars['localhost'].guests
and provision-tasks.yml:
- set_fact:
guests: "{{ guests|combine({ current_host: {'tempip': tempip_reg.stdout_lines }}, recursive=True) }}"
delegate_to: localhost
This will get you the following result:
"hostvars['localhost'].guests": {
"testvm1": {
"clone": "template-centos",
"cpus": 1,
"ip": "192.168.1.60",
"mem": 512,
"tempip": [
"192.168.1.21"
],
"vmid": 102
},
"testvm2": {
"clone": "template-centos",
"cpus": 2,
"ip": "192.168.1.61",
"mem": 1536,
"tempip": [
"192.168.1.22"
],
"vmid": 102
}
}
Finally, in the above play, you used group_vars and roles/tasks directories in wrong context. I left the paths intact and they will work for the above code, but basically you should never use them this way, because again, they have special meaning and treatment in Ansible.

How to filter dictionaries in Jinja?

I have a dictionary of packages with package-name being the key and a dictionary of some details being the value:
{
"php7.1-readline": {
"latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"origins": [
"ppa.launchpad.net"
],
"version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1",
"www": "http://www.php.net/"
},
"php7.1-xml": {
"latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"origins": [
"ppa.launchpad.net"
],
"version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1",
"www": "http://www.php.net/"
},
"plymouth": {
"version": "0.8.8-0ubuntu17.1"
},
....
}
I'd like to reduce the above to a dictionary with only the packages, that have the latest-attribute in their values.
It would seem like json_query is the filter to use, but I can't figure out the syntax. The examples out there all seem to operate on lists of dictionaries, not dictionaries of same...
For example, if I "pipe" the above dictionary into json_query('*.latest'), I get the list of the actual latest versions:
[
"7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"7.1.6-2~ubuntu14.04.1+deb.sury.org+1"
]
How can I get the entire dictionary-elements instead?
Any hope?
With dict2items filter added in December 2017, it is possible using native functionality:
- debug:
msg: "{{ dict(pkg | dict2items | json_query('[?value.latest].[key, value.latest]')) }}"
The result:
"msg": {
"php7.1-readline": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"php7.1-xml": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1"
}
You can't perform this translation (I think) exclusively with Jinja filters, but you can get there by applying a little Ansible logic as well. The following playbook uses a with_dict loop to loop over the items in your dictionary, and build a new dictionary from matching ones:
- hosts: localhost
vars:
packages: {
"php7.1-readline": {
"latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"origins": [
"ppa.launchpad.net"
],
"version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1",
"www": "http://www.php.net/"
},
"php7.1-xml": {
"latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"origins": [
"ppa.launchpad.net"
],
"version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1",
"www": "http://www.php.net/"
},
"plymouth": {
"version": "0.8.8-0ubuntu17.1"
}
}
tasks:
- set_fact:
new_packages: >
{{ new_packages|default({})|
combine({item.key: item.value}) }}
with_dict: "{{ packages }}"
when: "{{ item.value.latest is defined }}"
- debug:
var: new_packages
You are correct to link this question to https://stackoverflow.com/a/41584889/2795592.
There are no options to manipulate keys and values simultaneously with json_query out of the box (as of Ansible 2.4.0).
Here's patched json_query.py that supports jq-like to_entries/from_entries functions.
You can put it into ./filter_plugins near your playbook and make this query:
- debug:
msg: "{{ pkg | json_query('to_entries(#) | [?value.latest].{key:key, value:value.latest} | from_entries(#)')}}"
to get this result:
"msg": {
"php7.1-readline": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
"php7.1-xml": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1"
}
I'll make PR to ansible as soon as I have some spare time.

How to remove a single key from an Ansible dictionary?

I'd like to remove a single key from a dictionary in Ansible.
For example, I'd like this:
- debug: var=dict2
vars:
dict:
a: 1
b: 2
c: 3
dict2: "{{ dict | filter_to_remove_key('a') }}"
To print this:
ok: [localhost] => {
"dict2": {
"b": 2,
"c": 3
}
}
Please note that the dictionary is loaded from a json file and I POST it to the Grafana REST API. I'd like to allow saving an 'id' key in the file and remove the key before POSTing it.
This is closer to the actual use I have for the removal:
- name: Install Dashboards
uri:
url: "{{ grafana_api_url }}/dashboards/db"
method: POST
headers:
Authorization: Bearer {{ grafana_api_token }}
body:
overwrite: true
dashboard:
"{{ lookup('file', item) | from_json | removekey('id') }}"
body_format: json with_fileglob:
- "dashboards/*.json"
- "../../../dashboards/*.json"
- set_fact:
dict:
a: 1
b: 2
c: 3
dict2: {}
- set_fact:
dict2: "{{dict2 |combine({item.key: item.value})}}"
when: "{{item.key not in ['a']}}"
with_dict: "{{dict}}"
- debug: var=dict2
or create a filter plugin and use it.
Here's an approach inspired by an article by John Mazzitelli that can be used inline without additional set_fact tasks, etc.:
Task:
tasks:
- debug: var=dict2
vars:
dict:
a: 1
b: 2
c: 3
# It is important that there be NO WHITESPACE outside of `{% ... %}` and `{{ ... }}`
# or else the var will be converted to a string. The copy() step optionally prevents
# modifying the original. If you don't care, then: "{% set removed=dict.pop('a') %}{{dict}}"
dict2: "{% set copy=dict.copy() %}{% set removed=copy.pop('a') %}{{ copy }}"
Outputs:
TASK [debug] ***********
ok: [localhost] => {
"dict2": {
"b": 2,
"c": 3
}
}
- debug: var=dict2
vars:
dict:
a: 1
b: 2
c: 3
dict2: '{{ dict | dict2items | rejectattr("key", "eq", "a") | list | items2dict }}'
#dict2: '{{ dict | dict2items | rejectattr("key", "match", "^(a|b)$") | list | items2dict }}'
Output:
ok: [localhost] => {
"dict2": {
"b": 2,
"c": 3
}
}
If you interested in filter (which is in my opinion the cleanest way to delete an item in dict) then create filter_plugins/dicts.py in directory, which your playbook resides in, and fill it with:
'''Custom ansible filters for dicts'''
import copy
class FilterModule(object):
def filters(self):
return {
'del_by_list': self.del_by_list
}
def del_by_list(self, dict_to_reduce, list_of_keys):
'''Deletes items of dict by list of keys provided'''
dict_to_return = copy.deepcopy(dict_to_reduce)
for item in list_of_keys:
if item in dict_to_return:
del dict_to_return[item]
return dict_to_return
And you good to go:
---
- hosts: hadoop
gather_facts: no
tasks:
- debug:
msg: "{{ {123: 456, 789: 123} | del_by_list([123]) }}"
This will yield {789: 123}
You can accomplish this by using
combine and omit
- set_fact:
dict:
a: 1
b: 2
c: 3
- debug:
var: dict
- debug:
msg: "{{ dict | combine({ 'a': omit }) }}"
TASK [test : set_fact]
ok: [server]
TASK [test: debug]
ok: [server] => {
"dict": {
"a": 1,
"b": 2,
"c": 3
}
}
TASK [test : debug]
ok: [server] => {
"msg": {
"b": 2,
"c": 3
}
}

Create a list of dictionaries from host groups in ansible playbook

I want to dynamically create a list of dictionaries that looks like this:
[ {'host': 'hostname1', 'id': 1}, {'host': 'hostname2', 'id': 2}, ]
And assign it to a variable in my playbook.
This variable is needed for a role I am using.
My attempt is the following:
- hosts:
- some-hosts
vars:
zk_hosts: []
tasks:
- name: create my var
set_fact:
zk_hosts: "{{ zk_hosts + [ {'host': item.1, 'id': item.0} ] }}"
with_indexed_items: "{{ groups.some-hosts }}"
However, when I run the playbook I have this warning:
[WARNING]: While constructing a mapping from stack.yml, line 16, column 3, found a duplicate dict key (vars). Using last defined value only.
And and error at this play:
fatal: [192.168.0.21]: FAILED! => {"failed": true, "msg": "ERROR! 'zk_hosts' is undefined"}
If I don't define zk_hosts before trying to set the fact, I get an error that the variable is undefined.
How can I solve?
EDIT
Easy fix, I just defined zk_hosts within the same task...
tasks:
- name: create my var
vars:
zk_hosts: []
set_fact:
zk_hosts: "{{ zk_hosts + [ {'host': item.1, 'id': item.0} ] }}"
with_indexed_items: "{{ groups.some-hosts }}"
Anyway, if there is a less cumbersome way of achieving the same, please advise!
You can use default filter:
set_fact:
zk_hosts: "{{ zk_hosts|default([]) + [ {'host': item.1, 'id': item.0} ] }}"
with_indexed_items: "{{ groups.some-hosts }}"

Resources