ansible flattened map filter results - dictionary

I'm using Ansible's map filter to extract data, but the output is a list of lists; what I need is a flattened list. The closest I've come is illustrated by the "energy.yml" playbook below. Invoke as
ansible-playbook ./energy.yml --extra-vars='src=solar'
---
- hosts: localhost
vars:
region: [ 'east', 'west' ]
sources:
wind:
east:
filenames:
- noreaster.txt
- gusts.txt
- drafty.txt
west:
filenames:
- zephyr.txt
- jetstream.txt
solar:
east:
filenames:
- sunny.txt
- cloudy.txt
west:
filenames:
- blazing.txt
- frybaby.txt
- skynuke.txt
src: wind
tasks:
- name: Do the {{ src }} data
debug:
msg: "tweak file '/energy/{{src}}/{{ item[0] }}/{{ item[1] }}'."
with_nested:
- "{{ region }}"
- "{{
(region|map('extract',sources[src],'filenames')|list)[0] +
(region|map('extract',sources[src],'filenames')|list)[1]
}}"
when: "item[1] in sources[src][item[0]].filenames"
The output of the map() filter is a number of lists the same length as "region". Jinja's "+" operator is the only mechanism I've found to join lists, but since it's a binary operator rather than a filter, I can't apply it to an arbitrary number of lists. The code above depends on "region" having length 2, and having to map() multiple times is ugly in the extreme.
Restructuring the data (or the problem) is not an option. The aspect I'd like to focus on is flattening the map() output, or some other way of generating the correct "msg:" lines the code above does

sum filter with start=[] is your friend:
region | map('extract',sources[src],'filenames') | sum(start=[])
From this:
[
[
"noreaster.txt",
"gusts.txt",
"drafty.txt"
],
[
"zephyr.txt",
"jetstream.txt"
]
]
It will do this:
[
"noreaster.txt",
"gusts.txt",
"drafty.txt",
"zephyr.txt",
"jetstream.txt"
]

Related

Apply map filter to value of dictionary with ansible/jinja2

I'm trying to write an ansible playbook that outputs some details about a system, in a nicely formatted way. In particular, disk sizes.
Input variable looks something like:
- friendly_name: 'disk1 name'
size: 123456
- friendly_name: 'disk2 name'
size: 654321
{{ dict(ansible_facts.disks | json_query('[].[friendly_name, size]')) }}
I'm struggling to come up with a way to apply a function to the 'value' of the dictionary (or the second value of the nested list, prior to converting it to a dict) - I'd like to apply human_readable(unit='G') or similar, without resorting to set_fact or FilterPlugins
So ideally I'd have an output variable of the form:
{'disk1 name': '1024G', 'disk2 name': '8192G'}
You could split the dictionary ansible_facts.disks into two lists, one containing the size and the other one the friendly name, then apply the human_readable filter to the list containing the size with the map filter, then zip the two lists back together.
Given the task:
- debug:
msg: "{{ dict(
ansible_facts.disks | map(attribute='friendly_name') |
zip(ansible_facts.disks | map(attribute='size') | map('human_readable','unit','G'))
) }}"
vars:
ansible_facts:
disks:
- friendly_name: 'disk1 name'
size: 1099511627776
- friendly_name: 'disk2 name'
size: 8796093022208
This yields:
TASK [debug] ********************************************************************
ok: [localhost] => {
"msg": {
"disk1 name": "1024.00 Gb",
"disk2 name": "8192.00 Gb"
}
}
Without the formatting you could simply use items2dict
- debug:
msg: "{{ ansible_facts.disks|items2dict(key_name='friendly_name',
value_name='size') }}"
gives
msg:
disk1 name: 1099511627776
disk2 name: 8796093022208
Use Jinja to change the format, e.g.
- debug:
msg: "{{ _disks|from_yaml }}"
vars:
_disks: |
{% for i in ansible_facts_disks %}
{{ i.friendly_name }}: {{ i.size|human_readable(unit='G') }}
{% endfor %}
gives
msg:
disk1 name: 1024.00 GB
disk2 name: 8192.00 GB

Iterate through a dictionary with subelements in Ansible

I have to create a list based on a dictionary.
To get the element from the dictionary, I need to join "the server" + "the domain". The problem is that I have 3 different domains.
At the moment, I'm repeating the code to be able to use the 3 different domains.
- name: "Get server instances {{ ansible_fqdn }}"
set_fact:
app_ps_mon_list: "{{ app_ps_mon_list | default ([]) + [ app_instance ] }}"
vars:
app_instance: |
[Java,<event_type>]
java critical 1-
*ARGS {{item.value.INSTANCE_NAME.split("/")[1]}}
with_dict: '{{ server_instances[ansible_hostname + "<DOMAIN1>"] }}'
when: '{{server_instances[ansible_hostname + "<DOMAIN1>"] is defined and item.key != "SERVER_IMPACT"}}'
- name: "Get server instances {{ ansible_fqdn }}"
set_fact:
app_ps_mon_list: "{{ app_ps_mon_list | default ([]) + [ app_instance ] }}"
vars:
app_instance: |
[Java,<event_type>]
java critical 1-
*ARGS {{item.value.INSTANCE_NAME.split("/")[1]}}
with_dict: '{{ server_instances[ansible_hostname + "<DOMAIN2>"] }}'
when: '{{server_instances[ansible_hostname + "<DOMAIN2>"] is defined and item.key != "SERVER_IMPACT"}}'
- name: "Get server instances {{ ansible_fqdn }}"
set_fact:
app_ps_mon_list: "{{ app_ps_mon_list | default ([]) + [ app_instance ] }}"
vars:
app_instance: |
[Java,<event_type>]
java critical 1-
*ARGS {{item.value.INSTANCE_NAME.split("/")[1]}}
with_dict: '{{ server_instances[ansible_hostname + "<DOMAIN3>"] }}'
when: '{{server_instances[ansible_hostname + "<DOMAIN3>"] is defined and item.key != "SERVER_IMPACT"}}'
I've been trying to do the same with_subelements without success. Also I tried to use "ansible_fqdn", but the fqdn domain usually don't match the actual domain (I know its a mess).
Is there any workaround I could use to avoid repeating the code?
UPDATE
The idea of the playbook is to obtain the solutions from the server I'm using as host (ansible_hostname varaible).
Once it get the solutions, create a list using some information from the INSTANCE_NAME
This is a generic version of the dictionary:
{
"<server_name><domain>": {
"<solution_id>": {
"INSTANCE_NAME": "",
"SOLUTION_CATEGORY": "",
"SOLUTION_NAME": ""
},
"<solution_id>": {
"INSTANCE_NAME": "",
"SOLUTION_CATEGORY": "",
"SOLUTION_NAME": ""
},
SERVER_IMPACT: "",
...,
}
This is how the list should look like (its a multiline string variable, dumb but useful):
[
[Java]
java critical 1-
*ARGS <iINSTANCE_NAME info>
],
[
[Java]
java critical 1-
*ARGS <iINSTANCE_NAME info>
]

Jinja for constructing dictionary in map: [1, 2] into [{'x': 1}, {'x': 2}]

Most Jinja filters are about reducing the amount of data.
What if I want to make it bigger?
Input
A list. The values could be simple, or complex.
- 1
- 'a'
- ['b', 'c']
- {'d': 'e'}
Desired Output
I want to produce a list.
same length as the input list
Each item in the new list is a dictionary
with one (key,value) pair.
The key is hard coded, the same for every item in the output list.
The value is the corresponding item in the input list.
- x: 1
- x: a
- x: ['b', 'c']
- x: {'d': 'e'}
What I'm looking for is something like
{{ input | map(some_filter, key='x') | list }}
What can I use for some_filter?
Notes
I'm using Ansible for this.
So a solution using JMESPath with the json_query filter is valid.
Similarly Ansible's a solution using dict2items or items2dict somehow would also be valid.
The task below does the job
- debug:
msg: "{{ input|json_query('[*].{x: #}') }}"
gives
msg:
- x: 1
- x: a
- x:
- b
- c
- x:
d: e
dict2items is useless here because the input is a list. Also items2dict is useless here because the result shall be a list too. In addition, dict is also useless because it's not a filter and can't be used in map. Without json_query a loop must be used. For example
- set_fact:
output: "{{ output + [{'x': item}] }}"
loop: "{{ input }}"
vars:
output: []
It's possible to write a filter. For example
shell> cat filter_plugins/item2dict.py
def item2dict(t):
h = {t[0]:t[1]}
return h
class FilterModule(object):
''' Ansible filters. item2dict'''
def filters(self):
return {
'item2dict': item2dict
}
Then the task below gives the same result
- debug:
msg: "{{ 'x'|product(input)|map('item2dict')|list }}"

Combine nested dictionaries in ansible

I have 2 different dictionaries that contains application information I need to join together.
landscape_dictionary:
{
"app_1": {
"Category": "application",
"SolutionID": "194833",
"Availability": null,
"Environment": "stage",
"Vendor/Manufacturer": null
},
"app_2": false
}
app_info_dictionary:
{
"app_1": {
"app_id": "6886817",
"owner": "owner1#nomail.com",
"prod": [
"server1"
],
"stage": []
},
"app_2": {
"app_id": "3415012",
"owner": "owner2#nomail.com",
"prod": [
"server2"
],
"stage": [
"server3"
]
}
}
This is the code I'm using to join both dictionaries
- set_fact:
uber_dict: "{{app_info_dictionary}}"
- set_fact:
uber_dict: "{{ uber_dict | default ({}) | combine(new_item, recursive=true) }}"
vars:
new_item: "{ '{{item.key}}' : { 'landscape': '{{landscape_dictionary[item.key]|default(false)}}' } }"
with_dict: "{{ uber_dict }}"
- debug:
msg: "{{item.key}}: {{item.value}}"
with_dict: "{{uber_dict}}"
If the value in the landscape_dictionary is false it will add it to the uber_dict without problems. But if the value contains information, it fails.
This is the error:
fatal: [127.0.0.1]: FAILED! => {"msg": "|combine expects dictionaries, got u\"{ 'app_1' : { 'landscape': '{u'Category': u'application', u'SolutionID': u'194820', u'Availability': None, u'Environment': 'stage', u'Vendor/Manufacturer': None}' } }\""}
What could be the problem?
Do I need to do an extra combine when I set the var in the set_fact?
Thanks
As #DustWolf notes in the comments,
For anyone from the Internet looking for the answer to: "How tp combine nested dictionaries in ansible", the answer is | combine(new_item, recursive=true)
This solves a closely related issue that has baffled myself and my team for months.
I will demonstrate:
Code:
---
- hosts: localhost
gather_facts: false
vars:
my_default_values:
key1: value1
key2:
a: 10
b: 20
my_custom_values:
key3: value3
key2:
a: 30
my_values: "{{ my_default_values | combine(my_custom_values, recursive=true) }}"
tasks:
- debug: var=my_default_values
- debug: var=my_values
Output:
ok: [localhost] =>
my_values:
key1: value1
key2:
a: 30
key3: value3
Note how key2 was completely replaced, thus losing key2.b
We changed this to:
my_values: "{{ my_default_values | combine(my_custom_values, recursive=true) }}"
Output:
my_values:
key1: value1
key2:
a: 30
b: 20
key3: value3
This syntax is not legal, or at the very least doesn't do what you think:
new_item: "{ '{{item.key}}' : { 'landscape': '{{landscape_dictionary[item.key]|default(false)}}' } }"
Foremost, ansible will only auto-coerce JSON strings into a dict, but you have used python syntax.
Secondarily, the way to dynamically construct a dict is not to use jinja2 to build up text but rather use the fact that jinja2 is almost a programming language:
new_item: "{{
{
item.key: {
'landscape': landscape_dictionary[item.key]|default(false)
}
}
}}"
Any time you find yourself with nested jinja2 interpolation blocks, that is a code smell that you are thinking about the problem too much as text (by nested, I mean {{ something {{nested}} else }})

Create a list of dictionaries from host groups in ansible playbook

I want to dynamically create a list of dictionaries that looks like this:
[ {'host': 'hostname1', 'id': 1}, {'host': 'hostname2', 'id': 2}, ]
And assign it to a variable in my playbook.
This variable is needed for a role I am using.
My attempt is the following:
- hosts:
- some-hosts
vars:
zk_hosts: []
tasks:
- name: create my var
set_fact:
zk_hosts: "{{ zk_hosts + [ {'host': item.1, 'id': item.0} ] }}"
with_indexed_items: "{{ groups.some-hosts }}"
However, when I run the playbook I have this warning:
[WARNING]: While constructing a mapping from stack.yml, line 16, column 3, found a duplicate dict key (vars). Using last defined value only.
And and error at this play:
fatal: [192.168.0.21]: FAILED! => {"failed": true, "msg": "ERROR! 'zk_hosts' is undefined"}
If I don't define zk_hosts before trying to set the fact, I get an error that the variable is undefined.
How can I solve?
EDIT
Easy fix, I just defined zk_hosts within the same task...
tasks:
- name: create my var
vars:
zk_hosts: []
set_fact:
zk_hosts: "{{ zk_hosts + [ {'host': item.1, 'id': item.0} ] }}"
with_indexed_items: "{{ groups.some-hosts }}"
Anyway, if there is a less cumbersome way of achieving the same, please advise!
You can use default filter:
set_fact:
zk_hosts: "{{ zk_hosts|default([]) + [ {'host': item.1, 'id': item.0} ] }}"
with_indexed_items: "{{ groups.some-hosts }}"

Resources