I have a problem with the usage of Jinja templating in Ansible. I have this nested dictionary:
"fruits": {
"summer": {
"june": {
"fruit": {
"cherry": "cherry"
},
"vars": {}
},
"july": {
"fruit": {
"peach": "peach"
},
"vars": {}
},
"august": {
"fruit": {
"watermelon": "watermelon",
"strawberry": "strawberry"
},
"vars": {}
}
}}
and I want to remove the object "peach". I have tried with pop method, and It works, but I want to use the filter rejectattr
I tried:
tasks:
- set_fact:
test: |
{%- for x,y in fruits.items() -%}
{%- for j,k in y.items() -%}
{{ k.fruit | dict2items | rejectattr ('key', 'eq', 'peach') | items2dict }}
{%- endfor -%}
{%- endfor -%}`
It gives me the following:
"test": "{
"cherry": "cherry"}
{}
{"watermelon": "watermelon", "strawberry": "strawberry"}"
But I want to get the starting structure, less "peach" object.
Any suggestion? Thank You
Ansible isn't a great tool for modifying deeply nested data structures (or really for modifying data structures in general). While I'm sure someone is going to drop by with a clever json_query filter or something, this is trivial to solve by writing a custom filter plugin. If we put the following in filter_plugins/badfruit.py:
def filter_badfruit(v, *exclude):
for months in v.values():
for data in months.values():
data["fruit"] = {k: v for k, v in data["fruit"].items() if k not in exclude}
return v
class FilterModule(object):
filter_map = {"badfruit": filter_badfruit}
def filters(self):
return self.filter_map
Then we can write a playbook like this:
- hosts: localhost
gather_facts: false
vars:
"fruits": {
"summer": {
"june": {
"fruit": {
"cherry": "cherry"
},
"vars": {}
},
"july": {
"fruit": {
"peach": "peach"
},
"vars": {}
},
"august": {
"fruit": {
"watermelon": "watermelon",
"strawberry": "strawberry"
},
"vars": {}
}
}}
tasks:
- debug:
var: >-
fruits|badfruit('peach')
And get this output:
PLAY [localhost] ***************************************************************
TASK [debug] *******************************************************************
ok: [localhost] => {
"fruits|badfruit('peach')": {
"summer": {
"august": {
"fruit": {
"strawberry": "strawberry",
"watermelon": "watermelon"
},
"vars": {}
},
"july": {
"fruit": {},
"vars": {}
},
"june": {
"fruit": {
"cherry": "cherry"
},
"vars": {}
}
}
}
}
PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The filter is written so that we can provide multiple keys to exclude:
- debug:
var: >-
fruits|badfruit('peach', 'strawberry')
Q: "I want to use the filter rejectattr."
A: The rejectattr line from your code can be used as is. Combine the dictionaries
test_str: |
summer:
{% for i in fruits.summer|dict2items %}
{{ i.key }}:
{{ i.value|combine({'fruit': i.value.fruit|dict2items|rejectattr('key', 'eq', 'peach')|items2dict}) }}
{% endfor %}
test: "{{ test_str|from_yaml }}"
gives the expected result
test:
summer:
august:
fruit:
strawberry: strawberry
watermelon: watermelon
vars: {}
july:
fruit: {}
vars: {}
june:
fruit:
cherry: cherry
vars: {}
Optionally, put the excluded items into a dictionary, e.g.
exclude:
fruit: ['peach']
and walk the tree. The variable test will expand to the expected result as well
test_str: |
{% for season,sa in fruits.items() %}
{{ season }}:
{% for month,ma in sa.items() %}
{{ month }}:
{% for a,b in ma.items() %}
{{ a }}:
{
{% for k,v in b.items() %}
{% if k not in exclude[a] %}
{{ k }}: {{ v }},
{% endif %}
{% endfor %}
}
{% endfor %}
{% endfor %}
{% endfor %}
test: "{{ test_str|from_yaml }}"
Related
I've a CSV which I converted into a dict using read_csv module.
My aim is to group the data by a field within the dictionary.
For example, in below data I want to use firstname & secondname group by departmentGroup, so that I get a dict array to be re-used later
Below is the input dict created from csv
{
{
"ID02": {
"department": "IT",
"departmentGroup": "IT-development",
"firstname": "first02",
"id": "ID02",
"salary": "40000",
"secondname": "surnam2",
"subDepartment": "development"
}
},
{
"ID03": {
"department": "IT",
"departmentGroup": "IT-development",
"firstname": "first03",
"id": "ID03",
"salary": "42000",
"secondname": "surnam3",
"subDepartment": "development"
}
},
{
"ID04": {
"department": "IT",
"departmentGroup": "IT-operations",
"firstname": "first04",
"id": "ID04",
"salary": "46000",
"secondname": "surnam4",
"subDepartment": "operations"
}
},
{
"ID05": {
"department": "IT",
"departmentGroup": "IT-operations",
"firstname": "first05",
"id": "ID05",
"salary": "42000",
"secondname": "surnam5",
"subDepartment": "operations"
}
}
I'm looking for output of values grouped by departmentGroup, so i can loop it for future tasks to build-up templates
{
"IT-development":
[
{"id": "ID02", "firstname": "first02", "secondname": "surnam2"},
{"id": "ID03", "firstname": "first03", "secondname": "surnam3"}
],
"IT-operations":
[
{"id": "ID04", "firstname": "first04", "secondname": "surnam4"},
{"id": "ID05", "firstname": "first05", "secondname": "surnam5"}
]
}
I've tried using map but couldn't reach much.
Any ideas on how to group in such a way in ansible?
I confirmed that I could do it by using the filters only.
The below is simpler than using Jinja2 code.
---
- name: Example Playbook for 65447303
hosts: localhost
gather_facts: false
tasks:
- name: Read a csv file and converts to dict
read_csv:
path: test.csv
key: name
register: csv_result
- debug: msg="{{ csv_result }}"
- name: Set combined data variable
set_fact:
combined_data: >-
{{ combined_data | default({})
| combine({item.value.departmentGroup: []
+ [{
'id': item.value.id,
'firstname': item.value.firstname,
'secondname': item.value.secondname
}]
+ combined_data[item.value.departmentGroup] | default([]) })
}}
with_dict: "{{ csv_result.dict }}"
- debug: msg="{{ combined_data }}"
I could do it by using Jinja2.
It wasn't easy to do it using the Ansible or Jinja2 filters only.
So I implemented the code using Jinja2.
How about this like?
csv file
(venv) $ vi test.csv
name,department,departmentGroup,firstname,id,salary,secondname,subDepartment
ID02,IT,IT-development,first02,ID02,40000,surnam2,development
ID03,IT,IT-development,first03,ID03,42000,surnam3,development
ID04,IT,IT-operations,first04,ID04,46000,surnam4,operations
ID05,IT,IT-operations,first05,ID05,42000,surnam5,operations
Playbook
(venv) $ vi main.yml
---
- name: Example Playbook for 65447303
hosts: localhost
gather_facts: false
tasks:
- name: Read a csv file and converts to dict
read_csv:
path: test.csv
key: name
register: csv_result
- debug: msg="{{ csv_result }}"
- name: Set combined data variable
set_fact:
combined_data: |
{% set data = {} %}
{% set tmp = {} %}
{# initialize data variable #}
{% for key in csv_result.dict.keys() %}
{% set _ = tmp.update(csv_result.dict[key]) %}
{% set _ = data.update({tmp['departmentGroup']: []}) %}
{# Append the data to each department key #}
{% endfor %}
{% for key in csv_result.dict.keys() %}
{% set _ = tmp.update(csv_result.dict[key]) %}
{% set _ = data[tmp['departmentGroup']].append({
'id': tmp['id'],
'firstname': tmp['firstname'],
'secondname': tmp['secondname']
}) %}
{% endfor %}
{{ data }}
- debug: msg="{{ combined_data | type_debug }}"
- debug: msg="{{ combined_data }}"
Execute and Result
(venv) $ ansible-playbook main.yml
(snip)
TASK [debug] ***************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": {
"IT-development": [
{
"firstname": "first02",
"id": "ID02",
"secondname": "surnam2"
},
{
"firstname": "first03",
"id": "ID03",
"secondname": "surnam3"
}
],
"IT-operations": [
{
"firstname": "first04",
"id": "ID04",
"secondname": "surnam4"
},
{
"firstname": "first05",
"id": "ID05",
"secondname": "surnam5"
}
]
}
}
(snip)
I'm trying to populate dictionary from register output result.containers which happens to override the item.Image with same value.
Here is my output.
"result.containers": [
{
"Image": "lna-docker/webproxy:2.4.0",
"Names": [
"/se-webproxyui-dev-01"
],
},
{
"Image": "lna-docker-dev-local/webproxy:1.8.1",
"Names": [
"/se-webproxy-dev-01"
],
},
{
"Image": "docker-release/consul:1.0.1",
"Names": [
"/consul-client"
],
},
{
"Image": "docker.dev/webproxy:0.6.1",
"Names": [
"/webproxy-dev-01"
],
},
]
}
Here is my code.
- name: Populate dictonary for containerup
set_fact:
containeruplist: "{{ containeruplist|default({}) | combine({item.Image.split('/')[-1].split(':')[0]:item.Names[0][1:]} ) }}"
loop: "{{ result.containers }}"
here is my output
ok: [VM3node.lite.com] => {
"containeruplist": {
"consul": "consul-client",
"webproxy": "webproxy-dev-01"
}
}
I'm trying to populate dictionary from register output result.containers which happens to override the item.Image with same value. Any help would be greatly appreciated.
Ideally I should be getting below output. It's overriding the first two item.Image with value "webroxy" . Could someone help me achieve the below.
containeruplist": {
"webproxy":"se-webproxyui-dev-01"
"webproxy":"se-webproxy-dev-01"
"consul": "consul-client",
"webproxy": "webproxy-dev-01"
}
Taking this cut down example set of inventory hostvars:
{
"_meta": {
"hostvars": {
"host-a.foo.com": {
"host_domain": "foo.com",
"subnet_address": "192.168.1.0",
"subnet_mask": "255.255.254.0"
},
"host-b.foo.com": {
"host_domain": "foo.com",
"subnet_address": "192.168.2.0",
"subnet_mask": "255.255.254.0"
},
"host-c.bar.com": {
"host_domain": "bar.com",
"subnet_address": "192.168.2.0",
"subnet_mask": "255.255.254.0"
}
}
}
}
I'm trying to produce something representing the following in ansible:
[
{
"192.168.1.0": {
"mask": "255.255.254.0",
"domains": [
"foo.com"
]
},
"192.168.2.0": {
"mask": "255.255.254.0",
"domains": [
"foo.com",
"bar.com"
]
}
}
]
So far I have this, but I'm unsure how to append to the domains list for each dictionary item (instead of overwriting it):
- name: inventory subnets
set_fact:
inventory_subnets: "{{ inventory_subnets | default({}) | combine({
hostvars[item].subnet_address: {
'mask': hostvars[item]['subnet_mask'],
'domains': [
# How to build this list?
]
}
}) }}"
loop: "{{ query('inventory_hostnames', 'all') }}"
Is it possible to perform another combine within a dictionary definition somehow?
The tasks below
- set_fact:
hostvars_list: "{{ hostvars_list|default([]) +
[{'host': item.key}|combine(item.value)] }}"
loop: "{{ hostvars|dict2items }}"
- set_fact:
subnets: "{{ subnets|default([]) +
[{item.0: {'mask': item.1|json_query('[].subnet_mask')|first,
'domains': item.1|json_query('[].host_domain')}}] }}"
loop: "{{ hostvars_list|groupby('subnet_address') }}"
- debug:
var: subnets
give
"subnets": [
{
"192.168.1.0": {
"domains": [
"foo.com"
],
"mask": "255.255.254.0"
}
},
{
"192.168.2.0": {
"domains": [
"foo.com",
"bar.com"
],
"mask": "255.255.254.0"
}
}
]
I'm getting serialization errors when retrieving a list of edges from a remote JanusGraph server using Gremlin in a Java application (error message and configurations bellow).
This is the traversal that raises the error:
List<Edge> eList = g.E().toList();
Trying to get only one edge also raises an error:
Edge edge= g.E().next();
Nevertheless, when retrieving a vertex or a list of vertices, it works perfectly:
List<Vertex> vList = g.V().toList();
This is my remote-objects.yaml file:
hosts: [localhost]
port: 8182
serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0,
config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
This is the code to get the remote traversal:
PropertiesConfiguration conf = new PropertiesConfiguration(remote-objects.yaml);
cluster = Cluster.open(conf.getString("gremlin.remote.driver.clusterFile"));
client = cluster.connect();
g = AnonymousTraversalSource.traversal().withRemote(propFileName);
This is the serializers in my server configuration:
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
# Older serialization versions for backwards compatibility:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV1d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV1d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV2d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV1d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV1d0, org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
And this the error message:
java.util.concurrent.CompletionException: io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.kryo.KryoException: java.lang.NegativeArraySizeException
Serialization trace:
id (org.apache.tinkerpop.gremlin.structure.util.reference.ReferenceEdge)
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
CompletableFuture.java:375
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
CompletableFuture.java:1934
at org.apache.tinkerpop.gremlin.driver.ResultSet.one(ResultSet.java:119)
ResultSet.java:119
at org.apache.tinkerpop.gremlin.driver.ResultSet$1.hasNext(ResultSet.java:171)
ResultSet.java:171
at org.apache.tinkerpop.gremlin.driver.ResultSet$1.next(ResultSet.java:178)
ResultSet.java:178
at org.apache.tinkerpop.gremlin.driver.ResultSet$1.next(ResultSet.java:165)
ResultSet.java:165
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteTraversal$TraverserIterator.next(DriverRemoteTraversal.java:140)
DriverRemoteTraversal.java:140
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteTraversal$TraverserIterator.next(DriverRemoteTraversal.java:125)
DriverRemoteTraversal.java:125
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteTraversal.nextTraverser(DriverRemoteTraversal.java:106)
DriverRemoteTraversal.java:106
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.processNextStart(RemoteStep.java:80)
RemoteStep.java:80
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)
AbstractStep.java:128
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)
AbstractStep.java:38
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.fill(Traversal.java:179)
Traversal.java:179
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.toList(Traversal.java:117)
Traversal.java:117
at com.ibm.graph.App.main(App.java:25)
App.java:25
Caused by: io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.kryo.KryoException: java.lang.NegativeArraySizeException
I believe that your problem is in a mismatch of serializer configuration between driver and server. For the driver you do:
serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0,
config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
but for the server you have:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0,
config: { ioRegistries [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0,
org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
You've omitted TinkerIoRegistryV3d0 on the driver side. With ioRegistries the order in which the registries are added matters. So, you might have gotten lucky if you'd added JanusGraphIoRegistry first on the server and then accidentally omitted TinkerIoRegistryV3d0 on the driver because the serializer registration identifiers would have aligned for JansuGraph.
Anyway, add TinkerIoRegistryV3d0 to the driver configuration OR remove it from the server configuration and things should work. As a side note, you typically only need the TinkerIoRegistryV3d0 if you're serializing TinkerGraph instances over the wire (e.g. using subgraph() step).
I have below output from the debug, I need to set variable and loop thru commands
Debug output:
ok: [leafsw] => {
"msg": [
{
"cl_list": "AWSCL",
"delete": [
{
"list": "11111:10000",
"seq": 1
},
{
"list": "22222:10000",
"seq": 2
}
],
"name": "AWSCL",
"permit": [
"11111:10000",
"22222:10000"
]
},
{
"cl_list": "NORM_CL",
"name": "NORM_CL",
"permit": [
"33333:10000",
"44444:10000"
]
}
]
}
I need to fetch cl_list then next task is to use "with_items" to run other commands.
First: how to fetch dict value cl_list
Second: Add to variable so that I can use it a loop.
I tried:
- name: Get CL Name
debug: var="{{ item }}"
with_items: "{{ getclname.cl_list }}"
doesn't worked, also I tried:
- name: Get CL Name
debug: var="{{ item.cl_list }}"
with_items: "{{ getclname }}"
What I would like:
variable = ['AWSCL','NORM_CL'] so that I can use that in with_items loops
Any ideas?
you were almost there! try this task:
- name: get the cl_list from the variable
debug:
var: item.cl_list
with_items:
- "{{ my_var }}"
result:
TASK [get the cl_list from the variable] ****************************************************************************************************************************************************************************
ok: [localhost] => (item=None) => {
"item.cl_list": "AWSCL"
}
ok: [localhost] => (item=None) => {
"item.cl_list": "NORM_CL"
}
PLAY RECAP
its ready to be processed item by item.
SECOND WAY:
you could do this to get them in a list variable:
- name: get the cl_list from the variable
debug:
var: my_var | map(attribute="cl_list") | list
result:
TASK [get the cl_list from the variable] ****************************************************************************************************************************************************************************
ok: [localhost] => {
"my_var | map(attribute=\"cl_list\") | list": [
"AWSCL",
"NORM_CL"
]
}