dict(zip()) function compressing two lists in Python - dictionary

These two lists imported from json file and I can't use dict(zip() properly.
l1 = ['grant_type', 'scope', 'client_id', 'client_id', 'grant_type', 'scope', 'code', 'redirect_uri']
l2 = ['client_credentials', 'AIS', '{{client_id}}', '{{client_id}}', 'authorization_code', '{{scope}}:{{consents_id}}', '{{response}}', '{{redirect_url}}']
lists = dict(zip(l1, l2))
print(lists)
Output:
{'grant_type': 'authorization_code', 'scope': '{{scope}}:{{consents_id}}', 'client_id': '{{client_id}}', 'code': '{{response}}', 'redirect_uri': '{{redirect_url}}'}
I guess there is something with double curly brackets "{{ }}", but I didn't find any information about them and how I can solve this problem.
Thank you for the response!

Related

Jinja for constructing dictionary in map: [1, 2] into [{'x': 1}, {'x': 2}]

Most Jinja filters are about reducing the amount of data.
What if I want to make it bigger?
Input
A list. The values could be simple, or complex.
- 1
- 'a'
- ['b', 'c']
- {'d': 'e'}
Desired Output
I want to produce a list.
same length as the input list
Each item in the new list is a dictionary
with one (key,value) pair.
The key is hard coded, the same for every item in the output list.
The value is the corresponding item in the input list.
- x: 1
- x: a
- x: ['b', 'c']
- x: {'d': 'e'}
What I'm looking for is something like
{{ input | map(some_filter, key='x') | list }}
What can I use for some_filter?
Notes
I'm using Ansible for this.
So a solution using JMESPath with the json_query filter is valid.
Similarly Ansible's a solution using dict2items or items2dict somehow would also be valid.

The task below does the job
- debug:
msg: "{{ input|json_query('[*].{x: #}') }}"
gives
msg:
- x: 1
- x: a
- x:
- b
- c
- x:
d: e
dict2items is useless here because the input is a list. Also items2dict is useless here because the result shall be a list too. In addition, dict is also useless because it's not a filter and can't be used in map. Without json_query a loop must be used. For example
- set_fact:
output: "{{ output + [{'x': item}] }}"
loop: "{{ input }}"
vars:
output: []
It's possible to write a filter. For example
shell> cat filter_plugins/item2dict.py
def item2dict(t):
h = {t[0]:t[1]}
return h
class FilterModule(object):
''' Ansible filters. item2dict'''
def filters(self):
return {
'item2dict': item2dict
}
Then the task below gives the same result
- debug:
msg: "{{ 'x'|product(input)|map('item2dict')|list }}"

Parsing JSON dict of CloudFormation parameters for '--parameter-overrides'

I'm using AWS CloudFormation at the moment, and I need to parse out parameters due to differences between stack creation and deployment. Command aws cloudformation create accepts a JSON file, but aws cloudformation deploy only accepts inlined application parameters of Key=Value type.
I have this JSON file:
[
{
"ParameterKey": "EC2KeyPair",
"ParameterValue": "$YOUR_EC2_KEY_PAIR"
},
{
"ParameterKey": "SSHLocation",
"ParameterValue": "$YOUR_SSH_LOCATION"
},
{
"ParameterKey": "DjangoEnvVarDebug",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DEBUG"
},
{
"ParameterKey": "DjangoEnvVarSecretKey",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_SECRET_KEY"
},
{
"ParameterKey": "DjangoEnvVarDBName",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_NAME"
},
{
"ParameterKey": "DjangoEnvVarDBUser",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_USER"
},
{
"ParameterKey": "DjangoEnvVarDBPassword",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_PASSWORD"
},
{
"ParameterKey": "DjangoEnvVarDBHost",
"ParameterValue": "$YOUR_DJANGO_ENV_VAR_DB_HOST"
}
]
And I want to turn it into this:
'EC2KeyPair=$YOUR_EC2_KEY_PAIR SSHLocation=$YOUR_SSH_LOCATION DjangoEnvVarDebug=$YOUR_DJANGO_ENV_VAR_DEBU
G DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY DjangoEnvVarDBName=$YOUR_DJANGO_ENV_VAR_DB_NAME D
jangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER DjangoEnvVarDBPassword=$YOUR_DJANGO_ENV_VAR_DB_PASSWORD Dj
angoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST'
This would be the equivalent Python code:
thing = json.load(open('stack-params.example.json', 'r'))
convert = lambda item: f'{item["ParameterKey"]}={item["ParameterValue"]}'
>>> print(list(map(convert, thing)))
['EC2KeyPair=$YOUR_EC2_KEY_PAIR', 'SSHLocation=$YOUR_SSH_LOCATION', 'DjangoEnvVarDebug=$YOUR_DJANGO_ENV_V
AR_DEBUG', 'DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY', 'DjangoEnvVarDBName=$YOUR_DJANGO_ENV_
VAR_DB_NAME', 'DjangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER', 'DjangoEnvVarDBPassword=$YOUR_DJANGO_EN$
_VAR_DB_PASSWORD', 'DjangoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST']
>>> ' '.join(map(convert, thing))
'EC2KeyPair=$YOUR_EC2_KEY_PAIR SSHLocation=$YOUR_SSH_LOCATION DjangoEnvVarDebug=$YOUR_DJANGO_ENV_VAR_DEBU
G DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY DjangoEnvVarDBName=$YOUR_DJANGO_ENV_VAR_DB_NAME D
jangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER DjangoEnvVarDBPassword=$YOUR_DJANGO_ENV_VAR_DB_PASSWORD Dj
angoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST'
I have this little snippet:
$ cat stack-params.example.json | jq '.[] | "\(.ParameterKey)=\(.ParameterValue)"'
"EC2KeyPair=$YOUR_EC2_KEY_PAIR"
"SSHLocation=$YOUR_SSH_LOCATION"
"DjangoEnvVarDebug=$YOUR_DJANGO_ENV_VAR_DEBUG"
"DjangoEnvVarSecretKey=$YOUR_DJANGO_ENV_VAR_SECRET_KEY"
"DjangoEnvVarDBName=$YOUR_DJANGO_ENV_VAR_DB_NAME"
"DjangoEnvVarDBUser=$YOUR_DJANGO_ENV_VAR_DB_USER"
"DjangoEnvVarDBPassword=$YOUR_DJANGO_ENV_VAR_DB_PASSWORD"
"DjangoEnvVarDBHost=$YOUR_DJANGO_ENV_VAR_DB_HOST"
But I'm not sure how to join the strings together. I was looking at reduce but I think it only works on lists, and streams of strings aren't lists. So I'm thinking the correct approach is to convert the key : value association into 'key=value' strings within the list, then join altogether, though I have trouble working with the regex. Does anybody have any tips?

The goal as exemplified by the illustrative output seems highly dubious, but it can easily be achieved using the -r command-line option together with the filter:
map("\(.ParameterKey)=\(.ParameterValue)") | "'" + join(" ") + "'"
Footnote
I was looking at reduce but I think it only works on lists, and streams of strings aren't lists.
To use reduce on a list, say $l, you could simply use [] as in:
reduce $l[] as $x (_;_)

Saltstack running a bash script through cmd.script via rest_cherrypy with multiple args

Trying to use rest_cherrypy to run a bash script on minion. Script takes multiple args. In the following example it works fine with one argument:
resp = session.post('https://192.168.50.10:8000/', json=[{
'client': 'local',
'tgt': 'minion1',
'fun': 'cmd.script',
'arg': ['salt://scripts/my_script.sh', 'arg1'],
}])
return resp.json()
Output: {'return': [{'minion1': {'pid': 2836, 'retcode': 0, 'stderr': '', 'stdout': 'arg1'}}]}
However trying to use multiple arguments gives me this:
resp = session.post('https://192.168.50.10:8000/', json=[{
'client': 'local',
'tgt': 'minion1',
'fun': 'cmd.script',
'arg': ['salt://scripts/my_script.sh', ('arg1', 'arg2')],
}])
return resp.json()
Output:
{'return': [{'minion1': {'pid': 2808, 'retcode': 0, 'stderr': '', 'stdout': '[uarg1,'}}]}
What's the correct way to pass multiple arguments to a cmd.script function through rest_cherrypy?

Managed to solve it this way:
resp = session.post('https://192.168.50.10:8000/', json=[{
'client': 'local',
'tgt': 'minion1',
'fun': 'cmd.script',
'arg': ['salt://scripts/my_script.sh', """ {}, {}, {}, {} """.format(mail_from, mail_to, date_from, date_to)],
}])

ansible flattened map filter results

I'm using Ansible's map filter to extract data, but the output is a list of lists; what I need is a flattened list. The closest I've come is illustrated by the "energy.yml" playbook below. Invoke as
ansible-playbook ./energy.yml --extra-vars='src=solar'
---
- hosts: localhost
vars:
region: [ 'east', 'west' ]
sources:
wind:
east:
filenames:
- noreaster.txt
- gusts.txt
- drafty.txt
west:
filenames:
- zephyr.txt
- jetstream.txt
solar:
east:
filenames:
- sunny.txt
- cloudy.txt
west:
filenames:
- blazing.txt
- frybaby.txt
- skynuke.txt
src: wind
tasks:
- name: Do the {{ src }} data
debug:
msg: "tweak file '/energy/{{src}}/{{ item[0] }}/{{ item[1] }}'."
with_nested:
- "{{ region }}"
- "{{
(region|map('extract',sources[src],'filenames')|list)[0] +
(region|map('extract',sources[src],'filenames')|list)[1]
}}"
when: "item[1] in sources[src][item[0]].filenames"
The output of the map() filter is a number of lists the same length as "region". Jinja's "+" operator is the only mechanism I've found to join lists, but since it's a binary operator rather than a filter, I can't apply it to an arbitrary number of lists. The code above depends on "region" having length 2, and having to map() multiple times is ugly in the extreme.
Restructuring the data (or the problem) is not an option. The aspect I'd like to focus on is flattening the map() output, or some other way of generating the correct "msg:" lines the code above does

sum filter with start=[] is your friend:
region | map('extract',sources[src],'filenames') | sum(start=[])
From this:
[
[
"noreaster.txt",
"gusts.txt",
"drafty.txt"
],
[
"zephyr.txt",
"jetstream.txt"
]
]
It will do this:
[
"noreaster.txt",
"gusts.txt",
"drafty.txt",
"zephyr.txt",
"jetstream.txt"
]

Ejabberd: error in simple module to handle offline messages

I have an Ejabberd 17.01 installation where I need to push a notification in case a recipient is offline. This seems the be a common task and solutions using a customized Ejabberd module can be found everywhere. However, I just don't get it running. First, here's me script:
-module(mod_offline_push).
-behaviour(gen_mod).
-export([start/2, stop/1]).
-export([push_message/3]).
-include("ejabberd.hrl").
-include("logger.hrl").
-include("jlib.hrl").
start(Host, _Opts) ->
?INFO_MSG("mod_offline_push loading", []),
ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_message, 10),
ok.
stop(Host) ->
?INFO_MSG("mod_offline_push stopping", []),
ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_message, 10),
ok.
push_message(From, To, Packet) ->
?INFO_MSG("mod_offline_push -> push_message", [To]),
Type = fxml:get_tag_attr_s(<<"type">>, Packet), % Supposedly since 16.04
%Type = xml:get_tag_attr_s(<<"type">>, Packet), % Supposedly since 13.XX
%Type = xml:get_tag_attr_s("type", Packet),
%Type = xml:get_tag_attr_s(list_to_binary("type"), Packet),
?INFO_MSG("mod_offline_push -> push_message", []),
ok.
The problem is the line Type = ... line in method push_message; without that line the last info message is logged (so the hook definitely works). When browsing online, I can find all kinds of function calls to extract elements from Packet. As far as I understand it changed over time with new releases. But it's not good, all variants lead in some kind of error. The current way returns:
2017-01-25 20:38:08.701 [error] <0.21678.0>#ejabberd_hooks:run1:332 {function_clause,[{fxml,get_tag_attr_s,[<<"type">>,{message,<<>>,normal,<<>>,{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>},[],[{text,<<>>,<<"sfsdfsdf">>}],undefined,[],#{}}],[{file,"src/fxml.erl"},{line,169}]},{mod_offline_push,push_message,3,[{file,"mod_offline_push.erl"},{line,33}]},{ejabberd_hooks,safe_apply,3,[{file,"src/ejabberd_hooks.erl"},{line,382}]},{ejabberd_hooks,run1,3,[{file,"src/ejabberd_hooks.erl"},{line,329}]},{ejabberd_sm,route,3,[{file,"src/ejabberd_sm.erl"},{line,126}]},{ejabberd_local,route,3,[{file,"src/ejabberd_local.erl"},{line,110}]},{ejabberd_router,route,3,[{file,"src/ejabberd_router.erl"},{line,87}]},{ejabberd_c2s,check_privacy_route,5,[{file,"src/ejabberd_c2s.erl"},{line,1886}]}]}
running hook: {offline_message_hook,[{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>},{message,<<>>,normal,<<>>,{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>},[],[{text,<<>>,<<"sfsdfsdf">>}],undefined,[],#{}}]}
I'm new Ejabberd and Erlang, so I cannot really interpret the error, but the Line 33 as mentioned in {mod_offline_push,push_message,3,[{file,"mod_offline_push.erl"}, {line,33}]} is definitely the line calling get_tag_attr_s.
UPDATE 2017/01/27: Since this cost me a lot of headache -- and I'm still not perfectly happy -- I post here my current working module in the hopes it might help others. My setup is Ejabberd 17.01 running on Ubuntu 16.04. Most stuff I tried and failed with seem to for older versions of Ejabberd:
-module(mod_fcm_fork).
-behaviour(gen_mod).
%% public methods for this module
-export([start/2, stop/1]).
-export([push_notification/3]).
%% included for writing to ejabberd log file
-include("ejabberd.hrl").
-include("logger.hrl").
-include("xmpp_codec.hrl").
%% Copied this record definition from jlib.hrl
%% Including "xmpp_codec.hrl" and "jlib.hrl" resulted in errors ("XYZ already defined")
-record(jid, {user = <<"">> :: binary(),
server = <<"">> :: binary(),
resource = <<"">> :: binary(),
luser = <<"">> :: binary(),
lserver = <<"">> :: binary(),
lresource = <<"">> :: binary()}).
start(Host, _Opts) ->
?INFO_MSG("mod_fcm_fork loading", []),
% Providing the most basic API to the clients and servers that are part of the Inets application
inets:start(),
% Add hook to handle message to user who are offline
ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_notification, 10),
ok.
stop(Host) ->
?INFO_MSG("mod_fcm_fork stopping", []),
ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_notification, 10),
ok.
push_notification(From, To, Packet) ->
% Generate JID of sender and receiver
FromJid = lists:concat([binary_to_list(From#jid.user), "#", binary_to_list(From#jid.server), "/", binary_to_list(From#jid.resource)]),
ToJid = lists:concat([binary_to_list(To#jid.user), "#", binary_to_list(To#jid.server), "/", binary_to_list(To#jid.resource)]),
% Get message body
MessageBody = Packet#message.body,
% Check of MessageBody is not empty
case MessageBody/=[] of
true ->
% Get first element (no idea when this list can have more elements)
[First | _ ] = MessageBody,
% Get message data and convert to string
MessageBodyText = binary_to_list(First#text.data),
send_post_request(FromJid, ToJid, MessageBodyText);
false ->
?INFO_MSG("mod_fcm_fork -> push_notification: MessageBody is empty",[])
end,
ok.
send_post_request(FromJid, ToJid, MessageBodyText) ->
%?INFO_MSG("mod_fcm_fork -> send_post_request -> MessageBodyText = ~p", [Demo]),
Method = post,
PostURL = gen_mod:get_module_opt(global, ?MODULE, post_url,fun(X) -> X end, all),
% Add data as query string. Not nice, query body would be preferable
% Problem: message body itself can be in a JSON string, and I couldn't figure out the correct encoding.
URL = lists:concat([binary_to_list(PostURL), "?", "fromjid=", FromJid,"&tojid=", ToJid,"&body=", edoc_lib:escape_uri(MessageBodyText)]),
Header = [],
ContentType = "application/json",
Body = [],
?INFO_MSG("mod_fcm_fork -> send_post_request -> URL = ~p", [URL]),
% ADD SSL CONFIG BELOW!
%HTTPOptions = [{ssl,[{versions, ['tlsv1.2']}]}],
HTTPOptions = [],
Options = [],
httpc:request(Method, {URL, Header, ContentType, Body}, HTTPOptions, Options),
ok.

Actually it fails with second arg Packet you pass to fxml:get_tag_attr_s in push_message function
{message,<<>>,normal,<<>>,
{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,
<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},
{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,
<<"xxx.xxx.xxx.xxx">>,<<>>},
[],
[{text,<<>>,<<"sfsdfsdf">>}],
undefined,[],#{}}
because it is not xmlel
Looks like it is record "message" defined in tools/xmpp_codec.hrl
with <<>> id and type 'normal'
xmpp_codec.hrl
-record(message, {id :: binary(),
type = normal :: 'chat' | 'error' | 'groupchat' | 'headline' | 'normal',
lang :: binary(),
from :: any(),
to :: any(),
subject = [] :: [#text{}],
body = [] :: [#text{}],
thread :: binary(),
error :: #error{},
sub_els = [] :: [any()]}).
Include this file and use just
Type = Packet#message.type
or, if you expect binary value
Type = erlang:atom_to_binary(Packet#message.type, utf8)

The newest way to do that seems to be with xmpp:get_type/1:
Type = xmpp:get_type(Packet),
It returns an atom, in this case normal.

Resources