I'm using a bit of airflow recently and now my dags are up and running I've looked at some data in the data profiling section of the UI.
Made some charts, some table full of data.
I would like to be able to send me once a week those chart and data in an email so I can have some regular global update about my different dag runs and tasks, not just an email in the end of each run saying success or fail.
Is there any convenient way of doing that ? or do I have to build a custom dag with jinja template email operator and horrible SQLAlchemy syntax to re-extract data from the database ?
To my knowledge there's nothing "out of the box" that will do what you want; you're likely correct on the "I have to build a custom dag […]" part of your question. Luckily since you're in Airflow, you can leverage its codebase to help:
from airflow import models, settings
def python_task(**context):
chart_label = context.params["chart_label"]
query_filters = [models.Chart.label == chart_label]
session = settings.Session()
chart_object = session.query(models.Chart).filter(*query_filters).first()
[...] build and send the email [...]
You will likely want to look at the Airflow source code, specifically the email handling portions and interface rendering. Note that if you can make your task relatively abstract you can reuse it pretty easily.
So I found some kind of solution. Feel free to comment on the way I do things if there's better way of doing it. I would be glad to here
I've created a static function to recover data from a chart:
from airflow import settings
def get_data_profilling_data( chart_label):
"""
Get an array containing data from a data profilling chart.
Args:
chart_label: a string that is the name of a an available chart in Data Profiling section.
session(None, optional): Not in use.
Returns:
A list of tuple with first element representing the header of column.
"""
import logging
query_filters = [models.Chart.label == chart_label]
session = settings.Session()
# Query database to get a chart object.
chart_object = session.query(models.Chart).filter(*query_filters).first()
# Replace useless caracter in query and replace % by %% so it's not interpreted by SQLAlchemy
# Also add a ; at the end of the statement.
sql="{}{}".format(chart_object.sql.replace('\n', ' ').replace('\r',' ').replace('%','%%'), ';')
query = session.connection().engine.execute(sql)
#recover sql request header.
headers = tuple(query.keys())
#recover all data
data = query.fetchall()
#close database connection
#session.connection().close()
# Add headers to first position in the list
data.insert(0,headers)
return data
Once this is extracted I've used jinja templating :
jinja_str = """<!DOCTYPE html>
<html>
<head>
{% if title %}
<title>{{ title }}</title>
{% else %}
<title>Airflow reports</title>
{% endif %}
</head>
<body>
<table>
{% if rows %}
{% for row in rows %}
{% if loop.index == 1 %}
<!-- table header -->
<tr>{% for elem in row %}
<th>{{ elem }}</th>
{% endfor %}</tr>
{% endif %}
{% if loop.index > 1 %}
<tr>{% for elem in row %}
<td>{{ elem }}</td>
{% endfor %}</tr>
{% endif %}
{% endfor %}
{% endif %}
</table>
</body>
</html>"""
def format_data_from_chart(chart, **kwargs):
data = DagStatic.get_data_profilling_data(chart)
report_template = Template( jinja_str )
report_html = report_template.render({"title": chart,"rows":data})
kwargs["ti"].xcom_push(key='report_html', value=report_html)
This would push into xcom the result of the data from chart rendered with jinja template.
I can now use Email Operator to send my mail:
send_report= email_task = EmailOperator(
to='you.mail#mail.com',
task_id='send_report',
subject='Airflow HTML report start_date {{ ds }}',
html_content="{{ ti.xcom_pull(key='report_html') }}",
dag=dag)
Hope this will help you. Feel free to comment on the way I do things if there's better way of doing it. I would be glad to here
Related
In Drupal views, I have a twig variable available to me called {{ name }}. This stores a list of taxonomy terms depending on the type of content used. I currently have a function running something like this that works:
{% if name == "Tax1" %}
Download
{% elseif name == "Tax2" %}
Download
{% endif %}
However, I feel like there is a better way for me to go about doing this. As an example isn't something like this supposed to work?
{%
set newval = [
name == "Tax1" ? 'file1.pdf' : 'file2.pdf'
]
%}
Download
Pretty much what I'm trying to state above is, if the value of name is equal to "Tax1", print out file1.pdf, else print file2.pdf.
I have very basic twig knowledge and I haven't touched it in a couple years so if anyone could help me out with this, that would be great.
Thanks to #DarkBee for the help. I must have been overthinking as the answer was fairly simple.
The below solution worked for what I was initially asking:
Download
However, I ended up needing a third default option if not the first two so the below code is what I ended up using:
{% if name == "Tax1" or name == "Tax2" %}
Download
{% else %}
<span class="stat">Other Option</span>
{% endif %}
I have a base template that includes multiple sub-templates and the code runs something like this:
<ul class="row portfolio list-unstyled mt-3 lightbox" id="grid">
<!-- summary section -->
{{ render(controller('App\\Controller\\ReadController::summary')) }}
{{ render(controller('App\\Controller\\BookController::random', {num: 3})) }}
{{ render(controller('App\\Controller\\WikiController::random')) }}
</ul><!-- / portfolio row -->
As can be seen, these "items" appear in a fixed order:summary goes first, then 3 random and then one random.
What I intend to do is to "shuffle" these items (in the above code snippet, there will be 5 items) so that the order is different in each refresh to give the end user some variation.
Is this possible to do in Twig?
UPDATE
I used #hcoat method and it is working. Will try the shuffle filter later.
As suggested to you in the comments above you can use the Array Extension or send it through a controller. I think having a controller that sends in the random list is the way to go.
However, sometimes it is useful to randomize a list with standard twig and in such cases you can do something like the following:
// Path and pram Array, pass empty hash if no params
{% set arr = [
["App\\Controller\\ReadController::summary", {}],
["App\\Controller\\BookController::random", {'num': 3}],
["App\\Controller\\WikiController::random", {}]
] %}
// create a list and merge arr array with random key
{% set list = {} %}
{% for item in arr %}
// There is a bug in some twig verions
// so concat a letter to ensure random key works
{% set list = list|merge({ (random()~'a'):(item) }) %}
{% endfor %}
// sort the list by the random key and render the output
{% for key in list|keys|sort %}
{{ render(controller(list[key][0], list[key][1])) }}
{% endfor %}
Now the sub-templates will be rendered in a random order.
I try to visualize my error messages from the validator.
When i write the following example:-
{{ form_errors(form.pGWeek) }}
it works fine and i get the message. But my form has 200 fields and so it's not practical.
So i want to iterate over an Array with all messages, at the end of the form like this:
{% if form.name.vars.errors|length > 0 %}
<ul class="form-errors name">
{% for error in form.name.vars.errors %}
{{ error }}
{% endfor %}
</ul>
{% endif %}
But i did not get some messages. As well i tried some another versions.. but nothing worked. I'm using Symfony 2.7.
Can give me somebody a tip?
Thanks for a short feedback.
So you just want to show all errors for all children of a form without displaying them beside each input field?
Then you could iterate over all the children of your form, check if any errors appeared on this children and if so, iterate over all errors of this children. That could be something like that:
{% for children in form.children %}
{% if children.vars.errors is defined %}
{% for error in children.vars.errors %}
{#{{ dump(children) }}#}
{#{{ dump(error) }}#}
{{ dump(children.vars.name ~ ': ' ~ error.message) }}
{% endfor %}
{% endif %}
{% endfor %}
Which result in an error like description: This value should not be blank..
You can make that loop and condition in your controller using dependency injector, and then in your view you iterate that array of errors and put it into a div o wherever you want, I guess there is something like this:
$errors = $this->get('validator')->validate(yourObject);
if (!empty($errors)) {
foreach ($errors as $error)
throw new \Exception($error->getMessage());
}
That is if you want to stop the process and get an Exception, but you want to show that errors, then you make something similar, you only needs to delete that foreach and render your twig template and give it the $errors variable, in your template then you only need to make a for loop for get the errors!
In Jekyll 2.5.3, I use albums collection (because I need not only data stored, but also pages generated).
When Jekyll Data Files are used, you can get a data for particular item as simple as: site.data.albums[foo]
But with collections, things are much worse. All those ways I've tried just do nothing:
site.albums[foo]
site.collections.albums[foo]
site.collections.albums.docs[foo]
site.collections.albums.files[foo]
So I need to:
Loop through all collection items
For each of them get a bare name
Compare this name with some target name
If the name matches, finally assign collection item data to some variable to use
Any better suggestions?
I have just done this today, you are correct with your assertions. Here's how I did it:
<!-- this is in a partial with a 'name' parameter -->
{% for item in site.albums %}
{% assign name = item.path | split:"/" | last | split:"." | first %}
{% if name == include.name %}
{% assign collectionItem = item %}
{% endif %}
{% endfor %}
Usage
{{ collectionItem.title }}
{{ collectionItem.url }}
{{ collectionItem.path }}
It can even be used to populate values in other partials, like so:
{% include card.html title=workItem.title bg=workItem.card.bg href=workItem.url %}
As of Jekyll 3.2, you can use the filter where_exp to filter a collection by any of its properties. So if I have the following item in an albums collection:
---
title: My Amazing Album
---
...
I can grab the first match for an item by that name:
{% assign album = site.albums
| where_exp:"album", "album.title == 'My Amazing Album'"
| first %}
Note that I use the first filter because where_exp returns an array of matched items.
And then use it any way I like:
<h1>{{ album.title }}</h1>
I can't vouch for the build performance of this method, but I have to imagine it's better than a Liquid loop.
In Symfony / Twig, I could use tags by using percentages in my translated block. For example:
Hello {{nickname}}
would become
{% trans %}Hello %nickname%{% endtrans %}
This works as expected. The array with placeholders that I pass to Twig, are automatically mapped to %placeHolder%. No extra work involved. So this works with my PHP array from the controller being:
Array('nickname' => 'rolandow')
When I want to use the nickname inside the translation block, all I have to do is surround it with percentages %. Unfortunately, this doesn't seem to work when I pass it to trans.
Now I would like to translate a whole block of text, using tags. I can't figure out how I can use the tags in my translation. So, my twig would look something like this:
{{ say.hello|trans }}
And my translation snippet
<trans-unit id="1">
<source>say.hello</source>
<target>Hello %nickName%, how are you doing today? lots-of-text-here</target>
</trans-unit>
I got it working by using this in my template, but it feels like doing things twice. I now need to put the array of placeholder into the trans function again. For example:
{{ say.hello|trans('%nickName%' : nickName) }}
If I want to use other tags that are given to twig in my controller, I need to pass them to the translator as well. Can't I just pass the complete array somehow?
{{ say.hello|trans('%nickname%': 'rolandow') }}
There are several questions here so let's cover them.
1) Twig's behaviour is not like a Doctrine query, where each parameter must be bounded. You can pass an array that contains unused parameters to trans, so if you don't want to specify {'key': 'value', 'key2': 'value2'...} to the filter, just pass the entire array (example: | trans(array)). That's #Luke point.
2) You can translate block of texts using several ways, the most simple is {% set %}. The {% set %} tag can be used two ways :
{% set var = expression %} or {% set var1, var2 = expression1, expression2 %} is the most known and used way: you just put some value inside one or several variables.
{% set var %} block of text {% endset %} allow you to set an entire block of text inside that variable. This is useful if you want to put that block into a filter (such as, escape, or in your case, trans).
So to translate a block of text, you'll do something like:
{% set variable %}
block to translate %placeholder%
{% endset %}
{{ variable | trans(array) }}
Anyway, I don't see any interest of translating a whole block in one time : we use | trans generally after a property (such as say.hello), and I can't imagine your xlf/yml translation file with such a design. If you want to use the translator just to fulfill placeholders, just use Twig as it is written for that job :-)
3) About replacing placeholder by %placeholder% in your parameters array's keys : the point of Twig is: put what you want as placeholder. In such a way, if your translated sentence contains several %, you can use $something$, #something# or even something as placeholder.
If your array keys does not contain those %, you need to add them, you don't have any choice. If you really want to do it on a Twig file, you can create a macro that will do the job for you, and put it in a file you import in your base layout.
Something like :
{% macro trans_pct(property, params) %}
{% set newParams = [] }
{% if params %}
{% for key, value in params %}
{% set newParams['%' ~ key ~ '%'] = value %}
{% endfor %}
{% endif %}
{{ property | trans(newParams) }}
{% endmacro %}
And then use it with {{ _self.trans_pct('hello.say', array) | trim }}.
Notes :
_self is the template where is stored the macro (see the documentation for more details).
trim is used because I wrote the macro with indentation and line breaks (that's cleaner to read). Those spaces are, by default, printed.