I try to pass execution date to httpsensor operator.
is_api_available = HttpSensor(
task_id='is_api_available',
http_conn_id='data_available',
endpoint='api/3/action/date= {{I want to set date in here}}'
)
I can get execution date parameter in python operator like this:
print("my start date : ",kwargs['execution_date'] )
it works but how can I get it in other operators?
thanks in advance
You can use Jinja template with the variable {{ ds }}, it format the datetime as YYYY-MM-DD
for more macros you can see at https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html
is_api_available = HttpSensor(
task_id='is_api_available',
http_conn_id='data_available',
endpoint='api/3/action/date={{ ds }}')
api/3/action/date=2022-06-25
Related
I have made a custom sensor in Airflow which inherits BashSensor.
Sensor :
class MySensor(BashSensor):
def __init__(self, time, **kwargs): # {{ ts }} is passed as time in the DAG
self.time = time
cmd = f"java some-other-stuff {self.time}" # rendered/correct value for self.time
super().__init__(**kwargs, bash_command=cmd)
def poke(self, context):
status = super().poke() # returns True or False
if status:
print(self.time) # {{ ts }} is printed instead of rendered value
else:
print("trying again")
return status
When I look at the rendered tab for the operator task in DAG I see bash_command has the correct rendered value ({{ ts }} is passed as time).
The problem is whenever poke is called and True is returned, I see {{ ts }} in the print statement instead of the rendered value.
I expect self.time to have the rendered value (some timestamp) not {{ ts }} when I print it in poke function.
Both cmd and time are not templated field in your code so Jinja engine does not handle them. The reason you see the command being templated is because in the super call you do:
bash_command=cmd
and bash_command is templated field of BashSensor
So while the command is parsed to the correct string as expected the individual components that created it does not contain the render value.
To explain in some more details: time = "{{ ds }}" will always stay as this string. it will never be rendered.
When you do cmd = f"java some-other-stuff {self.time}" it becomes:
"java some-other-stuff {{ ds }}"
This string is assigned to bash_command which is templated field and when the code is executed the value of {{ ds }} is rendered.
To solve your issue you can simply add the parameters you want to template to the sequence:
class MySensor(BashSensor):
...
template_fields: Sequence[str] = tuple({'time'} | set(BashSensor.template_fields))
...
Ok, so i want to change a datetime value from mysql database into a desired format. so my code goes:
file: app.py
from flask import Flask, render_template, flash, redirect, url_for, session, request, logging
from datetime import datetime
from flask_moment import Moment
app = Flask(__name__)
moment = Moment(app)
mysql = MySQL(cursorclass=DictCursor)
#app.route('/users')
def users():
# create cursor
cur = mysql.get_db().cursor()
# execute query
cur.execute('SELECT * FROM users')
# get results
users = cur.fetchall()
return render_template('users.html', title = 'User Accounts', users=users)
then on my users.html, i display the datetime value on a table which goes:
file: users.html
<tbody>
{% for row in users %}
<tr>
<td>{{row.id}}</td>
<td>{{row.username}}</td>
<td>{{row.fullname}}</td>
<td>{{moment(row.created_at).format('LLL')}}</td> # this causes an error
</tr>
{% endfor %}
</tbody>
But when i put in the following code for the datetime:
<td>{{moment().format('LLL')}}</td> # this is working
So in short,
# this is not working
# Causes an "AttributeError: 'str' object has no attribute 'strftime'" error
moment(row.created_at).format('LLL')
# this is working
# but i can't modify the data based on the value from mysql database
moment().format('LLL')
# by the way, this is also not working
# and it also causes the same error
row.created_at.strftime('%M %d, %Y')
What i need to know is how to format datetime value in the flask template and Flask-Moment seems to be the only way
EDIT:
You can also try using:
{{ row.created_at.strftime('%M %d, %Y') }}
or the jinja datetime filter:
{{ row.created_at|datetime }}
ORIGINAL ANSWER:
It appears you need to convert from MySQL DateTime to Python datetime. Without seeing more of your code, maybe something like this would work, though there is probably a more efficient way:
# get results
users = cur.fetchall()
for user in users:
user.created_at = datetime.strptime(user.created_at, '%Y-%M-%d') # you have to use the mysql format here
return render_template('users.html', title = 'User Accounts', users=users)
You are using strftime when it appears that python is interpreting the MySQL datetime as a string, not a datetime() object, so you have to use strptime
Momentjs couldn't parse it because it is an 'str'. Use datetime to parse the string before passing it to the momentjs object e.g
from datetime import datetime
row_created_at = datetime.strptime(row_created_at, %m %d,%Y ')
then pass it to the momentjs object like so
{{moment(row_created).format("LLL")}}
That should do the trick
I am trying to format a datetime value into a human readable format
For example:
value = 2019-12-17T08:12:58.472+00:00
expected result = 17/12/2019.
The code has to be in Jinja. Been trying to do the following, but getting an error message:
{{ 2019-12-17T08:12:58.472+00:00 | from.strftime('%Y-%m-%d') }}
Can anybody help?
This works for me to pull out a date in your requested "human readable" format %d/%m/%Y:
>>> from jinja2 import Template
>>> from datetime import datetime
>>> t = Template("Hello, {{ your_jinja_date_var.strftime('%d/%m/%Y') }}!")
>>> t.render(your_jinja_date_var=datetime.now())
'Hello, 17/12/2019!'
Here are the API docs where you can find other incredible features of jinja.
Your strftime function from your question above does imply standard ISO date conventions %Y-%m-%d. If you'd like that date format:
>>> t = Template("Hello, {{ your_jinja_date_var.strftime('%Y-%m-%d') }}!")
>>> t.render(your_jinja_date_var=datetime.now())
'Hello, 2019-12-17!'
And to top it off, here's a nice tutorial article on the topic which I found useful, from the Real Python platform.
<p>Last seen: {{ user.last_seen.strftime('%d-%m-%Y, %T') }}</p>
This worked for me for Date and Time both.
i create a Dag that will be scheduled to run every day once a day - the dag wont get any parameters . each day the dag ran i need to calculate yesterdays date (current Date -1) and pass it for all the operators in the dag (same date).
i saw that i can use Airflow-macros to compute the date , but the problem is ,the operator i use in t2 (must use this operator and cant change it) pass the data to the dictionary (default_param_dict) as a string and don't compute the macro.
is there any other way to compute the date without macros? using X-com is not relevant because i can only use the operators and cant change their code.
need you help:)
adding my dag example:
t1 = SimpleHttpOperator(
task_id='check_if_daily_report_ready',
method='GET',
endpoint="/bla/bla?date={date}".format(
date='{{ (execution_date - macros.timedelta(days=1)).strftime("%Y-%m-%d") }}'),
http_conn_id="conn1",
headers={"Content-Type": "application/json"},
response_check=lambda response: True if response.status_code == 200 else False,
dag=dag,
)
t2 = Queryperator(
task_id='cal',
query_file='ca.sql',
query_folder='include/sql_files/bla',
token='Token',
default_param_dict={"date": '{{ (execution_date - macros.timedelta(days=1)).strftime("%Y-%m-%d") }}'},
dag=dag
)
If I understood the question, you wanted to add yesterday's date the default_param_dict, if so, I would recommend using the datetime, something like below,
import datetime
t2 = Queryperator(
task_id='cal',
query_file='ca.sql',
query_folder='include/sql_files/bla',
token='Token',
default_param_dict={"date": (datetime.date.today() - datetime.timedelta(days=1)).strftime('%Y-%m-%d')},
dag=dag
)
I have a simple line of Crystal Reports code below:
EffectiveDateTimeString = ToText({Command.EffectiveDate} , "dd-MM-yyyy hh:mm:ss" );
However, when I try to validate the formula, I get a error saying "Too many arguments have been given to this function" with "dd-MM-yyyy hh:mm:ss" selected. Command.EffectiveDate is the DateTime object. How can I convert this to a string?
Thanks!
You need to use the assignment operator :=, not the equivalency one =:
EffectiveDateTimeString := ToText({Command.EffectiveDate} , "dd-MM-yyyy hh:mm:ss" );
*edit *
This snippet works as expected:
ToText(CurrentDate + CurrentTime, "dd-MM-yyyy hh:mm:ss");
Ensure that your field is actually returning a date/time, rather than one or the other.
Try this:
EffectiveDateTimeString := CStr(DateTime({Command.EffectiveDate} , "dd/MM/yyyy hh:mm:ss" ));
If {Command.EffectiveDate} is not in the right format, this will ensure that it is indeed DateTime.
If that doesn't work, I just created a simple formula in a sample report of mine, and the below code worked just fine on a DateTime field:
stringVar EffectiveDateTimeString;
EffectiveDateTimeString := CStr({Command.EffectiveDate}, "dd/MM/yyyy hh:mm:ss");
EffectiveDateTimeString