Airflow on_failure_callback - airflow

Question

I have an Airflow DAG with two tasks:
read_csv
process_file
They work fine on their own. I purposely created a typo in a pandas Dataframe to learn how on_failure_callback works and to see if it is being triggered. It seems likes from the log that it doesn't:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1197, in handle_failure
task.on_failure_callback(context)
TypeError: on_failure_callback() takes 0 positional arguments but 1 was given
Why isn't on_failure_callback working?
Here is a visual representation of the DAG:
Here is the code:
try:
from datetime import timedelta
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
import pandas as pd
# Setting up Triggers
from airflow.utils.trigger_rule import TriggerRule
# for Getting Variables from airlfow
from airflow.models import Variable
print("All Dag modules are ok ......")
except Exception as e:
print("Error {} ".format(e))
def read_csv(**context):
data = [{"name":"Soumil","title":"Full Stack Software Engineer"}, { "name":"Nitin","title":"Full Stack Software Engineer"},]
df = pd.DataFramee(data=data)
dag_config = Variable.get("VAR1")
print("VAR 1 is : {} ".format(dag_config))
context['ti'].xcom_push(key='mykey', value=df)
def process_file(**context):
instance = context.get("ti").xcom_pull(key='mykey')
print(instance.head(2))
return "Process complete "
def on_failure_callback(**context):
print("Fail works ! ")
with DAG(dag_id="invoices_dag",
schedule_interval="#once",
default_args={
"owner": "airflow",
"start_date": datetime(2020, 11, 1),
"retries": 1,
"retry_delay": timedelta(minutes=1),
'on_failure_callback': on_failure_callback,
},
catchup=False) as dag:
read_csv = PythonOperator(
task_id="read_csv",
python_callable=read_csv,
op_kwargs={'filename': "Soumil.csv"},
provide_context=True
)
process_file = PythonOperator(
task_id="process_file",
python_callable=process_file,
provide_context=True
)
read_csv >> process_file
# ====================================Notes====================================
# all_success -> triggers when all tasks arecomplete
# one_success -> trigger when one task is complete
# all_done -> Trigger when all Tasks are Done
# all_failed -> Trigger when all task Failed
# one_failed -> one task is failed
# none_failed -> No Task Failed
# ==============================================================================
# ============================== Executor====================================
# There are Three main types of executor
# -> Sequential Executor run single task in linear fashion wih no parllelism default Dev
# -> Local Exector run each task in seperate process
# -> Celery Executor Run each worker node within multi node architecture Most scalable
# ===========================================================================

Answer 1

You need to specify one argument to your function that can receive the context this is due to how Airflow triggers on_failure_callback
def on_failure_callback(context):
print("Fail works ! ")
Note that with your implementation you can't tell from the message which task has failed so you might want to add to your error message the task details like:
def on_failure_callback(context):
ti = context['task_instance']
print(f"task {ti.task_id } failed in dag { ti.dag_id } ")