Related
We are using on_retry_callback parameter available in the Airflow operators to do some cleanup activities before the task is retried. If there are exceptions thrown on the on_retry_callback function, the exceptions are not logged in the task_instance's log. Without the exception details, it is getting difficult to debug if there are issues in the on_retry_callback function. If this is the default behavior, is there a workaround to enable logging for the exceptions?.
Note: We are using the airflow 2.0.2 version.
Please let me know if there are any questions.
Sample Dag to explain this is given below.
from datetime import datetime
from airflow.operators.python import PythonOperator
from airflow.models.dag import DAG
def sample_function2():
var = 1 / 0
def on_retry_callback_sample(context):
print(f'on_retry_callback_started')
v = 1 / 0
print(f'on_retry_callback completed')
dag = DAG(
'venkat-test-dag',
description='This is a test dag',
start_date=datetime(2023, 1, 10, 18, 0),
schedule_interval='0 12 * * *',
catchup=False
)
func2 = PythonOperator(task_id='function2',
python_callable=sample_function2,
dag=dag,
retries=2,
on_retry_callback=on_retry_callback_sample)
func2
Log file of this run on the local airflow setup is given below. If you see the last message we see on the log file "on_retry_callback_started" but I expect some ZeroDivisionError after this line and finally the line "on_retry_callback completed". How can I achieve this?.
14f0fed99882
*** Reading local file: /usr/local/airflow/logs/venkat-test-dag/function2/2023-01-13T13:22:03.178261+00:00/1.log
[2023-01-13 13:22:05,091] {{taskinstance.py:877}} INFO - Dependencies all met for <TaskInstance: venkat-test-dag.function2 2023-01-13T13:22:03.178261+00:00 [queued]>
[2023-01-13 13:22:05,128] {{taskinstance.py:877}} INFO - Dependencies all met for <TaskInstance: venkat-test-dag.function2 2023-01-13T13:22:03.178261+00:00 [queued]>
[2023-01-13 13:22:05,128] {{taskinstance.py:1068}} INFO -
--------------------------------------------------------------------------------
[2023-01-13 13:22:05,128] {{taskinstance.py:1069}} INFO - Starting attempt 1 of 3
[2023-01-13 13:22:05,128] {{taskinstance.py:1070}} INFO -
--------------------------------------------------------------------------------
[2023-01-13 13:22:05,143] {{taskinstance.py:1089}} INFO - Executing <Task(PythonOperator): function2> on 2023-01-13T13:22:03.178261+00:00
[2023-01-13 13:22:05,145] {{standard_task_runner.py:52}} INFO - Started process 6947 to run task
[2023-01-13 13:22:05,150] {{standard_task_runner.py:76}} INFO - Running: ['airflow', 'tasks', 'run', 'venkat-test-dag', 'function2', '2023-01-13T13:22:03.178261+00:00', '--job-id', '356', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/dp-etl-mixpanel_stg-24H/dags/venkat-test-dag.py', '--cfg-path', '/tmp/tmpny0mhh4j', '--error-file', '/tmp/tmpul506kro']
[2023-01-13 13:22:05,151] {{standard_task_runner.py:77}} INFO - Job 356: Subtask function2
[2023-01-13 13:22:05,244] {{logging_mixin.py:104}} INFO - Running <TaskInstance: venkat-test-dag.function2 2023-01-13T13:22:03.178261+00:00 [running]> on host 14f0fed99882
[2023-01-13 13:22:05,345] {{taskinstance.py:1283}} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=venkat-test-dag
AIRFLOW_CTX_TASK_ID=function2
AIRFLOW_CTX_EXECUTION_DATE=2023-01-13T13:22:03.178261+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-01-13T13:22:03.178261+00:00
[2023-01-13 13:22:05,346] {{taskinstance.py:1482}} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 128, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/dp-etl-mixpanel_stg-24H/dags/venkat-test-dag.py", line 7, in sample_function2
var = 1 / 0
ZeroDivisionError: division by zero
[2023-01-13 13:22:05,349] {{taskinstance.py:1532}} INFO - Marking task as UP_FOR_RETRY. dag_id=venkat-test-dag, task_id=function2, execution_date=20230113T132203, start_date=20230113T132205, end_date=20230113T132205
[2023-01-13 13:22:05,402] {{local_task_job.py:146}} INFO - Task exited with return code 1
[2023-01-13 13:22:05,459] {{logging_mixin.py:104}} INFO - on_retry_callback_started
Adding as an answer for visibility:
This issue is likely related to a fix which was merged in Airflow version 2.1.3:
https://github.com/apache/airflow/pull/17347
I am running airflow 2.4.3 in a single machine, airflow is installed in a python virtualenv, the airflow_home is /wwx/airflow.
My DAG:
default_args = {
'owner': 'talend',
'start_date': datetime(2023, 2, 1),
'retries': 5,
'retry_delay': timedelta(minutes=5),
'run_as_user': 'talend'
}
dag = DAG(
'dag_fetch_public_holiday',
default_args=default_args,
description='Fetch public holiday and save to csv file.',
schedule_interval='0 6 * * *',
catchup=False,
tags=['wwx', 'elt']
)
download_csv = PythonOperator(task_id='task_download_csv', python_callable=download_public_holiday_csv, dag=dag)
DAG description:
The dag is owned and run by a user talend, this user is created in both OS and airflow level. In OS level, the user has group airflow, sudo; in airflow level, the user is admin role.
Inside the dag there is a PythonOperator task to save csv to a folder, it is expected that the csv file will be created and owned by the talend user.
Problem description:
When I trigger this dag in web UI, it is showing the error permission denied for the dag log folder:
*** Reading local file: /wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv/attempt=1.log
[2023-02-07, 18:07:23 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:07:23 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:07:23 CST] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:07:23 CST] {taskinstance.py:1363} INFO - Starting attempt 1 of 6
[2023-02-07, 18:07:23 CST] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:07:23 CST] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): task_download_csv> on 2023-02-06 06:00:00+00:00
[2023-02-07, 18:07:23 CST] {base_task_runner.py:129} INFO - Running on host: vmi1120376.contaboserver.net
[2023-02-07, 18:07:23 CST] {base_task_runner.py:130} INFO - Running: ['sudo', '-E', '-H', '-u', 'talend', 'airflow', 'tasks', 'run', 'dag_fetch_public_holiday', 'task_download_csv', 'scheduled__2023-02-06T06:00:00+00:00', '--job-id', '30', '--raw', '--subdir', 'DAGS_FOLDER/fetch_public_holiday.py', '--cfg-path', '/tmp/tmpejvrehc1']
[2023-02-07, 18:07:24 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv /wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-02-07, 18:07:24 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv [[34m2023-02-08, 02:07:24 CST[0m] {[34mdagbag.py:[0m537} INFO[0m - Filling up the DagBag from /wwx/airflow/dags/fetch_public_holiday.py[0m
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv self._accessor.mkdir(self, mode)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv During handling of the above exception, another exception occurred:
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/local/bin/airflow", line 8, in <module>
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv sys.exit(main())
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv args.func(args)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return func(*args, **kwargs)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 103, in wrapper
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return f(*args, **kwargs)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 372, in task_run
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv ti.init_run_context(raw=args.raw)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2503, in init_run_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv self._set_context(self)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 77, in _set_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv set_context(self.log, context)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 213, in set_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv flag = cast(FileTaskHandler, handler).set_context(value)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 70, in set_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv local_loc = self._init_file(ti)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 320, in _init_file
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv Path(directory).mkdir(mode=0o777, parents=True, exist_ok=True)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1297, in mkdir
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv if not exist_ok or not self.is_dir():
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1422, in is_dir
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return S_ISDIR(self.stat().st_mode)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1198, in stat
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return self._accessor.stat(self)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:07:28 CST] {local_task_job.py:159} INFO - Task exited with return code 1
[2023-02-07, 18:07:28 CST] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2023-02-07, 18:39:50 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:39:50 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:39:50 CST] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:39:50 CST] {taskinstance.py:1363} INFO - Starting attempt 1 of 6
[2023-02-07, 18:39:50 CST] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:39:51 CST] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): task_download_csv> on 2023-02-06 06:00:00+00:00
[2023-02-07, 18:39:51 CST] {base_task_runner.py:129} INFO - Running on host: vmi1120376.contaboserver.net
[2023-02-07, 18:39:51 CST] {base_task_runner.py:130} INFO - Running: ['sudo', '-E', '-H', '-u', 'talend', 'airflow', 'tasks', 'run', 'dag_fetch_public_holiday', 'task_download_csv', 'scheduled__2023-02-06T06:00:00+00:00', '--job-id', '32', '--raw', '--subdir', 'DAGS_FOLDER/fetch_public_holiday.py', '--cfg-path', '/tmp/tmp26yeooeq']
[2023-02-07, 18:39:53 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv /wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-02-07, 18:39:54 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv [[34m2023-02-08, 02:39:54 CST[0m] {[34mdagbag.py:[0m537} INFO[0m - Filling up the DagBag from /wwx/airflow/dags/fetch_public_holiday.py[0m
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv self._accessor.mkdir(self, mode)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv During handling of the above exception, another exception occurred:
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/local/bin/airflow", line 8, in <module>
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv sys.exit(main())
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv args.func(args)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return func(*args, **kwargs)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 103, in wrapper
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return f(*args, **kwargs)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 372, in task_run
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv ti.init_run_context(raw=args.raw)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2503, in init_run_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv self._set_context(self)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 77, in _set_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv set_context(self.log, context)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 213, in set_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv flag = cast(FileTaskHandler, handler).set_context(value)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 70, in set_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv local_loc = self._init_file(ti)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 320, in _init_file
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv Path(directory).mkdir(mode=0o777, parents=True, exist_ok=True)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1297, in mkdir
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv if not exist_ok or not self.is_dir():
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1422, in is_dir
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return S_ISDIR(self.stat().st_mode)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1198, in stat
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return self._accessor.stat(self)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:40:01 CST] {local_task_job.py:159} INFO - Task exited with return code 1
[2023-02-07, 18:40:01 CST] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
Then I have examined the problematic folder and found it is permission 700.
I could chmod 777 to the folder manually, however, if there are other new dags/tasks, any new dag/task folder is automatically created with permission 700 and I have to manually chmod.
What I have tried:
I have followed https://airflow.apache.org/docs/apache-airflow/1.10.10/security.html?highlight=impersonation#impersonation
Added the following line to /etc/sudoers
airflow ALL=(ALL) NOPASSWD: ALL
Using sudo to start webserver and scheduler
sudo sh -c 'export AIRFLOW_HOME=/wwx/airflow; /wwx/airflow/venv/bin/airflow scheduler -D'
sudo sh -c 'export AIRFLOW_HOME=/wwx/airflow; /wwx/airflow/venv/bin/airflow webserver -D -p 8090'
Added airflow command to /usr/local/bin so that other users can use
sudo ln -s /wwx/airflow/venv/bin/airflow /usr/local/bin/airflow
I have found the solution as follows:
Add the user to airflow group.
usermod -aG airflow <username>
Set the acl of airflow log directory to have files/folders created with group permission rwx, so the dag run user can access these logs
setfacl -d -m group:airflow:rwx $AIRFLOW_HOME/logs
I am not sure if this is a intended behavior in airflow when running dag as other users, since I cannot find any hint from the airflow documentation.
When I'm running data_ingestion_gcs_dag DAG in Airflow.I get error that it can not find a specified bucket, however, I rechecked it and the bucket name is fine. I have specified access to Google account with docker-compose, here is code down below, i have inserted only first part of code:
version: '3'
x-airflow-common:
&airflow-common
# In order to add custom dependencies or upgrade provider packages you can use your extended image.
# Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
# and uncomment the "build" line below, Then run `docker-compose build` to build the images.
build:
context: .
dockerfile: ./Dockerfile
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
GOOGLE_APPLICATION_CREDENTIALS: /.google/credentials/google_credentials.json
AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT: 'google-cloud-platform://?extra__google_cloud_platform__key_path=/.google/credentials/google_credentials.json'
# TODO: Please change GCP_PROJECT_ID & GCP_GCS_BUCKET, as per your config
GCP_PROJECT_ID: 'real-dtc-de'
GCP_GCS_BUCKET: 'dtc_data_lake_real-dtc-de'
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- ~/.google/credentials/:/.google/credentials:ro
And here is code from DAG code, presented down below:
PROJECT_ID = os.environ.get("GCP_PROJECT_ID")
BUCKET = os.environ.get("GCP_GCS_BUCKET")
Here is logs from DAG:
*** Reading local file: /opt/airflow/logs/data_ingestion_gcs_dag/local_to_gcs_task/2022-06-13T02:47:29.654918+00:00/1.log
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: data_ingestion_gcs_dag.local_to_gcs_task manual__2022-06-13T02:47:29.654918+00:00 [queued]>
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: data_ingestion_gcs_dag.local_to_gcs_task manual__2022-06-13T02:47:29.654918+00:00 [queued]>
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1238} INFO -
--------------------------------------------------------------------------------
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1239} INFO - Starting attempt 1 of 2
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1240} INFO -
--------------------------------------------------------------------------------
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1259} INFO - Executing <Task(PythonOperator): local_to_gcs_task> on 2022-06-13 02:47:29.654918+00:00
[2022-06-13, 02:47:36 UTC] {standard_task_runner.py:52} INFO - Started process 1042 to run task
[2022-06-13, 02:47:36 UTC] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'data_ingestion_gcs_dag', 'local_to_gcs_task', 'manual__2022-06-13T02:47:29.654918+00:00', '--job-id', '11', '--raw', '--subdir', 'DAGS_FOLDER/data_ingestion_gcs_dag.py', '--cfg-path', '/tmp/tmp11gg9aoy', '--error-file', '/tmp/tmpjbp6yrks']
[2022-06-13, 02:47:36 UTC] {standard_task_runner.py:77} INFO - Job 11: Subtask local_to_gcs_task
[2022-06-13, 02:47:36 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: data_ingestion_gcs_dag.local_to_gcs_task manual__2022-06-13T02:47:29.654918+00:00 [running]> on host aea7312db396
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1426} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=data_ingestion_gcs_dag
AIRFLOW_CTX_TASK_ID=local_to_gcs_task
AIRFLOW_CTX_EXECUTION_DATE=2022-06-13T02:47:29.654918+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-06-13T02:47:29.654918+00:00
[2022-06-13, 02:47:36 UTC] {taskinstance.py:1700} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2594, in upload_from_file
retry=retry,
File "/home/airflow/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2396, in _do_upload
retry=retry,
File "/home/airflow/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1917, in _do_multipart_upload
transport, data, object_metadata, content_type, timeout=timeout
File "/home/airflow/.local/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 154, in transmit
retriable_request, self._get_status_code, self._retry_strategy
File "/home/airflow/.local/lib/python3.7/site-packages/google/resumable_media/requests/_request_helpers.py", line 147, in wait_and_retry
response = func()
File "/home/airflow/.local/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 149, in retriable_request
self._process_response(result)
File "/home/airflow/.local/lib/python3.7/site-packages/google/resumable_media/_upload.py", line 113, in _process_response
_helpers.require_status_code(response, (http.client.OK,), self._get_status_code)
File "/home/airflow/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 104, in require_status_code
*status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task
result = execute_callable(context=context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 174, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 185, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/data_ingestion_gcs_dag.py", line 51, in upload_to_gcs
blob.upload_from_filename(local_file)
File "/home/airflow/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2735, in upload_from_filename
retry=retry,
File "/home/airflow/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2598, in upload_from_file
_raise_from_invalid_response(exc)
File "/home/airflow/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 4466, in _raise_from_invalid_response
raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 POST https://storage.googleapis.com/upload/storage/v1/b/dtc_data_lake_animated-surfer-338618/o?uploadType=multipart: {
"error": {
"code": 404,
"message": "The specified bucket does not exist.",
"errors": [
{
"message": "The specified bucket does not exist.",
"domain": "global",
"reason": "notFound"
}
]
}
}
I am trying to create dependency between multiple dags.
Lets say Dag_A, Dab_B and and running every day at 14:15 and 14:30 respectively.
now i want to run Dag_C which runs at 14:30 having 2 sensors ( ExternalTaskSensors) each for above dags. I am also using execution_date_fn parameter which provides 3 execution date each for above dags. So basically sensor checks for 14:15 and 14:30 for each dag. But still sensor keeps on waiting and doesn't succeed. It going for up_for_schedule
Am i doing anything wrong? Please suggest how to deal with such cases.
I am using airflow version 2
Below is the code for
DAG_A:
with DAG(
dag_id="dag_a",
default_args=DEFAULT_ARGS,
max_active_runs=1,
schedule_interval="15 2 * * *",
catchup=True
) as dag:
dummy_task = DummyOperator(task_id="Task_A")
DAG_B:
with DAG(
dag_id="dag_b",
default_args=DEFAULT_ARGS,
max_active_runs=1,
schedule_interval="30 2 * * *",
catchup=True
) as dag:
dummy_task = DummyOperator(task_id="Task_B")
DAG_C:
with DAG(
dag_id="dag_c",
default_args=DEFAULT_ARGS,
max_active_runs=1,
schedule_interval="30 2 * * *",
catchup=True
) as dag:
wait_task_a = ExternalTaskSensor(
task_id=f"wait_for_task_a",
external_dag_id="dag_a",
execution_date_fn=lambda dt: [dt + timedelta(minutes=-i) for i in range(0, 30, 15)],
timeout=60 * 60 * 3, # 3 hours
poke_interval=60, # 5 minutes
mode="reschedule"
)
wait_task_b = ExternalTaskSensor(
task_id=f"wait_for_task_b",
external_dag_id="dag_b",
execution_date_fn=lambda dt: [dt + timedelta(minutes=-i) for i in range(0, 30, 15)],
timeout=60 * 60 * 3, # 3 hours
poke_interval=60, # 5 minutes
mode="reschedule"
)
dummy_task = DummyOperator(task_id="Task_C")
wait_task_a >> dummy_task
wait_task_b >> dummy_task
Sensor logs :
It keeps on poking although tasks are present
[2022-05-23, 16:25:20 UTC] {taskinstance.py:1043} INFO - Dependencies all met for <TaskInstance: dag_c.wait_for_task_b scheduled__2022-05-19T02:30:00+00:00 [queued]>
[2022-05-23, 16:25:20 UTC] {taskinstance.py:1043} INFO - Dependencies all met for <TaskInstance: dag_c.wait_for_task_b scheduled__2022-05-19T02:30:00+00:00 [queued]>
[2022-05-23, 16:25:20 UTC] {taskinstance.py:1249} INFO -
--------------------------------------------------------------------------------
[2022-05-23, 16:25:20 UTC] {taskinstance.py:1250} INFO - Starting attempt 1 of 2
[2022-05-23, 16:25:20 UTC] {taskinstance.py:1251} INFO -
--------------------------------------------------------------------------------
[2022-05-23, 16:25:20 UTC] {taskinstance.py:1270} INFO - Executing <Task(ExternalTaskSensor): wait_for_task_b> on 2022-05-19 02:30:00+00:00
[2022-05-23, 16:25:20 UTC] {standard_task_runner.py:52} INFO - Started process 17603 to run task
[2022-05-23, 16:25:20 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'dag_c', 'wait_for_task_b', 'scheduled__2022-05-19T02:30:00+00:00', '--job-id', '4', '--raw', '--subdir', 'DAGS_FOLDER/sample/dagc.py', '--cfg-path', '/var/folders/q1/dztb0bzn0fn8mvfm7_q9ms0m0000gn/T/tmpb27mns7u', '--error-file', '/var/folders/q1/dztb0bzn0fn8mvfm7_q9ms0m0000gn/T/tmpc6y4_6cx']
[2022-05-23, 16:25:20 UTC] {standard_task_runner.py:80} INFO - Job 4: Subtask wait_for_task_b
[2022-05-23, 16:25:25 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: dag_c.wait_for_task_b scheduled__2022-05-19T02:30:00+00:00 [running]> on host yahoo-MacBook-Pro.local
[2022-05-23, 16:25:30 UTC] {taskinstance.py:1448} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=dag_c
AIRFLOW_CTX_TASK_ID=wait_for_task_b
AIRFLOW_CTX_EXECUTION_DATE=2022-05-19T02:30:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-05-19T02:30:00+00:00
[2022-05-23, 16:25:30 UTC] {external_task.py:175} INFO - Poking for tasks None in dag dag_b on 2022-05-19T02:30:00+00:00,2022-05-19T02:15:00+00:00 ...
[2022-05-23, 16:25:30 UTC] {taskinstance.py:1726} INFO - Rescheduling task, marking task as UP_FOR_RESCHEDULE
[2022-05-23, 16:25:30 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-05-23, 16:25:30 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check
So when I run the job locally using jar, it deploys and finishes successfully i.e. I can see the output files in GCS
java -cp /Users/zainqasmi/Workspace/vasa/dataflow/build/libs/vasa-dataflow-2022-03-25-12-27-14-784-all.jar com.nianticproject.geodata.extraction.ExtractGeodata \
--project=vasa-dev \
--configurationPath=/Users/zainqasmi/Workspace/vasa/dataflow/src/main/resources/foursquare/extract.pb.txt \
--region=us-central1 \
--runner=DataflowRunner \
--dryRun=false \
--workerMachineType=n2d-highmem-16
However, when I push the dag to airflow, it apparently runs successfully i.e. Marking task as SUCCESS and return code 0. But I can't find the dataflow being executed anywhere in GCP UI. Am I missing something? Using environment composer-2-0-7-airflow-2-2-3
Logs from airflow:
*** Reading remote log from gs://us-central1-airflow-dev-b0cc30af-bucket/logs/foursquare_1/extract_geodata/2022-03-25T22:52:15.382542+00:00/1.log.
[2022-03-25, 22:52:21 UTC] {taskinstance.py:1033} INFO - Dependencies all met for <TaskInstance: foursquare_1.extract_geodata manual__2022-03-25T22:52:15.382542+00:00 [queued]>
[2022-03-25, 22:52:21 UTC] {taskinstance.py:1033} INFO - Dependencies all met for <TaskInstance: foursquare_1.extract_geodata manual__2022-03-25T22:52:15.382542+00:00 [queued]>
[2022-03-25, 22:52:21 UTC] {taskinstance.py:1239} INFO -
--------------------------------------------------------------------------------
[2022-03-25, 22:52:21 UTC] {taskinstance.py:1240} INFO - Starting attempt 1 of 2
[2022-03-25, 22:52:21 UTC] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2022-03-25, 22:52:21 UTC] {taskinstance.py:1260} INFO - Executing <Task(DataFlowJavaOperator): extract_geodata> on 2022-03-25 22:52:15.382542+00:00
[2022-03-25, 22:52:21 UTC] {standard_task_runner.py:52} INFO - Started process 57323 to run task
[2022-03-25, 22:52:21 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'foursquare_1', 'extract_geodata', 'manual__2022-03-25T22:52:15.382542+00:00', '--job-id', '1531', '--raw', '--subdir', 'DAGS_FOLDER/dataflow_operator_test.py', '--cfg-path', '/tmp/tmp4thgd6do', '--error-file', '/tmp/tmpu6crkval']
[2022-03-25, 22:52:21 UTC] {standard_task_runner.py:77} INFO - Job 1531: Subtask extract_geodata
[2022-03-25, 22:52:22 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: foursquare_1.extract_geodata manual__2022-03-25T22:52:15.382542+00:00 [running]> on host airflow-worker-9rz89
[2022-03-25, 22:52:22 UTC] {taskinstance.py:1426} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=foursquare_1
AIRFLOW_CTX_TASK_ID=extract_geodata
AIRFLOW_CTX_EXECUTION_DATE=2022-03-25T22:52:15.382542+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-03-25T22:52:15.382542+00:00
[2022-03-25, 22:52:22 UTC] {credentials_provider.py:312} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
[2022-03-25, 22:52:22 UTC] {taskinstance.py:1268} INFO - Marking task as SUCCESS. dag_id=foursquare_1, task_id=extract_geodata, execution_date=20220325T225215, start_date=20220325T225221, end_date=20220325T225222
[2022-03-25, 22:52:22 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-25, 22:52:22 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check
Dag:
GCP_PROJECT = "vasa-dev"
CONNECTION_ID = 'bigquery_default'
VASA_DATAFLOW_JAR = '/home/airflow/gcs/data/bin/vasa-dataflow-2022-03-25-16-36-09-008-all.jar'
default_args = {
'owner': 'airflow',
'depends_on_past': True,
'wait_for_downstream' : True,
'max_active_runs' : 1,
'start_date': days_ago(1),
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(days=1),
}
with DAG(
dag_id = 'foursquare_1',
schedule_interval=timedelta(days=1),
default_args=default_args
) as dag:
kick_off_dag = DummyOperator(task_id='run_this_first')
extract_geodata = DataFlowJavaOperator(
task_id='extract_geodata',
jar=VASA_DATAFLOW_JAR,
job_class='com.nianticproject.geodata.extraction.ExtractGeodata',
options= {
"project": "vasa-dev",
"configurationPath": "/home/airflow/gcs/foursquare/extract.pb.txt",
"region": "us-central1",
"runner": "DataflowRunner",
"dryRun": "false",
"workerMachineType":"n2d-highmem-16",
},
dag=dag)
end_task = BashOperator(
task_id='end_task',
bash_command='echo {{ execution_date.subtract(months=1).replace(day=1).strftime("%Y-%m-%d") }}',
dag=dag,
)
kick_off_dag >> extract_geodata >> end_task