Airflow run_as_user permission denied - airflow

Question

I am running airflow 2.4.3 in a single machine, airflow is installed in a python virtualenv, the airflow_home is /wwx/airflow.
My DAG:
default_args = {
'owner': 'talend',
'start_date': datetime(2023, 2, 1),
'retries': 5,
'retry_delay': timedelta(minutes=5),
'run_as_user': 'talend'
}
dag = DAG(
'dag_fetch_public_holiday',
default_args=default_args,
description='Fetch public holiday and save to csv file.',
schedule_interval='0 6 * * *',
catchup=False,
tags=['wwx', 'elt']
)
download_csv = PythonOperator(task_id='task_download_csv', python_callable=download_public_holiday_csv, dag=dag)
DAG description:
The dag is owned and run by a user talend, this user is created in both OS and airflow level. In OS level, the user has group airflow, sudo; in airflow level, the user is admin role.
Inside the dag there is a PythonOperator task to save csv to a folder, it is expected that the csv file will be created and owned by the talend user.
Problem description:
When I trigger this dag in web UI, it is showing the error permission denied for the dag log folder:
*** Reading local file: /wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv/attempt=1.log
[2023-02-07, 18:07:23 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:07:23 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:07:23 CST] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:07:23 CST] {taskinstance.py:1363} INFO - Starting attempt 1 of 6
[2023-02-07, 18:07:23 CST] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:07:23 CST] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): task_download_csv> on 2023-02-06 06:00:00+00:00
[2023-02-07, 18:07:23 CST] {base_task_runner.py:129} INFO - Running on host: vmi1120376.contaboserver.net
[2023-02-07, 18:07:23 CST] {base_task_runner.py:130} INFO - Running: ['sudo', '-E', '-H', '-u', 'talend', 'airflow', 'tasks', 'run', 'dag_fetch_public_holiday', 'task_download_csv', 'scheduled__2023-02-06T06:00:00+00:00', '--job-id', '30', '--raw', '--subdir', 'DAGS_FOLDER/fetch_public_holiday.py', '--cfg-path', '/tmp/tmpejvrehc1']
[2023-02-07, 18:07:24 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv /wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-02-07, 18:07:24 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv [[34m2023-02-08, 02:07:24 CST[0m] {[34mdagbag.py:[0m537} INFO[0m - Filling up the DagBag from /wwx/airflow/dags/fetch_public_holiday.py[0m
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv self._accessor.mkdir(self, mode)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv During handling of the above exception, another exception occurred:
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/local/bin/airflow", line 8, in <module>
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv sys.exit(main())
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv args.func(args)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return func(*args, **kwargs)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 103, in wrapper
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return f(*args, **kwargs)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 372, in task_run
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv ti.init_run_context(raw=args.raw)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2503, in init_run_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv self._set_context(self)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 77, in _set_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv set_context(self.log, context)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 213, in set_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv flag = cast(FileTaskHandler, handler).set_context(value)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 70, in set_context
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv local_loc = self._init_file(ti)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 320, in _init_file
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv Path(directory).mkdir(mode=0o777, parents=True, exist_ok=True)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1297, in mkdir
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv if not exist_ok or not self.is_dir():
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1422, in is_dir
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return S_ISDIR(self.stat().st_mode)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1198, in stat
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv return self._accessor.stat(self)
[2023-02-07, 18:07:27 CST] {base_task_runner.py:111} INFO - Job 30: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:07:28 CST] {local_task_job.py:159} INFO - Task exited with return code 1
[2023-02-07, 18:07:28 CST] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2023-02-07, 18:39:50 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:39:50 CST] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dag_fetch_public_holiday.task_download_csv scheduled__2023-02-06T06:00:00+00:00 [queued]>
[2023-02-07, 18:39:50 CST] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:39:50 CST] {taskinstance.py:1363} INFO - Starting attempt 1 of 6
[2023-02-07, 18:39:50 CST] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-02-07, 18:39:51 CST] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): task_download_csv> on 2023-02-06 06:00:00+00:00
[2023-02-07, 18:39:51 CST] {base_task_runner.py:129} INFO - Running on host: vmi1120376.contaboserver.net
[2023-02-07, 18:39:51 CST] {base_task_runner.py:130} INFO - Running: ['sudo', '-E', '-H', '-u', 'talend', 'airflow', 'tasks', 'run', 'dag_fetch_public_holiday', 'task_download_csv', 'scheduled__2023-02-06T06:00:00+00:00', '--job-id', '32', '--raw', '--subdir', 'DAGS_FOLDER/fetch_public_holiday.py', '--cfg-path', '/tmp/tmp26yeooeq']
[2023-02-07, 18:39:53 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv /wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-02-07, 18:39:54 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv [[34m2023-02-08, 02:39:54 CST[0m] {[34mdagbag.py:[0m537} INFO[0m - Filling up the DagBag from /wwx/airflow/dags/fetch_public_holiday.py[0m
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv self._accessor.mkdir(self, mode)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv During handling of the above exception, another exception occurred:
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv Traceback (most recent call last):
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/local/bin/airflow", line 8, in <module>
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv sys.exit(main())
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv args.func(args)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return func(*args, **kwargs)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 103, in wrapper
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return f(*args, **kwargs)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 372, in task_run
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv ti.init_run_context(raw=args.raw)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2503, in init_run_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv self._set_context(self)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 77, in _set_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv set_context(self.log, context)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 213, in set_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv flag = cast(FileTaskHandler, handler).set_context(value)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 70, in set_context
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv local_loc = self._init_file(ti)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/wwx/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 320, in _init_file
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv Path(directory).mkdir(mode=0o777, parents=True, exist_ok=True)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1297, in mkdir
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv if not exist_ok or not self.is_dir():
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1422, in is_dir
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return S_ISDIR(self.stat().st_mode)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv File "/usr/lib/python3.8/pathlib.py", line 1198, in stat
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv return self._accessor.stat(self)
[2023-02-07, 18:40:00 CST] {base_task_runner.py:111} INFO - Job 32: Subtask task_download_csv PermissionError: [Errno 13] Permission denied: '/wwx/airflow/logs/dag_id=dag_fetch_public_holiday/run_id=scheduled__2023-02-06T06:00:00+00:00/task_id=task_download_csv'
[2023-02-07, 18:40:01 CST] {local_task_job.py:159} INFO - Task exited with return code 1
[2023-02-07, 18:40:01 CST] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
Then I have examined the problematic folder and found it is permission 700.
I could chmod 777 to the folder manually, however, if there are other new dags/tasks, any new dag/task folder is automatically created with permission 700 and I have to manually chmod.
What I have tried:
I have followed https://airflow.apache.org/docs/apache-airflow/1.10.10/security.html?highlight=impersonation#impersonation
Added the following line to /etc/sudoers
airflow ALL=(ALL) NOPASSWD: ALL
Using sudo to start webserver and scheduler
sudo sh -c 'export AIRFLOW_HOME=/wwx/airflow; /wwx/airflow/venv/bin/airflow scheduler -D'
sudo sh -c 'export AIRFLOW_HOME=/wwx/airflow; /wwx/airflow/venv/bin/airflow webserver -D -p 8090'
Added airflow command to /usr/local/bin so that other users can use
sudo ln -s /wwx/airflow/venv/bin/airflow /usr/local/bin/airflow

Answer 1

I have found the solution as follows:
Add the user to airflow group.
usermod -aG airflow <username>
Set the acl of airflow log directory to have files/folders created with group permission rwx, so the dag run user can access these logs
setfacl -d -m group:airflow:rwx $AIRFLOW_HOME/logs
I am not sure if this is a intended behavior in airflow when running dag as other users, since I cannot find any hint from the airflow documentation.