How to use requirements.txt file in Airflow's PythonVirtualenvOperator? - airflow

Question

In Airflow 2.2.3, PythonVirtualenvOperator was updated to allow templated requirements.txt files in the requirements parameter. However, I am unable to properly utilize that parameter. I am seeking guidance on how to properly utilize a requirements.txt file with Airflow's PythonVirtualenvOperator.
Here's my /dags format:
dags
├── .airflowignore
├── daily.py
└── modules
└── monday
├── monday.py
└── requirements.txt
In daily.py, I define my daily DAG. The task I'd like to reference requirements.txt for is defined like so:
#task.virtualenv(requirements='modules/monday/requirements.txt')
def sync_board_items():
from modules.monday.monday import sync_board_items
import logging
logging.basicConfig(level=logging.INFO)
sync_board_items(board_id=XXXX, table=XXXX)
This seems to fit the implementation described in GitHub, because requirements is a string, not a list, and it complies with the *.txt template. However, when the task runs, I quickly receive an error:
Executing cmd: /tmp/venvfn63dy3c/bin/pip install m o d u l e s / m o n d a y / r e q u i r e m e n t s . t x t
ERROR: Directory '/' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
This seems to indicate that PythonVirtualenvOperator is treating my requirements param like a list instead of a string. In other words, I am doing something wrong and the PythonVirtualenvOperator is not properly handling my requirements.txt file. What am I missing or how can I leverage a requirements.txt file in Airflow's PythonVirtualenvOperator?

Answer 1

The feature is available only for Airflow>=2.3.0 it's not available in 2.2.3
Currently 2.3.0 is under testing. 2.3.0b1 is available via pip.
if you want to use this feature in 2.2.3 you will need to create a custom operator by backporting the code in the PR.
Creating a MyPythonVirtualenvOperator with the code in the PR and utils should work fine in 2.2.3