Airflow: Create DAG from a separate file - airflow

Question

In airflow, I'm trying to make a function that is dedicated to generate DAGs in a file:
dynamic_dags.py:
def generate_dag(name):
with DAG(
dag_id=f'dag_{name}',
default_args=args,
start_date=days_ago(2),
schedule_interval='5 5 * * *',
tags=['Test'],
catchup=False
) as dag:
dummy_task=DummyOperator(
task_id="dynamic_dummy_task",
dag=dag
)
return dag
Then in another file I'm trying to import the dags from a separate file:
load_dags.py:
from dynamic_dag import generate_dag
globals()["Dynamic_DAG_A"] = generate_dag('A')
However, the dags are not shown up on web UI.
But if I do them in a single file as below code, it will work:
def generate_dag(name):
with DAG(
dag_id=f'dag_{name}',
default_args=args,
start_date=days_ago(2),
schedule_interval='5 5 * * *',
tags=['Test'],
catchup=False
) as dag:
dummy_task=DummyOperator(
task_id="dynamic_dummy_task",
dag=dag
)
return dag
globals()["Dynamic_DAG_A"] = generate_dag('A')
I'm wondering why doing it in two separate files doesn't work.

Answer 1

I think if you are using Airflow 1.10, then the DAG files should contain DAG and airlfow:
https://airflow.apache.org/docs/apache-airflow/1.10.15/concepts.html?highlight=airflowignore#dags
When searching for DAGs, Airflow only considers python files that contain the strings “airflow” and “DAG” by default. To consider all python files instead, disable the DAG_DISCOVERY_SAFE_MODE configuration flag.
In Airflow 2 it's been changed (slightly - dag is case-insensitive):
https://airflow.apache.org/docs/apache-airflow/2.2.2/concepts/dags.html
When searching for DAGs inside the DAG_FOLDER, Airflow only considers Python files that contain the strings airflow and dag (case-insensitively) as an optimization.
To consider all Python files instead, disable the DAG_DISCOVERY_SAFE_MODE configuration flag.
I think you simply miss 'airflow' in your load_dags.py. You can add it wherever - including comments.