How to enforce max active run = 1 for a group of DAGs in Airflow? - airflow

Question

I have a group of DAGs and I only want one of them to run at any given time.
ExternalTaskSensor will not work if I trigger a backfill job for one of them for a very old date.
I am aware of pool and priority weights method.
Another approach could be to make a custom operator and check all the dag runs of all the dags in the group.
Is there any other method to achieve this?

Answer 1

Airflow doesn't support this feature, even if you use pools, you need to use the same pool for all the tasks from all the dags in the group, and set the pool slots to 1, which break the parallelism.
You can achieve this by merging the dags in one dag and adding a branch operator which processes a param from dag_run conf to know which dag should it runs:
import pendulum
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import BranchPythonOperator
from airflow.models.param import Param
from airflow.models import DAG
from airflow.decorators import task
default_args = {}
def dag_1(main_dag: DAG):
dag_id = "dag_1"
start_task = EmptyOperator(
task_id=dag_id,
dag=main_dag
)
task_1 = EmptyOperator(
task_id=f"{dag_id}.task1",
dag=main_dag
)
task_2 = EmptyOperator(
task_id=f"{dag_id}.task2",
dag=main_dag
)
start_task >> task_1 >> task_2
return start_task
def dag_2(main_dag: DAG):
dag_id = "dag_2"
start_task = EmptyOperator(
task_id=dag_id,
dag=main_dag
)
task_1 = EmptyOperator(
task_id=f"{dag_id}.task1",
dag=main_dag
)
task_2 = EmptyOperator(
task_id=f"{dag_id}.task2",
dag=main_dag
)
task_3 = EmptyOperator(
task_id=f"{dag_id}.task3",
dag=main_dag
)
start_task >> [task_1, task_2] >> task_3
return start_task
with DAG(
dag_id='multiple_dags',
default_args=default_args,
start_date=pendulum.datetime(2023, 1, 1),
schedule=None,
max_active_runs=1,
params={
"dag_id": Param(default="dag_1", enum=["dag_1", "dag_2"])
}
) as dag:
#task.branch(task_id="start_task")
def branch(**context):
return context["params"]["dag_id"]
branch() >> [
dag_1(dag),
dag_2(dag)
]
for param dag_1:
for param dag_2:
Then if you want to run these dags on different schedules, you can create N new dags contains one task from TriggerDagRunOperator to trigger the main dag and pass the dag id as param:
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
def create_trigger_dag(dag_id, schedule):
with DAG(
dag_id=dag_id,
start_date=pendulum.datetime(2023, 1, 1),
schedule=schedule,
catchup=False
) as dag:
TriggerDagRunOperator(
task_id="trigger_dag",
trigger_dag_id="multiple_dags",
conf={
"dag_id": dag_id
}
)
return dag
trigger_dag_1 = create_trigger_dag(dag_id="dag_1", schedule="*/1 * * * *")
trigger_dag_2 = create_trigger_dag(dag_id="dag_2", schedule="*/2 * * * *")
And here is the result, 2 runs from dag_1 for each run from dag_2: