Airflow tasks set to `no_status` when catchup is True - airflow

Question

I'm attempting to configure a series of Airflow tasks to backfill some data (catchup=True). Once the DAG is deployed and unpaused, the first job runs successfully, but all subsequent runs have their tasks set to no_status and they never start.
I've tried variations on renaming the DAG, restarting the Airflow server and scheduler, clearing out old logs, but I'm not making any progress here.
Thoughts?
DAG code:
default_args = {
"owner": "me",
"retries": 2,
"retry_delay": timedelta(minutes=2),
"sla": timedelta(hours=1),
"start_date": "2021-01-01T00:00",
}
dag = DAG(
catchup=True,
dag_id="ingest_dag_testing_6",
dagrun_timeout=timedelta(hours=1),
default_args=default_args,
max_active_runs=1,
schedule_interval="30 * * * *",
)
DATA_SOURCE_TYPES = [
{
"target_name": "task_a",
"children": [
{
"target_name": "subtask_a1",
},
{
"target_name": "subtask_a2",
},
],
}
]
with dag:
for dst in DATA_SOURCE_TYPES:
sub_ingest_tasks = []
ingest_task = PythonOperator(
task_id=f"ingest_{dst.get('target_name')}",
python_callable=run_ingestion,
op_args=[logger, exe_date, dst],
)
if dst.get("children"):
for sdst in dst.get("children"):
sub_ingest_tasks.append(
PythonOperator(
task_id=f"ingest_{sdst.get('target_name')}",
python_callable=run_ingestion,
op_args=[logger, exe_date, sdst],
)
)
ingest_task >> sub_ingest_tasks

Answer 1

Your code is executed just fine.
I created a runnable example from your code (as it lack imports/callables):
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import timedelta
def run_ingestion(**context):
print("Hello World")
default_args = {
"owner": "me",
"retries": 2,
"retry_delay": timedelta(minutes=2),
"sla": timedelta(hours=1),
"start_date": "2021-01-01T00:00",
}
dag = DAG(
catchup=True,
dag_id="ingest_dag_testing_6",
dagrun_timeout=timedelta(hours=1),
default_args=default_args,
max_active_runs=1,
schedule_interval="30 * * * *",
)
DATA_SOURCE_TYPES = [
{
"target_name": "task_a",
"children": [
{
"target_name": "subtask_a1",
},
{
"target_name": "subtask_a2",
},
],
}
]
with dag:
for dst in DATA_SOURCE_TYPES:
sub_ingest_tasks = []
ingest_task = PythonOperator(
task_id=f"ingest_{dst.get('target_name')}",
python_callable=run_ingestion,
#op_args=[logger, exe_date, dst],
)
if dst.get("children"):
for sdst in dst.get("children"):
sub_ingest_tasks.append(
PythonOperator(
task_id=f"ingest_{sdst.get('target_name')}",
python_callable=run_ingestion,
#op_args=[logger, exe_date, sdst],
)
)
ingest_task >> sub_ingest_tasks
You can see it's working fine:
If you are running old Airflow version it's possible that changing the dag_id may fix the problem. It could be that there are some old traces of db records related to this dag_id which were not cleaned properly. The scheduler was refactored significantly in later versions.
If the above did not help probably the only solution would be to upgrade to the latest Airflow version as it's probably a bug in older versions which was fixed along the way (Since the code you shared doesn't reproduce the problem you are describing in latest Airflow version).