Is there a possibility to prevent the scheduler from triggering a DAG as long as there is still a running instance from the same DAG?
Thanks!
Found the answer in the Docs. Passing the flag max_active_runs while constructing the DAG object, does the trick:
DAG(max_active_runs=1)
https://airflow.apache.org/docs/stable/_api/airflow/models/dag/index.html
Related
Recently, we have been getting some errors on airflow where certain dags will not run any tasks but are being marked as complete.
We had the start_date using days_ago from airflow.
from airflow.utils.dates import days_ago
From: https://forum.astronomer.io/t/dag-run-marked-as-success-but-no-tasks-even-started/1423
If you see dag runs that are marked as success but don’t have any task runs, this means the dag runs’ execution_date was earlier than the dag’s start_date.
This is most commonly seen when the start_date is set to some dynamic value e.g. airflow.utils.dates.days_ago(0). This creates the opportunity for the execution date of a delayed dag execution to be before what the dag now thinks is it’s start_date. This can even happen in a cyclic pattern, where a few dagruns will work, and then at the beginning of every day a dagrun will experience this problem.
This simplest way to avoid this problem is the never use dynamic start_date. It is always better to specify a static start_date. If you are concerned about accidentally triggering multiple runs of the same dag, just set catchup=False.
There is an open ticket in Airflow project with this issue: https://github.com/apache/airflow/issues/17977
I have a question about the TriggerDagRunOperator , specifically the wait_for_completion parameter.
Before moving to Airflow 2.2, we used this operator to trigger another DAG and a ExternalTaskSensor to wait for its completion.
In Airflow 2.2, there is a new parameter that is called wait_for_completion that if sets to True, will make the task complete only when the triggered DAG completed.
This is great, but I was wondering about wether the worker will be released between pokes or not. I know that the ExternalTaskSensor used to have a parameter reschedule that you can use for pokes larger than 1m which will release the worker slot between pokes - but I don’t see it in the documentation anymore.
My question is if the wait_for_completion parameter causes the operator to release the worker between pokes or not? From looking at the code I don’t think that is the case, so I just want to verify.
If it isn’t releasing the worker and the triggered DAG is bound to take more than 1m to finish, what should be the best approach here?
We are using MWAA Airflow 2.2 so I guess deferred operators are not an option (if it is a solution in this case)
When using wait_for_completion=True in TriggerDagRunOperator the worker will not be released as long as the operator is running. You can see that in the operator implementation. The operator use time.sleep(self.poke_interval)
As you pointed there are two ways to achieve the goal of verifying the triggered dag completed:
DAG A Using TriggerDagRunOperator followed by ExternalTaskSensor
Using TriggerDagRunOperator with wait_for_completion=True
However other than resources issue which you mentioned the two options are not really equivalent.
In option 1 if the triggered DAG fails then the ExternalTaskSensor will fail.
In option 2 consider:
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
my_op = TriggerDagRunOperator (
task_id='task',
trigger_dag_id="dag_b",
...,
wait_for_completion=True,
retries=2
)
if the dag_b fails then TriggerDagRunOperator will retry which will invoke another DagRun of dag_b.
Both options are valid. You need to decide which behavior suitable for your use case.
Is there anyway to start the dag automatically without manually triggering dag once it is available in dagbag, considering is_paused_upon_creation=false is set.
Please, try to create your DAG with schedule_interval='#once'. It should works.
I have DAG with ExternalTaskSensor in it. I correctly set execution_delta and all work perfectly unless I want to run that DAG manually. My ExternalTaskSensor has state running and after timeout interval it's failed with exception airflow.exceptions.AirflowSensorTimeout: Snap. Time is OUT.
I know way to run it - after manually triggered DAG set every ExternalTaskSensor in DAG to state success in web interface.
But is the better way to run it manually without to set every ExternalTaskSensor in DAG to success ?
I'm very confused by how airflow picks up the changes in new DAG code with the scheduler.
Can someone clarify how the airflow scheduler works with new code? Do I need to stop and rerun airflow scheduler every time I change code in DAGs? or can I just set --num_runs 1and run it every time I make new changes?
Thanks!
The scheduler should be running all the time. You should just run airflow scheduler without a num_runs param. The scheduler is designed to be a long running process, an infinite loop. It orchestrates the work that is being done, it is the heart of airflow. If it's not running, you aren't scheduling more work to be done.
Each iteration of the loop will reload what is called the DagBag, a collection of loaded DAGs. Any modifications to a DAG, as well as removal/addition of DAGs should be reflected the next scheduler loop.
Airflow's scheduler checks periodically and continuously the DAGs location to scan and refresh DAGs. If you didn't change the config, it's done with just few seconds of pause between each round.
The --num_run parameter was not introduced for refreshing purposes but for reliability:
Airflow officially advises here that the scheduler should be restarted frequently using the num_runs and/or the run_duration configuration parameters.