Some of our DAGs are scheduled to run once per minute for the first 5 minutes of a given hour. For example, the DAG might run at:
20:01
20:02
20:03
20:04
20:05
Most of the time this works fine, but some of the time an entire DAG run is missing. For example, the DAG might run at:
20:01
20:03
20:04
20:05
We have verbose logging turned on, but I don't see anything in the Scheduler logs at 20:02 that obviously explains why the DAG never ran. I also don't see prolonged DAG parsing time that other issues point to as the potential culprit:
Number of DAGs: 177
Total task number: 259
DagBag parsing time: 7.171621999999997
The database resource utilization is minimal, which also is sometimes pointed to as a culprit.
If anyone has ideas of how to approach this problem or hypotheses as to what might cause a DAG not to run at a given 1 minute interval, but run fine at the next 1 minute interval please let me know. Or if you have advice as to what you'd expect to see in the Scheduler logs when it never schedules a DAG, that would he helpful.
Related
I set the schedule like '* 1,5,10,18 * * *' in airflow.
But 18 in yesterday wasn't executed. So I checked the logs.
then I found the job scheduled in 10 executed in 18.
I want to know why and how can I fix.
Note that if you run a DAG on a schedule_interval of one day, the run
stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
other words, the job instance is started once the period it covers has
ended.
~Airflow scheduler docs
So as you can see it will be scheduled after the period - 10->18 is closed, so after 18:00. Check task before, it should be ran just after 10:00.
You don't understand how the airflow scheduler works.
Airflow as DAG Run (data_interval_start) always takes the date of the previous execution, therefore the task performed at your place at 18, has DAG Run at 10. The same as the next task, call at 10 will have a DAG Run at 18 the previous day.
I don’t understand why SLA’s sometimes are triggered.
An example is a dag with an SLA of 'sla': timedelta(hours=1)
I received an email with this:
Here’s a list of tasks that missed their SLAs:
APP_task_group.copy_events on 2022-05-08T03:00:00+00:00
But checking the graph view in the UI I see it started at 04:28 and finished at 04:46, which is range of one hour.
I understand that the SLA is starting in the real UTC time when the dag starts, so the dag with id 2022-05-08T03:00:00+00:00 has between 04:00 and 04:59 UTC to run without raising an SLA.
Am I wrong?
What does explain why this SLA is raising?
After pausing a dag for 2-3 days, when resuming the dag with catchup=False, will run immediately with the last execution.
For example a dag that sends data to an external system is scheduled to run everyday on 19:00.
Stopping the dag for 4 days and enabling on 11:00 would run the dag immediately with yesterdays execution and then again on 19:00 for that day.
In this case the dag runs two times on the day it's resumed.
Is it possible to resume the dag and the first run will happen actually on 19:00?
With default operators, we cannot achieve what you are expecting. Closest to that, what airflow has is LatestOnlyOperator. This is one of the simplest Operators and needs only following configuration
latest_only = LatestOnlyOperator(task_id='latest_only')
This would let the downstream tasks run only if the current time falls between current execution date and next execution date. So, in your case, it would skip execution of three days, but yesterday's run would trigger the jobs.
I am trying to run an Airflow DAG at 04:30 UTC time. I want to run it on all days excepting Tuesday. I have set the schedule_interval in the following way:
schedule_interval='30 04 * * 0,1,3,4,5,6'
However, I see the DAG is not running at the anticipated time instances. If I run manually, then it runs fine. So there is no error in any other place. Could you please point out the mistake in scheduler? Is it not the right approach to exclude a particular day from the scheduler?
Thanks for the help.
I have an airflow dag specified as shown in the picture above.
The git_pull_datagenerator_batch_2 is supposed to be delayed by the TimeDeltaSensor wait_an_hour.
However, the task git_pull_datagenerator seems to be delayed as well although it does not have a dependency on wait_an_hour. (The whole dag is scheduled at 2019-12-10T20:00:00, but git_pull_datagenerator started one hour later than that)
I have checked all documents of airflow but could not find any clues.
I'm assuming your schedule interval is hourly? A DAG run with an execution date of 2019-12-10T20:00:00 on an #hourly schedule interval is expected to run at or shortly after 2019-12-10T21:00:00 when hour 20 has "completed". I don't think it has anything to do with the sensor.
This is a common Airflow pitfall:
Airflow was developed as a solution for ETL needs. In the ETL world,
you typically summarize data. So, if I want to summarize data for
2016-02-19, I would do it at 2016-02-20 midnight GMT, which would be
right after all data for 2016-02-19 becomes available.
If this is what is happening, wait_an_hour started at 2019-12-10T21:00:00 and git_pull_datagenerator_batch_2 at 2019-12-10T22:00:00.
It turns out that the default executor is a SequentialExecutor, which causes all of the tasks to run in a linear order.