I'm looking for a way to get the queued time duration of an Airflow task.
I'm running on all tasks of a DAG, but if its state is Queued - there is no start_date and therefore I can't tell how long it is in a Queued state.
Does anyone know a way ?
I have an idea for a rough queued time, but it's only possible after the task has actually started.
You can find the queued time by substracting the end_date of the "Parent" task from the start_date of your desired task once it started
If there's no "parent" task, you can might use the DAG start_date
Notes:
Took into consideration the downstream task to start after ALL upstream tasks done, if your rule is different, please adapt the logic
If there's more than one upstream task, use the latest end_date
Related
I'm just learning Apache Airflow. I understand that the execution date is not the same time as the actual time a dag run is triggered.
Note that if you run a DAG on a schedule_interval of one day, the run stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In other words, the job instance is started once the period it covers has ended.
Let’s Repeat That The scheduler runs your job one schedule_interval AFTER the start date, at the END of the period.
Yeah, For a daily job, cron jobs run at the start of the day; Airflow jobs run at the end of the day.
I humbly ask: Anyway to set the execution date same as the trigger time?
You generally structure your tasks such that you'll provide a date to the job via kwargs (for idempotency, etc).
Airflow provides macros (https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html) that expose both the data_interval_start and the data_interval_end.
I believe you're looking for the data_interval_end which aligns with the logical date that the job is running.
I am trying to define periods of time where the scheduler will not mark tasks as ready.
For example, I have a DAG with a lengthy backfill period. The execution of the backfill will run throughout the day until the DAG is caught up. However I do not want and executions of the DAG to execute between midnight and 2 am.
Is it possible to achieve this through configuration?
I have an airflow dag specified as shown in the picture above.
The git_pull_datagenerator_batch_2 is supposed to be delayed by the TimeDeltaSensor wait_an_hour.
However, the task git_pull_datagenerator seems to be delayed as well although it does not have a dependency on wait_an_hour. (The whole dag is scheduled at 2019-12-10T20:00:00, but git_pull_datagenerator started one hour later than that)
I have checked all documents of airflow but could not find any clues.
I'm assuming your schedule interval is hourly? A DAG run with an execution date of 2019-12-10T20:00:00 on an #hourly schedule interval is expected to run at or shortly after 2019-12-10T21:00:00 when hour 20 has "completed". I don't think it has anything to do with the sensor.
This is a common Airflow pitfall:
Airflow was developed as a solution for ETL needs. In the ETL world,
you typically summarize data. So, if I want to summarize data for
2016-02-19, I would do it at 2016-02-20 midnight GMT, which would be
right after all data for 2016-02-19 becomes available.
If this is what is happening, wait_an_hour started at 2019-12-10T21:00:00 and git_pull_datagenerator_batch_2 at 2019-12-10T22:00:00.
It turns out that the default executor is a SequentialExecutor, which causes all of the tasks to run in a linear order.
I am in a situation where I have started getting some data scheduled daily at a certain time and I have to create ETL for that data.
Meanwhile, when I am still creating the DAGs for scheduling the tasks in Airflow. The data keeps on arriving daily. So when I will start running my DAGs from today I want to schedule it daily and also wants to backfill all the data from past days which I missed while I was creating DAGs.
I know that if I put start_date as the date from which the data started arriving airflow will start backfilling from that date, but wouldn't in that case, my DAGs will always be behind of current day? How can I achieve backfilling and scheduling at the same time? Do I need to create separate DAGs/tasks for backfill and scheduling?
There are several things you need to consider.
1. Is your daily data independent or the next run is dependent on the previous run?
If the data is dependent on previous state you can run backfill in Airflow.
How backfilling works in Airflow ?
Airflow gives you the facility to run past DAGs. The process of running past DAGs is called Backfill. The process of Backfill actually let Airflow forset some status of all DAGs since it’s inception.
I know that if I put start_date as the date from which the data
started arriving airflow will start backfilling from that date, but
wouldn't in that case, my DAGs will always be behind of current day?
Yes setting a past start_date is the correct way of backfilling in airflow.
No, If you use celery executer, the jobs will be running in parallel and it will eventually catch up to the current day , obviously depending upon your execution duration.
How can I achieve backfilling and scheduling at the same time? Do I
need to create separate DAGs/tasks for backfill and scheduling?
You do not need to do anything extra to achieve scheduling and backfilling at the same time, Airflow will take care of both depending on your start_date
Finally , If this activity is going to be one time task I recommend , you process your data(manually) offline to airflow , this will give you more control over the execution.
and then either mark the backfilled tasks as succeed or below
Run an airflow backfill command like this: airflow backfill -m -s "2016-12-10 12:00" -e "2016-12-10 14:00" users_etl.
This command will create task instances for all schedule from 12:00 PM to 02:00 PM and mark it as success without executing the task at all. Ensure that you set your depends_on_past config to False, it will make this process a lot faster. When you’re done with it, set it back to True.
Or
Even simpler set the start_date to current date
When clearing a task of a DAG for January and Februrary 2019, I noticed that all tasks of this DAG that did not exist at the time were triggered.
I'm wondering why this happens. I suppose the scheduler is kind of "forced" to look at the DAG runs of January and February, and because the tasks that did not exist at the time never ran for these execution dates, they get triggered. But I'd like to put concrete words on this vague understanding of the situation.
Can I avoid this? This creates unexpected behavior and has me doubting before launching a big replay of a month that is long past :)
We have also encountered this problem and I think it makes sense. As per Airflow documentation stated.
Once you clear a DAG, it will be cleared as if it never runs.
so in my understanding, it will check all dag and task instance all over again, run all the task until it reached the schedule time.
Can I avoid this? I'm no airflow expert but I think as of now, we can't. What we normally do is to duplicate the DAG we want to rerun and set start_date and end_date, so it will not intervene with the current DAG that is running normally.