I have a bunch of jobs that run M-F but have upstream dependencies that run M-S.
Job A was running at 2am before Job B had completed for the day because it was reading a success of Job B from Saturday's run. Starting conditions originally were s(job-b,0) but I've updated it to s(job-b,12.00). This works for the most part, but a lot of the jobs time out when their upstreams are running late. Is it possible to only start Job A when Job B has already started and successfully completed that same day?
Example: Job A only starts on Monday when Job B has already successfully started & completed that Monday - does not look at any other day's runs.
Related
I set the schedule like '* 1,5,10,18 * * *' in airflow.
But 18 in yesterday wasn't executed. So I checked the logs.
then I found the job scheduled in 10 executed in 18.
I want to know why and how can I fix.
Note that if you run a DAG on a schedule_interval of one day, the run
stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
other words, the job instance is started once the period it covers has
ended.
~Airflow scheduler docs
So as you can see it will be scheduled after the period - 10->18 is closed, so after 18:00. Check task before, it should be ran just after 10:00.
You don't understand how the airflow scheduler works.
Airflow as DAG Run (data_interval_start) always takes the date of the previous execution, therefore the task performed at your place at 18, has DAG Run at 10. The same as the next task, call at 10 will have a DAG Run at 18 the previous day.
A job has a predecessor. So as aoon as the predecessor is finished the job starts running. But is it possible to schedule the job as follows:
1)when the predecessor fonishes before say 5 o clock the job should wait until 5 to start.
2) When the predecessor finished after 5 oclock then the job should wait for it to finish
Edit the second job to have a "From Time" of 17:00. If you have a condition between the 2 jobs then they will run as you describe.
Good Day,
we use Airflow to orchestrate runs of our jobs.
The Job in question is usually scheduled for 2:30 and takes quite some time.
Due to a new data source it was expected to run a full day.
Since our jobs don't work parallel we set max_active_runs_per_dag to 1 to ensure that there are no multiple instances of the same job even when it takes more than 24h. In general this seem to work, but not in this case.
What happened:
We triggered a manual run at 13:00
at 2:30 (next day) the scheduled run is triggered and runs simultaneously
Expectation:
The scheduled run should wait for the manual run to finish
More Information:
The Airflow instance did not restart.
Airflow version 1.10.2
I thank you for any advice.
Seems to be an open Issue which will be fixed in 2.1 and 1.15.
A workaround was not provided yet.
https://github.com/apache/airflow/issues/9975
I am using latest AirFlow. I ran a dag file which just executes print and sleeps command for 10 seconds.
Once that dag complete 50 successful runs it automatically stoped. When I restart the web server , scheduler and worker then it again runs for another 50. I did this way 2 times and the same result.
There is some issue with the schedule.
Most likely reason is the schedule is made of Catchup or Backfill task with specific start_date and end_date parameters .
for more details https://airflow.apache.org/docs/stable/dag-run.html
I am trying to understand autosys job. Suppose I have Job A that runs every 15 minutes. Suppose for some reason if Job A takes more than 15 minutes, will another instance of it run or it will wait for the job to finish before running another instance?
In my experience, if the previous job run is still running, another instance will not run if the next scheduled time comes. The next time the job runs is when the previous run is finished and the next scheduled time comes.
Another user also experienced this according to this answer.
I did not find any AutoSys documentation that officially confirms what happens in this situation, but I guess the best way to find out is to test it on your AutoSys instance.
I have experienced this first hand and can confirm that there won't be two instances in the mentioned scenario. The job will wait on the previous run to complete and will immediately kick off the next instance if the time condition is met before the previous completes.
But this will be the case only when the job is in running state, if the job is in any other state it will kick off based on the given start_time condition.